JSON HTTP POST SERVER/LISTENER

[kudrow]

Hello,

I need help in creating a TCP listening server to accept a JSON HTTP POST from a remote server. I can do everything accept parse the JSON POST request. 

To elaborate - ScreenConnect/ConnectWise has the ability to create triggers and send information VIA an HTTP request. I wish it would just send it in plain text over TCP with the http request but it will not. So when the trigger fires, it will send the HTTP POST request to the URL I tell it to.

What I have tried - I have tried the "Pure Autoit Webserver" but it I am not sure how to alter it to accept Json POST. I have also tried just running the Example TCP server and listen for the connection. It does work and it does show information received but it does not show the data I need.

Any help is much appreciated.

Thanks!

[kudrow]

This is the example HTTP trigger from screenconnect using Json sending data to HipChat. I just need something in autoit to process this POST.

[argumentum]

https://www.autoitscript.com/forum/topic/148114-a-non-strict-json-udf-jsmn/  can help you.

 

[kudrow]

Thank you, I have been combing through it since my original post. I am trying to make sense of it. I will keep plugging away. 

[Jos]

@kudrow,

No need to PM me for support, just a thread like this is enough and I will have a look when I get to it.

Just post an example script with JSON data in a codebox so we can test and assist!  

Jos  

[Danp2]

8 hours ago, kudrow said:

I have also tried just running the Example TCP server and listen for the connection. It does work and it does show information received but it does not show the data I need

It would probably help if you showed us what it does return.

[kudrow]

Here is my SceenConnect settings and then what is captured on the server side. 

[Danp2]

Without your script, we can only guess at what you are doing to get this response. Are you using TCPRecv() to read the transmitted data?

[kudrow]

31 minutes ago, Danp2 said:

Without your script, we can only guess at what you are doing to get this response. Are you using TCPRecv() to read the transmitted data?

That’s exactly all I am doing. I am just using the “server” example from the help file with it to receive 2000 bytes and the above is all that comes through. 

[argumentum]

@kudrow, kindly copy the script you are using to the forum ( I understand that is that same as in the example but please show the code ).  Thanks  

Edit: ...and stop showing pictures, just the info., we are programmers after all.  

Edited February 8, 2020 by argumentum

[kudrow]

LOL well the pictures are because I cannot copy and past most of it. 

 

Ok so this is the example from the help file. I have modified the port to listen on 80 and I have changed the expected bytes to 2000 so there is enough room to receive all of the data that is sent.

 

expandcollapse popup

#include <GUIConstantsEx.au3>
#include <MsgBoxConstants.au3>

; Start First clicking on "1. Server"
; Then start a second instance of the script selecting "2. Client"

Example()

Func Example()
    TCPStartup() ; Start the TCP service.

    ; Register OnAutoItExit to be called when the script is closed.
    OnAutoItExitRegister("OnAutoItExit")

    ; Assign Local variables the loopback IP Address and the Port.
    Local $sIPAddress = "127.0.0.1" ; This IP Address only works for testing on your own computer.
    Local $iPort = 80 ; Port used for the connection.

    #Region GUI
    Local $sTitle = "TCP Start"
    Local $hGUI = GUICreate($sTitle, 250, 70)

    Local $idBtnServer = GUICtrlCreateButton("1. Server", 65, 10, 130, 22)

    Local $idBtnClient = GUICtrlCreateButton("2. Client", 65, 40, 130, 22)

    GUISetState(@SW_SHOW, $hGUI)

    While 1
        Switch GUIGetMsg()
            Case $GUI_EVENT_CLOSE
                ExitLoop
            Case $idBtnServer
                WinSetTitle($sTitle, "", "TCP Server started")
                GUICtrlSetState($idBtnClient, $GUI_HIDE)
                GUICtrlSetState($idBtnServer, $GUI_DISABLE)
                If Not MyTCP_Server($sIPAddress, $iPort) Then ExitLoop
            Case $idBtnClient
                WinSetTitle($sTitle, "", "TCP Client started")
                GUICtrlSetState($idBtnServer, $GUI_HIDE)
                GUICtrlSetState($idBtnClient, $GUI_DISABLE)
                If Not MyTCP_Client($sIPAddress, $iPort) Then ExitLoop
        EndSwitch

        Sleep(10)
    WEnd

    #EndRegion GUI
EndFunc   ;==>Example

Func MyTCP_Client($sIPAddress, $iPort)
    ; Assign a Local variable the socket and connect to a listening socket with the IP Address and Port specified.
    Local $iSocket = TCPConnect($sIPAddress, $iPort)
    Local $iError = 0

    ; If an error occurred display the error code and return False.
    If @error Then
        ; The server is probably offline/port is not opened on the server.
        $iError = @error
        MsgBox(BitOR($MB_SYSTEMMODAL, $MB_ICONHAND), "", "Client:" & @CRLF & "Could not connect, Error code: " & $iError)
        Return False
    EndIf

    ; Send the string "tata" to the server.
    TCPSend($iSocket, "tata")

    ; If an error occurred display the error code and return False.
    If @error Then
        $iError = @error
        MsgBox(BitOR($MB_SYSTEMMODAL, $MB_ICONHAND), "", "Client:" & @CRLF & "Could not send the data, Error code: " & $iError)
        Return False
    EndIf

    ; Close the socket.
    TCPCloseSocket($iSocket)
EndFunc   ;==>MyTCP_Client

Func MyTCP_Server($sIPAddress, $iPort)
    ; Assign a Local variable the socket and bind to the IP Address and Port specified with a maximum of 100 pending connexions.
    Local $iListenSocket = TCPListen($sIPAddress, $iPort, 100)
    Local $iError = 0

    If @error Then
        ; Someone is probably already listening on this IP Address and Port (script already running?).
        $iError = @error
        MsgBox(BitOR($MB_SYSTEMMODAL, $MB_ICONHAND), "", "Server:" & @CRLF & "Could not listen, Error code: " & $iError)
        Return False
    EndIf

    ; Assign a Local variable to be used by the Client socket.
    Local $iSocket = 0

    Do ; Wait for someone to connect (Unlimited).
        ; Accept incomming connexions if present (Socket to close when finished; one socket per client).
        $iSocket = TCPAccept($iListenSocket)

        ; If an error occurred display the error code and return False.
        If @error Then
            $iError = @error
            MsgBox(BitOR($MB_SYSTEMMODAL, $MB_ICONHAND), "", "Server:" & @CRLF & "Could not accept the incoming connection, Error code: " & $iError)
            Return False
        EndIf

        If GUIGetMsg() = $GUI_EVENT_CLOSE Then Return False
    Until $iSocket <> -1 ;if different from -1 a client is connected.

    ; Close the Listening socket to allow afterward binds.
    TCPCloseSocket($iListenSocket)

    ; Assign a Local variable the data received.
    Local $sReceived = TCPRecv($iSocket, 2000) ;we're waiting for the string "tata" OR "toto" (example script TCPRecv): 4 bytes length.

    ; Notes: If you don't know how much length will be the data,
    ; use e.g: 2048 for maxlen parameter and call the function until the it returns nothing/error.

    ; Display the string received.
    MsgBox($MB_SYSTEMMODAL, "", "Server:" & @CRLF & "Received: " & $sReceived)

    ; Close the socket.
    TCPCloseSocket($iSocket)
EndFunc   ;==>MyTCP_Server

Func OnAutoItExit()
    TCPShutdown() ; Close the TCP service.
EndFunc   ;==>OnAutoItExit

 

[argumentum]

1 minute ago, kudrow said:

Local $sReceived = TCPRecv($iSocket, 2000) ;we're waiting for the string "tata" OR "toto"

that is not in a loop. TCP is sent in chunks, so, if the message is in more than one chunk, it will be missing.  
There are web server examples in the forum. Use that as a starting point  

[kudrow]

I am confused.. I do not think it has to be in a loop because it simply waits until either all bytes are received or it stops receiving if it reaches the limit of 2000. A TCP packet (or chunk)  is 20 to 60 bytes. The original example has it at 4 to receive "tata" or "toto". If I leave it at 4, the result is that it receives only 4 bytes (which would be POST if I was sending the POST command. 

I tried the web servers with no luck as well. Here is the "POST" processing of the webserver but the test message box is completely empty. No matter where I move the test message box its always empty and I have tried using $sComment in the msg box. I think the problem is that this is looking for an HTML post vs JSON. 

ElseIf $sRequestType = "POST" Then ; user has come to us with data, we need to parse that data and based on that do something special

                    $aPOST = _HTTP_GetPost($sBuffer[$x]) ; parses the post data
                    MsgBox(0, "TEST", $aPOST)

                    $sComment = _HTTP_POST("wintext",$aPOST) ; Like PHPs _POST, but it requires the second parameter to be the return value from _Get_Post


                    _HTTP_ConvertString($sComment) ; Needs to convert the POST HTTP string into a normal string

 

[argumentum]

I'm busy right now but I'll code a basic "I'll take anything" example for you to start with ..in an hour or so  

[argumentum]

expandcollapse popup

#AutoIt3Wrapper_Au3Check_Parameters=-q -d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6 -w 7
#include <MsgBoxConstants.au3>
#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#Region ### START Koda GUI section ### Form=
Global $iListenSocket, $aForm[2] = [600, 400]
Global $Form = GUICreate(Chr(160) & 'TCP in a loop on "0.0.0.0" port 80 ( example of fake http deamon )', $aForm[0], $aForm[1], (@DesktopWidth / 3) * 2, (@DesktopHeight / 2), BitOR($GUI_SS_DEFAULT_GUI, $WS_MAXIMIZEBOX, $WS_SIZEBOX, $WS_THICKFRAME, $WS_TABSTOP), BitOR($WS_EX_TOPMOST, $WS_EX_WINDOWEDGE))
GUISetFont(10, 400, 0, "Courier New")
Global $Edit = GUICtrlCreateEdit("", 0, 0, $aForm[0], $aForm[1])
GUICtrlSetLimit(-1, 0, 0)
GUICtrlSetResizing(-1, $GUI_DOCKLEFT + $GUI_DOCKRIGHT + $GUI_DOCKTOP + $GUI_DOCKBOTTOM)
;~ Opt("GUIOnEventMode", 1)
;~ GUISetOnEvent($GUI_EVENT_CLOSE, "OnEventClose")
GUISetState(@SW_SHOW)
#EndRegion ### END Koda GUI section ###

Func OnEventClose()
    GUIDelete()
    Exit
EndFunc   ;==>OnEventClose

TCPStartup() ; Start the TCP service.
OnAutoItExitRegister("OnAutoItExit") ; Register OnAutoItExit to be called when the script is closed.
Func OnAutoItExit()
    TCPShutdown() ; Close the TCP service.
EndFunc   ;==>OnAutoItExit

mainloop()
; Close the Listening socket to allow afterward binds.
TCPCloseSocket($iListenSocket)

Func mainloop()
    Opt("TCPTimeout", 20)
    Local $iTestSocketCount = 0, $iError = 0, $buffer, $bReceived

    ; Assign a variable the socket and bind to the IP Address and Port specified with a maximum of 100 pending connexions.
    $iListenSocket = TCPListen("0.0.0.0", 80, 100)

    If @error Then
        $iError = @error ; Someone is probably already listening on this IP Address and Port (script already running?).
        MsgBox(BitOR($MB_SYSTEMMODAL, $MB_ICONHAND), "", "Server:" & @CRLF & "Could not listen, Error code: " & $iError, 60, $Form)
        Return False
    EndIf

    Local $iCountTryCloseSocket, $TCPRecvErr, $TCPRecvExt, $iSocket = -1 ; Assign a Local variable to be used by the Client socket.
    Local $sHtml, $sTimer, $hTimer, $urlReq, $method, $ContentLength, $ContentBody, $ContentType, $Referer, $aQueryData, $AcceptEncoding, $Header_Expect, $nBSp = "&nb" & "sp;"

    While 1

        If $iSocket = -1 Then
            $iSocket = TCPAccept($iListenSocket)
            If @error Then ; If an error occurred display the error code and return False.
                $iError = @error
                MsgBox(BitOR($MB_SYSTEMMODAL, $MB_ICONHAND), "", "Server:" & @CRLF & "Could not accept the incoming connection, Error code: " & $iError)
                Return False
            EndIf
            If $iSocket > 0 Then
                $iTestSocketCount += 1
                $hTimer = TimerInit()
                ContinueLoop
            EndIf
        Else
            $buffer = StringToBinary("")
            Do
                $bReceived = TCPRecv($iSocket, 4096, 1) ; $TCP_DATA_BINARY (1) - return binary data
                $TCPRecvErr = @error
                $TCPRecvExt = @extended
                $buffer &= $bReceived
                If DiscernData(BinaryToString($buffer), $urlReq, $method, $ContentLength, $ContentBody, $ContentType, $Referer, $aQueryData, $AcceptEncoding, $Header_Expect) Then ExitLoop
            Until $TCPRecvExt Or $bReceived = ""
            $sTimer = "[ " & Round(TimerDiff($hTimer), 2) & " ms. ]"
            $hTimer = TimerInit()
            If $TCPRecvErr Then
                TCPCloseSocket($iSocket) ; ; Close the socket.
                $iSocket = -1
                GUICtrlSetData($Edit, "=== " & $iTestSocketCount & " ===>" & "<=== Error: " & $TCPRecvErr & $sTimer & @CRLF & GUICtrlRead($Edit))
            Else
                If $Header_Expect = 100 Then
                    GUICtrlSetData($Edit, "=== " & $iTestSocketCount & "(TBC) ===>" & BinaryToString($buffer) & "<=== " & $sTimer & @CRLF & GUICtrlRead($Edit))
                    HttpSender($iSocket, $AcceptEncoding, "", "text/html", "100 Continue")
                    $Header_Expect = 0
                    Sleep(200)
                    ContinueLoop
                EndIf
                $sHtml = "<!DOCTYPE html><html lang=""en""><head></head><body>" & _
                        "The browser is waiting for a response, so, here it is " & $nBSp & $nBSp & $nBSp & $nBSp & " =)<br>" & _
                        "<br><h2>" & @MIN & ":" & @SEC & "." & @MSEC & "</h2></body></html>"
                HttpSender($iSocket, $AcceptEncoding, $sHtml)
                $sTimer &= "[ " & Round(TimerDiff($hTimer), 2) & " ms. ]"
                GUICtrlSetData($Edit, "=== " & $iTestSocketCount & " ===>" & BinaryToString($buffer) & "<=== " & $sTimer & @CRLF & GUICtrlRead($Edit))
                $iCountTryCloseSocket = 0
                Do
                    If $iCountTryCloseSocket Then Sleep(10)
                    $iCountTryCloseSocket += 1
                    If $iCountTryCloseSocket > 5 Then ExitLoop
                Until TCPCloseSocket($iSocket)
                $iSocket = -1
            EndIf
        EndIf
        If TimerDiff($hTimer) < 200 Then ContinueLoop
        Switch GUIGetMsg()
            Case -3
                GUIDelete()
                Exit
        EndSwitch
    WEnd
EndFunc   ;==>mainloop

; example of a "more proper" way to send back a response
Func HttpSender(ByRef $iSocket, ByRef $AcceptEncoding, $sHtmlText, $ContentType = "text/html", $ResponseStatusCode = "200 OK")
    #forceref $AcceptEncoding
    Local $_Head = "", $bHtmlText = Binary($sHtmlText)
    $_Head &= "HTTP/1.0 " & $ResponseStatusCode & @CRLF ; https://en.wikipedia.org/wiki/List_of_HTTP_status_codes
    $_Head &= "Content-Type: " & $ContentType & @CRLF ; ..so the client knows what is getting.
    $_Head &= "Content-Lenght: " & BinaryLen($bHtmlText) & @CRLF ; this is a must for binary file send. Might as well use it for everything.
    $_Head &= "Cache-Control: no-cache" & @CRLF
    $_Head &= "Cache-Control: no-store" & @CRLF
    $_Head &= "Connection: close" & @CRLF ; SideNote: ..an interaction takes 10 ms., is there a need for "muti-concurrent" code ?   ;)
    $_Head &= @CRLF ; end of header marker
    $_Head = Binary($_Head)

    TCPSendBinary($iSocket, $_Head) ; Send the head
    TCPSendBinary($iSocket, $bHtmlText) ; Send the body ( html or file, as delimited by the header )

EndFunc   ;==>HttpSender

Func TCPSendBinary(ByRef $hSocket, ByRef $bData)
    Local $iBytesSent
    Do
        $iBytesSent = TCPSend($hSocket, $bData)
        If @error Then ExitLoop
        $bData = BinaryMid($bData, $iBytesSent + 1, BinaryLen($bData) - $iBytesSent)
    Until 0 = BinaryLen($bData)
EndFunc   ;==>TCPSendBinary

Func DiscernData($buffer, ByRef $urlReq, ByRef $method, ByRef $ContentLength, ByRef $ContentBody, ByRef $ContentType, ByRef $Referer, ByRef $aQueryData, ByRef $AcceptEncoding, ByRef $Header_Expect)
    $urlReq = "" ;       $aQueryData holds the query as an array but a JSON string will need extra coding.
    $method = "" ;         If the $aQueryData[1][1] = "" then is likely a JSON string
    $ContentLength = 0 ; In this example, at first, all I wanted, is to know that the data sent is full
    $ContentBody = "" ;    and complete, to speed up the loop ( and save about 100 ms. of timeout )
    Local $_Head = StringLeft($buffer, StringInStr($buffer, @CRLF & @CRLF) - 1)
    $ContentBody = StringTrimLeft($buffer, StringLen($_Head) + 4)
    If $Header_Expect = 100 Then $ContentBody = $buffer
    Local $_BodyLen = StringLen($ContentBody)
    Local $aCRLF = StringSplit($_Head, @CRLF, 1)
    If UBound($aCRLF) < 2 Then Return
    Local $aSpaceSplit = StringSplit($aCRLF[1], " ")
    If UBound($aSpaceSplit) < 3 Then Return
    $method = $aSpaceSplit[1]
    $urlReq = $aSpaceSplit[2]
    For $n = 2 To $aCRLF[0]
        $aSpaceSplit = StringSplit($aCRLF[$n], ": ", 1)
        If UBound($aSpaceSplit) < 3 Then ContinueLoop
        If $aSpaceSplit[1] = "Content-Length" Then $ContentLength = Int($aSpaceSplit[2]) ; tis should be equal to StringLen($ContentBody)
        If $aSpaceSplit[1] = "Content-Type" Then $ContentType = $aSpaceSplit[2] ; Discern if is JSON, when needed
        If $aSpaceSplit[1] = "Referer" Then $Referer = $aSpaceSplit[2] ; Discern if is from "/home/"
        If $aSpaceSplit[1] = "Accept-Encoding" Then $AcceptEncoding = $aSpaceSplit[2] ; Discern if to use GZIP
        If $aSpaceSplit[1] = "Expect" Then $Header_Expect = Int($aSpaceSplit[2]) ; Discern if Expect: 100-continue
    Next
    If $Header_Expect = 100 Then Return SetError(0, 1, 1)
    Dim $aQueryData[1][2] = [[0, ""]]
    Switch $method
        Case "GET"
            $aQueryData = DiscernQuery2array($urlReq) ; data is part of the URL
        Case Else
            $aQueryData = DiscernQuery2array($urlReq, $ContentBody, $ContentType) ; data is part of the body
    EndSwitch
    Return $ContentLength = $_BodyLen And StringLen($method)
EndFunc   ;==>DiscernData

Func DiscernGetFragment($urlReq)
    Local $a = StringSplit($urlReq, "#")
    If UBound($a) < 3 Then Return ""
    Return $a[$a[0]]
EndFunc   ;==>DiscernGetFragment

Func DiscernQuery2array(ByRef $urlReq, $ContentBody = Default, $ContentType = Default)
    Local $sContentPath, $sContentQuery
    Local $sFragment = DiscernGetFragment($urlReq)
    If $ContentBody = Default Then
        Local $aQueryData[1][3] = [[0, "", $sFragment]]
        Local $i = StringInStr($urlReq, "?")
        If Not $i Then Return $aQueryData
        $sContentPath = StringLeft($urlReq, $i - 1)
        $sContentQuery = StringTrimLeft($urlReq, $i)
    ElseIf StringInStr($ContentType, "application/json") Then
        Local $aQuery[2][3] = [[1, $urlReq, $sFragment], [$ContentBody, "", ""]]
        Return $aQuery ; is a json string, so better not StringSplit() it.
    Else
        $sContentPath = $urlReq
        $sContentQuery = $ContentBody
    EndIf
    Local $b, $n, $a = StringSplit($sContentQuery, "&")
    $i = 0
    Local $aQuery[UBound($a) + 1][3]
    For $n = 1 To UBound($a) - 1
        $b = StringSplit($a[$n], "=")
        $i += 1
        If $b[0] > 0 Then $aQuery[$i][0] = $b[1] ; name
        If $b[0] > 1 Then $aQuery[$i][1] = $b[2] ; value
    Next
    $aQuery[0][0] = $i ; this holds the name=value count
    $aQuery[0][1] = $sContentPath ; this holds the path ( minus the query if is a GET mathod )
    $aQuery[0][2] = $sFragment ; this would hold the fragment ( https://en.wikipedia.org/wiki/Fragment_identifier )
    ReDim $aQuery[$i + 1][3]
    Return $aQuery
EndFunc   ;==>DiscernQuery2array

... now you can see   
Edit: Added the "100-continue" to this simpler code too.

Edited February 12, 2020 by argumentum
better code

[kudrow]

Oh wow! Can’t wait to give this a try. Apparently I have the flu now so I am just resting. Hopefully I will get to try it tomorrow. I will report back with an update ASAP.

 

thanks again for taking the time to do this. I greatly appreciate it.

[argumentum]

I'll have a nicer one in an hour  

[argumentum]

( moved the code to here )

...this one has an "index" page that makes testing ( hopefully ) better   
It is also much faster  

Edited February 11, 2020 by argumentum
better code

[argumentum]

updated the above with a better JSON PUT example
and the query in an array.

Edited February 10, 2020 by argumentum

[argumentum]

..and, added GZIP support, not needed to answer your original question but may end up been useful, as it is in a function that better describes sending HTTP