HttpApi UDF - HTTP Server API

[TheXman]

You're welcome, NRGZ28.  I'm glad that you were able to make use of it. 

[TheXman]

New version uploaded

• v1.1.2
  • Fixed an error in the $__HTTPAPI_gtagHTTP_BYTE_RANGE structure definition.  (unit64 -> uint64)

Edited January 20, 2021 by TheXman

[mrwilly]

I'm a bit late to the party, but this is awesome.

I'm probably a bit slow though - how do I respond with 302 Found, giving a new URL to redirect to? I don't see how I can add a "Location" header to _HTTPAPI_HttpSendHttpResponse in my response.

Many thanks!

[TheXman]

Thank you.  ☺️

You are correct that there is not a function in the UDF to set additional response headers.  That's only because I never had a reason to do it in the past.  I only used it to either remotely process a command on the server or to request text-based information.  I never had a reason to use it as a proxy of sorts, but it definitely could be made to do that.

Technically, to handle a redirect response, I would probably just add code to the _HTTPAPI_HttpSendHttpResponse() function that recognizes when a redirect $iStatusCode (301/302) is passed and add the additional "Location" header using the $__HTPPAPI_HttpHeaderLocation constant to set the header.  The target URL of the redirect could be passed in the  $sReason parameter of the _HTTPAPI_HttpSendHttpResponse() function.

Edited March 17, 2022 by TheXman

[mrwilly]

Thank you!

I did exactly what you suggested and it's working great.

I'm actually using this as a way to call a custom URI scheme via an http link (by redirecting to the custom URI), because many apps don't support custom URIs (e.g. MS Teams). Ideally I'd like a way to respond to a GET request with something that causes the web browser to just give up and close the tab (and my AutoIt code can call the custom URI scheme).

But failing that I think a redirect is probably the best I'll be able to do. It's just a shame that it leaves open a blank webpage in a web browser.

Thanks again!!

[TheXman]

You're welcome! 

 

[TheXman]

What's New in Version v1.1.4

• Corrected _HTTPAPI_HttpSendHttpResponse function header.
• Added the ability for _HTTPAPI_HttpSendHttpResponse() to recognize and process a redirect response.  This is achieved by setting $iStatusCode to 301 or 302 and setting $sReason to the target location URL. ( @mrwilly,  thanks for the suggestion )
• Added a redirect example to the example script.

Edited March 17, 2022 by TheXman

[mrwilly]

Brilliant, thank you so much!

[DonChunior]

11 hours ago, TheXman said:

What's New in Version v1.1.4

• Corrected _HTTPAPI_HttpSendHttpResponse function header.
• Added the ability for _HTTPAPI_HttpSendHttpResponse() to recognize and process a redirect response.  This is achieved by setting $iStatusCode to 301 or 302 and setting $sReason to the target location URL. ( @mrwilly,  thanks for the suggestion )
• Added a redirect example to the example script.

You forgot to increase the version number of the $__HTTPAPI_UDF_VERSION constant: It's still "1.1.3".

[TheXman]

Thanks @DonChunior.  I just corrected the $__HTTPAPI_UDF_VERSION constant and uploaded a new version of the UDF.

[sylremo]

Hi @TheXman, is there any way to access the request's body? 
I've noticed this line of code in example script but has no clue how to access the body...

$tRequest = DllStructCreate($__HTTPAPI_gtagHTTP_REQUEST_V2 & StringFormat("byte body[%i];", $__HTTPAPI_REQUEST_BODY_SIZE), $pRequest)

 

[sylremo]

Having absolute no experience in working with DLLs or any C/C++ documentations, it takes me a while to write my own function to retrieve the request body...

Func _HTTPAPI_GetRequestBody($tRequest)
    Local $tDataChunks = DllStructCreate($__HTTPAPI_gtagHTTP_DATA_CHUNK_FROM_MEMORY, $tRequest.pEntityChunks)

    If ($tRequest.EntityChunkCount < 1) Then Return ""

    Local $sBody = DllStructCreate(StringFormat('char data[%i]', $tDataChunks.pBufferLength), $tDataChunks.pBuffer).data

    Return $sBody
EndFunc

I've done a test of your UDF with others in this forum, i've noticed a huge different. When handling huge amount of requests concurrently, while other UDFs will eventually ignore some latter requests, yours really put these requests into queue and handle them one by one perfectly without any failures. 

In future versions, I hope to see a better implement of this. Adding customizations to response's header and auto parse url/query params (eg: ../user/:param1/action?key1=value1&key2=value2) would also be nice.

Great work btw!

Edited March 19, 2022 by sylremo

[TheXman]

On 3/18/2022 at 11:34 PM, sylremo said:

Hi @TheXman, is there any way to access the request's body? 

The example included with the UDF only processes GET requests.  Ordinarily, GET requests do not include a body.  The quick example file that I have attached below, recognizes and processes POST requests.  In the attached example, in the process_post_request() function, you will see how I parse the body into a variable and display the body of the POST request in the response and to the console.  The example is made to be run in the scite editor because it shows details about the request in the console window.

On 3/19/2022 at 4:15 AM, sylremo said:

yours really put these requests into queue and handle them one by one perfectly without any failures.

Yes, that is the way that Microsoft's HTTPAPI works.  You create URL groups and assign those URL groups to request queues.  Requests go into the queues and get processed accordingly.  Microsoft's HTTPAPI is a true, standards-based, web server.  My UDF gives a framework to implement some of its functionality using the included API wrappers.  One of the other nice things about using Microsoft's HTTPAPI as a web server is that it has the ability to handle secure communications (TLS) transparently.  In other words, other than binding a valid certificate to your specific ip:port (or hostname:port in Windows 10+), you don't need to do anything special in the code.  It just works.

On 3/19/2022 at 4:15 AM, sylremo said:

In future versions, I hope to see a better implement of this. Adding customizations to response's header and auto parse url/query params (eg: ../user/:param1/action?key1=value1&key2=value2) would also be nice.

The UDF was originally created for my personal use, not for public consumption.  I decided to make it fit AutoIt UDF standards and share it in case others might find it as useful as I do.  I have scripts that use the UDF to process RESTful GETs, POSTs, and all sort of requests.  Although I have had no plans to add functionality, I do take and implement features that are requested, if I think they are useful and would be widely used by others.  Of course, if someone finds a bug I will squash it as soon as possible.

There are examples of how to add headers to a response in the example script that comes with the UDF.  The parsing of a request path is in the example also.  There's not an example of processing RESTful parameters in the request path, but that is simple string manipulation.  However, I could see writing a generic function that parses the path and places the key/value parameters into a dictionary object or map.  In any case, I appreciate the suggestions

On 3/19/2022 at 4:15 AM, sylremo said:

Great work btw!

Thanks! 

example_http_server_get_post_requests.au3

Edited April 3, 2022 by TheXman

[TheXman]

@sylremo

Thanks again for the suggestion to have a function that just returns the request body, if it exists.  I will add that functionality to the UDF and publish a new version soon.

[TheXman]

What's New in Version v1.1.5

Released Just now

• Added _HTTPAPI_GetRequestBody().  As it implies, the function will return the request body.
  • Thanks @sylremo for the suggestion.
• Added sample POST request processing to the example script.
• Corrected some incomplete and/or erroneous function header info.

[dreamscd]

My English is not good.

about POST,  use "WinHttp.au3" to send a post Request, the "example_httpapi_web_server" don't got the post body, maybe can do this by “HttpReceiveRequestEntityBody”.

; example_httpapi_web_server.au3

;Func process_post_request($pRequest)

    If $tRequest.pEntityChunks > 0 Then
        $sResponse &= @CRLF & "Request EntityChunks Body:" & @CRLF
        $sResponse &= _HTTPAPI_GetRequestBody($tRequest) & @CRLF & @CRLF
    EndIf
    If BitAND( $tRequest.Flags , $__HTTPAPI_HTTP_REQUEST_FLAG_MORE_ENTITY_BODY_EXISTS ) Then
        $sResponse &= @CRLF & "Request Body:" & @CRLF
        $sResponse &= _HTTPAPI_HttpReceiveRequestEntityBody($ghRequestQueue, $tRequest.RequestId) & @CRLF & @CRLF
    EndIf

expandcollapse popup

; HttpApi.au3
;======================
;HTTP REQUEST FLAG
;======================
Global $__HTTPAPI_HTTP_REQUEST_FLAG_MORE_ENTITY_BODY_EXISTS = 1
Global $__HTTPAPI_HTTP_REQUEST_FLAG_IP_ROUTED = 2

; #FUNCTION# ====================================================================================================================
; Name ..........: _HTTPAPI_HttpReceiveRequestEntityBody
; Description ...: Returns the request body.
; Syntax ........: _HTTPAPI_HttpReceiveRequestEntityBody($hRequestQueue,$RequestId)
; Parameters ....: $hRequestQueue       A handle to the request queue to receive the request from.
;            : $RequestId           This value is returned in the RequestId member of the HTTP_REQUEST structure by a call to the HttpReceiveHttpRequest function.
; Return values .: Success:             A string containing the request body.
; Author ........: Dreamscd
; Related .......:
; Remarks .......:
; ===============================================================================================================================

Func _HTTPAPI_HttpReceiveRequestEntityBody($hRequestQueue,$RequestId)

    _DebugOut(@CRLF & "Function: HttpReceiveRequestEntityBody")
    
    Local $aResult = "", $sRequestBody = ""
    
    Local $EntityBufferLength = 2048

    Local $EntityBuffer = DllStructCreate( StringFormat("byte EntityBuffer[%i];", $EntityBufferLength) )

    Local $pEntityBuffer = DllStructGetPtr($EntityBuffer)

    While 1
        ;Call API function
        $aResult = DllCall($__HTTPAPI_ghHttpApiDll, "int", "HttpReceiveRequestEntityBody", _
                           "handle",  $hRequestQueue, _
                           "uint64",  $RequestId, _
                           "ulong",   0, _ 
                           "struct*", $pEntityBuffer, _
                           "ulong",   DllStructGetSize($EntityBuffer), _
                           "ulong*",  0, _
                           "ptr",     Null)
        If @error Then
            $__HTTPAPI_gsLastErrorMessage = "DllCall function failed."
            Return SetError(1, @error, "")
        EndIf
        
        ;Set last error message
        Switch $aResult[0]
            Case $__HTTPAPI_ERROR_NO_ERROR
                If $aResult[6] <> 0 Then    ;BytesRead
                    $sRequestBody &= _WinAPI_GetString($pEntityBuffer,False)
                EndIf
            Case $__HTTPAPI_ERROR_HANDLE_EOF
                If $aResult[6] <> 0 Then
                    $sRequestBody &= _WinAPI_GetString($pEntityBuffer,False)
                    ExitLoop
                Else
                    ExitLoop
                EndIf
                
            Case $__HTTPAPI_ERROR_INVALID_PARAMETER
                $__HTTPAPI_gsLastErrorMessage = "An invalid parameter was passed To the function."
                Return SetError(2, $aResult[0], "")
            Case $__HTTPAPI_ERROR_DLL_INIT_FAILED
                $__HTTPAPI_gsLastErrorMessage = "The calling application did not call HttpInitialize before calling this function."
                Return SetError(2, $aResult[0], "")
            Case Else
                $__HTTPAPI_gsLastErrorMessage = "Unrecognized error - check winerror.h for more information."
                Return SetError(2, $aResult[0], "")
                
        EndSwitch
        
    WEnd

    Return $sRequestBody
EndFunc

 

[TheXman]

10 hours ago, dreamscd said:

about POST,  use "WinHttp.au3" to send a post Request, the "example_httpapi_web_server" don't got the post body, maybe can do this by “HttpReceiveRequestEntityBody”.

Can you provide an example script showing the request that you sent using WinHTTP.au3 so that I can try to see why you didn't get the expected body of your POST request?

In the current version of the HTTPAPI UDF and as you've shown, there is already a function that returns the request body, it is named _HTTPAPI_GetRequestBody().  There is an example of its use in the example_httpapi_web_server.au3 that is provided with the HTTPAPI files.  It can be found  in the process_post_request() function.  When sending a POST request to the provided example web server, it sends a response with information about the POST request.

Below is an example of the response when you POST a request to the example web server script:

Hello from the AutoIt HTTP Server. POST request received!

POST Request Info:
Request struct size = 4960
Bytes received      = 552
Full URL            = http://127.0.0.1:9000/a3server?arg1=value1&arg2=value2&arg3=test%20string
Host                = 127.0.0.1:9000
AbsPath             = /a3server?arg1=value1&arg2=value2&arg3=test%20string
QueryString         = ?arg1=value1&arg2=value2&arg3=test%20string
Verb                = 6
EntityChunkCount    = 1

Request Headers:
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.5
Cache-Control: no-cache
Connection: keep-alive
Content-Length: 14
Content-Type: text/plain;charset=UTF-8
DNT: 1
Host: 127.0.0.1:9000
Origin: moz-extension://3efa9169-028a-4563-98ea-b81d14040721
Pragma: no-cache
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:100.0) Gecko/20100101 Firefox/100.0

Request Body:
This is a test

At the bottom of the output above, you can see the request's body, "This is a test".

 

As you can see above, the example DOES get the request body.  Are trying to say that you didn't get the complete body of your specific request or that there was an error trying to receive the request body?  If so, then there is a constant, $__HTTPAPI_REQUEST_BODY_SIZE, that can be increased if the buffer is too small to handle the request body in a single chunk or @error may have been set to say why it failed.  By default, the $__HTTPAPI_REQUEST_BODY_SIZE constant is set to 4K bytes.  I wrote the UDF that way because it met my needs and wasn't designed to process large files or chunks of data.  It was designed for me to remotely send relatively small requests and commands to carry out on my network when I was not sitting in front of one of my PC's.  But like I said above, I did give it the flexibility to handle larger requests (if necessary) by increasing the size of the request body buffer.

 

Looking at the function that you suggested, which uses a smaller chunk size and reads chunks until they have all been received, I think you are trying to propose a more robust & efficient way to read entity chunks, which is a very good & valid suggestion.     If I were to add the suggested logic, I'm not sure whether it would be better to make it a standalone function or if it would be better to add that logic to the existing function that receives the request, _HTTPAPI_HttpReceiveHttpRequest().  Of course, this assumes that I have correctly understood what you are trying to suggest.

 

Since you didn't provide a sample script or steps to recreate or reproduce your issue/point, I'm not sure that I have understood what is was you are trying to say.  I don't like making assumptions.  So if I didn't understand your point, maybe you can try to give a better explanation of what it is you are trying to suggest or trying to ask.  And if you do, please provide a way for me to see exactly what it is you want me to see or understand.

 

Edited June 1, 2022 by TheXman
Corrected name of function: process_post_request

[dreamscd]

Download httpapi_ v1.1.5. Zip. Use "example_httpapi_web_server.au3" to start the HTTP service. When I use "testgetpost.au3" to send a post request, the server returns

Entitychunkcount is always 0 and no body is returned

; ===========testgetpost.au3

expandcollapse popup

#include "WinHttp.au3"

$sUserName = "SomeUserName"
$sEmail = "some.email@something.com"
$sDomain = "http://127.0.0.1:9000"
$sPage = "/a3server/posttest"

$sAdditionalData = "name=" & $sUserName & "&email=" & $sEmail

$hOpen = _WinHttpOpen()
If @error Then MsgBox(0, "open", "open error")

$hConnect = _WinHttpConnect($hOpen, $sDomain)
If @error Then MsgBox(0, "Connect", "Connect error")

$hRequest = _WinHttpOpenRequest($hConnect, "POST", $sPage, Default, "http://192.168.1.196:9000/a3server/posttest", "*/*")
If @error Then MsgBox(0, "OpenRequest", "OpenRequest error")

_WinHttpSendRequest($hRequest, "Content-Type: application/x-www-form-urlencoded", Default, StringLen($sAdditionalData)) ;$sAdditionalData)
If @error Then MsgBox(0, "SendRequest", "SendRequest error:" & @error)

_WinHttpWriteData($hRequest, $sAdditionalData)

_WinHttpReceiveResponse($hRequest)
If @error Then MsgBox(0, "ReceiveResponse", "ReceiveResponse error")

If _WinHttpQueryDataAvailable($hRequest) Then
    Global $sHeader = _WinHttpQueryHeaders($hRequest)
    MsgBox(0, "header", $sHeader & @CRLF)

EndIf

Dim $sReturned
If _WinHttpQueryDataAvailable($hRequest) Then
    Do
        $sReturned &= _WinHttpReadData($hRequest)
    Until @error
EndIf
_WinHttpCloseHandle($hRequest)
_WinHttpCloseHandle($hConnect)
_WinHttpCloseHandle($hOpen)

MsgBox(4096, "return", $sReturned)
ClipPut($sReturned)
ConsoleWrite($sReturned & @CRLF)

; =============The following returned data

Hello from the AutoIt HTTP Server.  POST request received!<br><br><pre>POST Request Info:
Request struct size = 4960
Bytes received      = 302
Full URL            = http://127.0.0.1:9000/a3server/posttest
Host                = 127.0.0.1:9000
AbsPath             = /a3server/posttest
QueryString         = 
Verb                = 6
EntityChunkCount    = 0

Request Headers:
Accept: */*
Connection: Keep-Alive
Content-Length: 48
Content-Type: application/x-www-form-urlencoded
Host: 127.0.0.1:9000
Referer: http://192.168.1.196:9000/a3server/posttest
User-Agent: Mozilla/5.0 (Windows NT 10.0) WinHttp/1.6.4.2 (WinHTTP/5.1) like Gecko
</pre>

I don't know if I haven't done anything well. I use the post request sent by "winhttp.au3"，“Entitychunkcount” is always 0，I don't know why😵

Edited June 3, 2022 by dreamscd

[TheXman]

Thank you for providing a test script.  With your test script, I was able to recreate the issue.  I have been able to determine that there is a difference between the way WinHTTP (DLL or COM) sends POST requests and the way cURL or API testing tools like RESTer and Postman send POST requests.  When sending your request using an API testing tool like RESTer or Postman, or when sending the request using cURL, the request has an EntityCount of 1 and the body of the request is successfully processed.  However, for some yet unknown reason, when the request is sent using WinHTTP (DLL API's or the COM object), the request gets sent differently.  The EntityCount comes across as 0, which bypasses the processing of the body of the request (as you have pointed out).  I will need to take a deeper dive into the root differences and come up with a better way to identify and retrieve the content of the body of the request (regardless of how the request is sent). 

Examples using cURL & RESTer:

cURL command:  curl http://127.0.0.1:9000/a3server/posttest -d "name=Some Name&email=some.email@something.com"

Hello from the AutoIt HTTP Server.  POST request received!<br><br><pre>POST Request Info:
Request struct size = 4960
Bytes received      = 210
Full URL            = http://127.0.0.1:9000/a3server/posttest
Host                = 127.0.0.1:9000
AbsPath             = /a3server/posttest
QueryString         = 
Verb                = 6
EntityChunkCount    = 1

Request Headers:
Accept: */*
Content-Length: 45
Content-Type: application/x-www-form-urlencoded
Host: 127.0.0.1:9000
User-Agent: curl/7.83.1

Request Body:
name=Some Name&email=some.email@something.com

</pre>

 

Requests data can be sent data in several different forms (search "Global Data Chunk Structures/Unions" in the HTTPAPI UDF).  Currently, the _HTTPAPI_GetRequestBody() function only identifies and processes body content sent as Entities (EntityCount > 0).  I need to rework the _HTTPAPI_GetRequestBody() function and make it a little more generic/flexible in the way it identifies and retrieves body content.  I haven't look closely your proposed solution yet, but I will.  If it turns out to be a good solution, I'll definitely add it to the UDF lib.  However, I think the ultimate solution will most likely include the interrogation of the DataChunkType field.  It identifies in what form the data was sent and the structure needed to process it.  In any case, when I do come up with a solution, I will definitely give you credit for identifying and helping to resolve the issue. 

Thanks for bringing this to my attention.

 

Edited June 3, 2022 by TheXman

[TheXman]

I've had a chance to take a deeper dive into the issue and to look at your proposed solution.  I now see what it is you were trying to get me to see and understand.  Thanks to you, I  also have a better understanding of the differences between using WinHTTP (DLL or COM) and other tools like cURL and API testers, when sending a POST request.  You did a very nice job of making your proposed function follow the look & feel of the existing functions.  I may make a few minor changes to it before adding it to the HTTPAPI UDF lib, but I think your logic is sound (works well).  I will post an updated version of the HTTPAPI UDF lib with your suggestions soon. 

Thank you @dreamscd.

Edited June 3, 2022 by TheXman