HttpApi UDF - HTTP Server API

[dreamscd]

Thanks for such a quick reply👏

I don't know if you have figured out why the request sent by WinHTTP (DLL or COM) does not have the EntityCount flag, I have always wondered why it is, please let me know if you know, thank you!

Looking forward to your new version!!!

 

[TheXman]

If you read the WinHTTP Session Overview section called "Posting Data to the Server", it explains the 2 ways that data can be POST or PUT.  So when reading data chunks, one way sets the EntityCount and the other way relies on the "MORE DATA" flag.

[dreamscd]

9 hours ago, TheXman said:

If you read the WinHTTP Session Overview section called "Posting Data to the Server", it explains the 2 ways that data can be POST or PUT.  So when reading data chunks, one way sets the EntityCount and the other way relies on the "MORE DATA" flag.

Thanks for your reply, but I have used both ways to test.

You said about the WinHTTP Session Overview section called "Posting Data to the Server", I read it in detail, and also read the relevant content of the WinHttpSendRequest function. but I have tested the post request sent in these two ways, and the "EntityChunkCount" flag is always 0, I don't know if you noticed There is a comment after "_WinHttpSendRequest" in my test script. When I tested, I used two methods to send the request, and the results were the same. Let's talk about my test process:
Method 1: send immediately
 _WinHttpSendRequest($hRequest, "Content-Type: application/x-www-form-urlencoded", $sAdditionalData)
Use wireshark to capture packets and see that the post data is in the request headers, only one packet is sent
Method 2: WinHttpWriteData
_WinHttpSendRequest($hRequest, "Content-Type: application/x-www-form-urlencoded", Default, StringLen($sAdditionalData)) ;$sAdditionalData)
_WinHttpWriteData($hRequest, $sAdditionalData)
Also use wireshark to capture packets and see that the post data is not in the request headers, but in the latter packet, two packets are sent.
The same is that no matter which method is used to send the post request, the "EntityChunkCount" flag is always 0
The following is the test script and result of method 1

expandcollapse popup

; =============testgetPost.au3
#include "WinHttp.au3"

$sUserName = "SomeUserName"
$sEmail = "some.email@something.com"
$sDomain = "http://127.0.0.1:9000"
$sPage = "/a3server/posttest"

$sAdditionalData = "name=" & $sUserName & "&email=" & $sEmail

$hOpen = _WinHttpOpen()
If @error Then MsgBox(0, "open", "open error")

$hConnect = _WinHttpConnect($hOpen, $sDomain)
If @error Then MsgBox(0, "Connect", "Connect error")

$hRequest = _WinHttpOpenRequest($hConnect, "POST", $sPage, Default, "http://192.168.1.196:9000/a3server/posttest", "*/*")
If @error Then MsgBox(0, "OpenRequest", "OpenRequest error")

_WinHttpSendRequest($hRequest, "Content-Type: application/x-www-form-urlencoded", $sAdditionalData)
If @error Then MsgBox(0, "SendRequest", "SendRequest error:" & @error)

;_WinHttpWriteData($hRequest, $sAdditionalData)

_WinHttpReceiveResponse($hRequest)
If @error Then MsgBox(0, "ReceiveResponse", "ReceiveResponse error")

If _WinHttpQueryDataAvailable($hRequest) Then
    Global $sHeader = _WinHttpQueryHeaders($hRequest)
    MsgBox(0, "header", $sHeader & @CRLF)

EndIf

Dim $sReturned
If _WinHttpQueryDataAvailable($hRequest) Then
    Do
        $sReturned &= _WinHttpReadData($hRequest)
    Until @error
EndIf
_WinHttpCloseHandle($hRequest)
_WinHttpCloseHandle($hConnect)
_WinHttpCloseHandle($hOpen)

MsgBox(4096, "return", $sReturned)
ClipPut($sReturned)
ConsoleWrite($sReturned & @CRLF)

I added a "msgbox" before "_WinHttpSendRequest" of WinHttp UDF to view the call parameters of dllcall, and the result is as follows

;========WinHttp.au3
Func _WinHttpSendRequest($hRequest, $sHeaders = Default, $vOptional = Default, $iTotalLength = Default, $iContext = Default)
    __WinHttpDefault($sHeaders, $WINHTTP_NO_ADDITIONAL_HEADERS)
    __WinHttpDefault($vOptional, $WINHTTP_NO_REQUEST_DATA)
    __WinHttpDefault($iTotalLength, 0)
    __WinHttpDefault($iContext, 0)
    Local $pOptional = 0, $iOptionalLength = 0
    If @NumParams > 2 Then
        Local $tOptional
        $iOptionalLength = BinaryLen($vOptional)
        $tOptional = DllStructCreate("byte[" & $iOptionalLength & "]")
        If $iOptionalLength Then $pOptional = DllStructGetPtr($tOptional)
        DllStructSetData($tOptional, 1, $vOptional)
    EndIf
    If Not $iTotalLength Or $iTotalLength < $iOptionalLength Then $iTotalLength += $iOptionalLength
    MsgBox(0,"DllCall","WinHttpSendRequest:"&@crlf& _
            "hRequest:"&@TAB&@TAB& $hRequest&@crlf& _
            "lpszHeaders:"&@TAB& $sHeaders&@crlf& _
            "dwHeadersLength:"&@TAB& 0&@crlf& _
            "lpOptional:"&@TAB& $pOptional&@crlf& _
            "dwOptionalLength:"&@TAB& $iOptionalLength&@crlf& _
            "dwTotalLength:"&@TAB& $iTotalLength&@crlf& _
            "dwContext:"&@TAB& $iContext)
    Local $aCall = DllCall($hWINHTTPDLL__WINHTTP, "bool", "WinHttpSendRequest", _
            "handle", $hRequest, _
            "wstr", $sHeaders, _
            "dword", 0, _
            "ptr", $pOptional, _
            "dword", $iOptionalLength, _
            "dword", $iTotalLength, _
            "dword_ptr", $iContext)
    If @error Or Not $aCall[0] Then Return SetError(1, 0, 0)
    Return 1
EndFunc   ;==>_WinHttpSendRequest

Hello from the AutoIt HTTP Server.  POST request received!<br><br><pre>POST Request Info:
Request struct size = 4960
Bytes received      = 302
Full URL            = http://127.0.0.1:9000/a3server/posttest
Host                = 127.0.0.1:9000
AbsPath             = /a3server/posttest
QueryString         = 
Verb                = 6
EntityChunkCount    = 0

Request Headers:
Accept: */*
Connection: Keep-Alive
Content-Length: 48
Content-Type: application/x-www-form-urlencoded
Host: 127.0.0.1:9000
Referer: http://192.168.1.196:9000/a3server/posttest
User-Agent: Mozilla/5.0 (Windows NT 10.0) WinHttp/1.6.4.2 (WinHTTP/5.1) like Gecko
</pre>

Why is the value of EntityChunkCount not set, this is bothering me😵

[TheXman]

In my scripts, I rarely use the WinHTTP UDF (or the WinHTTP DLL API's in general) to send HTTP requests.  If I use WinHTTP at all, it is usually through the COM interface.  So if you really want to get answers related to the WinHTTP UDF, I would suggest you either post your questions in the WinHTTP UDF topic or create a new topic with your questions.  The only time I usually dig deeply into the internals of WinHTTP is when I'm trouble shooting an issue with HTTPAPI or looking into adding additional functionality to HTTPAPI.  Now that you've helped resolve a problem that I didn't even now existed with retrieving POST & PUT request body data , unfortunately, I don't have much desire to continue digging into the WinHTTP API's as it relates to sending requests.  If you have further questions about HTTPAPI, I would be more than happy to try to answer them.

I have finished adding your changes to the HTTPAPI UDF and to the example script.  I didn't need to change much at all.  I should have the new version uploaded sometime tomorrow afternoon or early evening after I have had a chance to do a little more testing.

Edited June 4, 2022 by TheXman

[dreamscd]

I'm glad you can reply to my poor English. OK, I'll get to know it another way.

[TheXman]

What's New in Version v1.2.1

• Added _HTTPAPI_HttpReceiveRequestEntityBody(), which was authored by @dreamscd.  The function returns additional request body entities.  This is required when POST and PUT requests have body data but the EntityChunkCount is not used.  i.e. When sending POST requests using WinHttp.
   

• The process_post_request function, in the example script, was modified to include the processing of additional request body entities, if they exist.  Thanks @dreamscd.
   
• Corrected an issue in _HTTPAPI_GetKnownRequestHeaderValue(). It was reading header value as wchar, when it is actually stored as char.
   
• Corrected an issue in _HTTPAPI_HttpSendHttpResponse().  Response status text was not getting set properly.
   
• Corrected a function header.  It listed a parameter that was not being used.

Edited June 4, 2022 by TheXman
Found and fixed a small issue after releasing v1.2.0. Updated and released v1.2.1.

[noellarkin]

How well does this server do when it comes to concurrent access? Say 20-30 users accessing it at the same time.

[TheXman]

I try not to answer such hypothetical questions or to make assumptions based on little to no information.  There's a big difference between 20-30 concurrent connections that are requesting information and 20-30 concurrent connections that are being served streaming media.  Regardless of the use case, if you are running an HTTP server on a woefully under-resourced PC and/or network, then obviously that would have an effect on the HTTP server's performance also.  Given that you've provided absolutely NO details about your intended use case, the hardware it would be running on, and the fact that I have no idea how proficient a coder you are, the best answer is to try it and see.  Even if you had provided more details, the answer would probably have been the same. 

This UDF is just a wrapper for some of the Microsoft's HTTP Server API's -- enough to provide general HTTP(s) request/response processing.  If you have specific questions about Microsoft's HTTPAPI and its features, functionality, implementation, and/or limitations, then I suggest you start looking for those answers at the source, by reading the documentation on Microsoft's website.  If/when you decide to use this UDF, and have specific questions, comments, or concerns related to the UDF itself, feel free to ask or comment.

Edited January 8, 2023 by TheXman

[noellarkin]

I'm sorry, I ought to have been more specific.
20-30 concurrent connections that are all on LAN, making HTTP requests to access textual data, JSON payloads. No streaming media. The server will run on a laptop I've repurposed into acting as a local storage (ie probably comes under "woefully under-resourced"). i3, Windows 7, 4GB RAM, SSD. I've used XAMPP server on this system with no issues.
I'll read up more on MS HTTPAPI and test out the UDF and see what happens. Thanks.

[TheXman]

If XAMPP is running on your laptop without any issues and currently providing responses to the requests that you've described, then HTTPAPI Server shouldn't have any issues handling the load either.  HTTAPI Server is much more lightweight than Apache.  Lightweight, in this context, means not as feature-rich or configurable, but very fast and efficient at low-level HTTP protocol processing. 😉  You can think of Microsoft's HTTPAPI Server as a subset of the Apache part of XAMPP.  It just provides HTTP(s) protocol services, no server-side scripting engines, no presentation layer, just HTTP connection, queuing, communications, and request/response handling.  It is very low-level.  Any processing of the request and response content is handled by the implementer.  

I would suggest taking a look at, and running, the example provided with the UDF to get a feel for the code necessary to process different types of requests and responses.  The closest example to the use case that you described would would probably be the "ipconfig" request.  When that request is received, it runs an ipconfig command locally on the server, formats a response from the output of the command, and sends the response back to the requestor.  So in your case, instead of running ipconfig, it could run whatever process is needed to gather the information for the response, create your JSON payload, and send that response back to the requestor.

Compared to other UDF's out there, using HTTPAPI UDF has its advantages and disadvantages.  One of the major advantages is that it is a fully compliant HTTP server including the ability to handle HTTPS connections.  One of the major disadvantages, if you see it that way, is that it works at a much lower level and requires a deeper knowledge of HTTP and coding expertise, to implement solutions.  Of course, I don't see lower level programmability as a disadvantage, but I can see how some may.  Other UDF's, such as AutoIt-API-WS, are much easier to implement and work with because the implementer is shielded from the lower level processing.  To be honest, unless one needs functionality or performance that some of the other HTTP Server UDF's can't provide, I would suggest using one of them.  However, for techies, those that appreciate the lower level of control, or those that have specific needs that can't be met by other UDF's, this UDF may be useful. 

Edited January 8, 2023 by TheXman

[Mr.Furious]

This is excellent, Thank you.

How would I go about getting this to run a program on the PC after receiving a get command/link?

[TheXman]

Run the example script that comes with the UDF.  The "ipconfig" link is probably the closest match.  When that specific URL is received by the HTTP server, it executes an ipconfig command on the server and sends back the formatted display of the output.  To see how it works, look at the code in the example script.

The 2nd screenshot HERE shows an example of the URL and its output.

 

 

Edited February 23, 2023 by TheXman

[Mr.Furious]

Excellent this is very helpful and appreciated. I briefly browsed the example script last night and read through all the forum comments but it was almost 3am. Thank you 

[TheXman]

My pleasure.  I hope you find it useful.  🙂

[Mr.Furious]

I was successful at getting a program to run and I am able to send commands from my SmartThings hub. It's very customizable, I am super excited.
I changed the IP from the default IP so that I can communicate with the server from other devices. In doing so I got an error regarding permission to register the URL. I figured out that running as administrator resolved that issue. However, the Windows 10 UAC shows up every time the server is launched. A bit irritating and I would like it to have it start at Windows login. I was able to get around this UAC behavior by using Task Scheduler. I am wondering if there is a better way or something I am not doing?

[TheXman]

Nice job getting your custom HTTPAPI server up and running so quickly!     

Discussions of ways to avoid the UAC prompt are outside the scope of this HTTPAPI topic and, in general, have been off limits in the forum for quite a while.  However, there was a time when it was discussed freely in the forum and, although I haven't searched for them, I think those topics still exist.  😉   Given that you want to launch your HTTPAPI server at login, the Task Scheduler would seem to be one of your best options anyway.

🙂

[sylremo]

Hello there, @TheXman. Much appreciation for all your efforts in maintaining this UDF! 

Last year, I utilized your UDF in my project which worked flawlessly, except for one specific feature - the one that I desired greatly but lacked sufficient research/knowledge to implement: Handling requests asynchronously.

It's mid-2023 and I decided to give it another shot 😅.

I'm aware that your UDF makes use of MS's HTTP APIs, rather than AutoIt's TCP protocols. So I think there's a good chance that requests can be handled simultaneously by utilizing child processes.

The main process, upon receiving requests, will subsequently transfer the required data to child processes for further handling.

==> In other words, main process is solely responsible for receiving requests, while all the actual work, which includes sending responses back, is handled by child processes.

What's your take on my opinion? If possible, it would be great if you can provide some good practices.

Regards

Edited May 5, 2023 by sylremo
clarify

[TheXman]

@sylremo

Thanks, I'm happy to hear that you found the HTTPAPI UDF useful and that what was included worked flawlessly for you. 

To be able to answer your question with any sort of specificity, I would need to know more about the specific use-case and the problem(s) you're trying to solve by implementing asynchronous processing.  Are you trying to handle more requests within a given time frame?  Are you trying to create a more distributed architecture?  Are you trying to do some sort of load balancing?  Are most of the requests that you are trying to handle fire-and-forget requests that are not expecting any sort of response?  Unfortunately, there's not enough detail for me to give any real opinions or advice of how I might implement a solution.  Is a workable solution possible?  Probably, but given that AutoIt's interpreter doesn't appear to have been written to internally handle scripts that require general asynchronous processing (other than timer-based events), the solution probably won't be as simple and efficient as it would be in other languages.  Although AutoIt is quite powerful, we need to keep in mind that it is, at it's core, an interpreted scripting language/engine built for automation, not application development.

Edited May 5, 2023 by TheXman

[sylremo]

20 hours ago, TheXman said:

I would need to know more about the specific use-case and the problem(s) you're trying to solve by implementing asynchronous processing.

My goal is to process more requests at the same time, thus reducing the response delay at each end.

 

20 hours ago, TheXman said:

Are most of the requests that you are trying to handle fire-and-forget requests that are not expecting any sort of response?

All incoming requests are expected to wait for a response.

 

20 hours ago, TheXman said:

AutoIt's interpreter doesn't appear to have been written to internally handle scripts that require general asynchronous processing

Maybe you didn't quite get what I meant. I was mentioning starting another script that will produce the response on its own without making the main controller (the server) wait.
The request orders are still being respected, which come first will be processed first, but are not guaranteed to finish before the latter ones.
This particular image from Architecture (HTTP Server API) - Win32 apps | Microsoft Learn would best describe this scenario:

 

Quote

The named request queue feature of the HTTP Server version 2.0 API allows multiple worker processes to receive requests on a request queue.

After skimming through Named Request Queue - Win32 apps | Microsoft Learn, I think my proposal is definitely doable in AutoIt. With the help of your existing UDF, I'll start implementing this and will let you know if I make any progress.

 

20 hours ago, TheXman said:

Although AutoIt is quite powerful, we need to keep in mind that it is, at it's core, an interpreted scripting language/engine built for automation, not application development.

Yep, the language itself is not scalable, at least to my limited knowledge. But if I believe that I can solve a specific problem in AutoIt regardless of its limitation in speed and flexibility, I would certainly give it a go.

[sylremo]

This is much simpler than I thought It would be 😅.

In the main controller, we'll create the queue with a name.

$aResult = DllCall($__HTTPAPI_ghHttpApiDll, "int", "HttpCreateRequestQueue", _
                                            "struct",  $__HTTPAPI_HTTPAPI_VERSION_2, _
                                            "wstr",    'main', _
                                            "ptr",     Null, _
                                            "ulong",   0, _
                                            "handle*", Null )

in the child process, we can easily access the existing queue by applying the HTTP_CREATE_REQUEST_QUEUE_FLAG_OPEN_EXISTING flag, which I believe is 0x1.
 

$aResult = DllCall($__HTTPAPI_ghHttpApiDll, "int", "HttpCreateRequestQueue", _
                                            "struct", $__HTTPAPI_HTTPAPI_VERSION_2, _
                                            "wstr", 'main', _
                                            "ptr", Null, _
                                            "ulong", 0x1, _
                                            "handle*", Null)

Thanks again for providing the community with such an awesome UDF!