WinHTTP functions

Flieder · July 8, 2019

Hello everyone!

Please excuse my probably dumb question and in case this is more suited for a seperate thread please let me know. I've been trying to solve this problem for over a week now and I'm at a point where I'm getting really frustrated. So any help, links, hints are appreciated!

It started with this small script that should help me populate our ecommerce database. It scrapes data from the manufacturers website, then looks for this product in the online shops of our distributors. It's necessary to be logged in to see details like product prices and stock - information we also maintain in our database.

So far my script is working with Internet Explorer. However, using IE has downsides: It's pretty slow and - worse - sometimes I don't stay logged in for whatever reason. This also happens when browsing manually.

So I thought it might be a good idea to use WinHttp-UDF instead, because it's more straightforward and lightweight. However, I'm having a hard time trying to figure out how I can request a product page after logging in with _WinHttpSimpleFormFill().

Here is what I managed to find out on my own:

#include <WinHttp.au3>
#include <String.au3>
#include <Array.au3>

Global Const $g_sZitecUser = "myUsername", $g_sZitecPass = "myPassword", $g_sZitecDomain = "www.zitec-shop.com/de/rillenkugellager-63/p-G1112003141"

Call("LogIn", $g_sZitecUser, $g_sZitecPass)

Func LogIn($sUser, $sPass)
  Local $sFileHTM = @ScriptDir & "\Form.htm"
  ; Open session
  Local $hOpen = _WinHttpOpen()
  ; Connect
  Local $hConnect = _WinHttpConnect($hOpen, "https://www.zitec-shop.com")
  ; Fill in form
  Local $sRead = _WinHttpSimpleFormFill($hConnect, "/de/login", "loginForm", _
        "name:j_username", $g_sZitecUser, _
        "name:j_password", $g_sZitecPass, _
        "type:submit", 1)
  ; This returns the frontpage with me logged in. Great! There is also a CSRF-Token within the response. So I put it in a variable
  If $sRead Then
    ; Get CSRF-Token
    If StringRegExp($sRead, "CSRFToken = '" & "((?:\w+?\-+?)*(?:\w+))" & "';?") Then
      Local $aCSRF[] = StringRegExp($sRead, "CSRFToken = '" & "((?:\w+?\-+?)*(?:\w+))" & "';?", 1)
    Else
      Msgbox(0, "Fehlermeldung", "Es konnte kein CSRF-Token gefunden werden.")
      ; Close handles
      _WinHttpCloseHandle($hConnect)
      _WinHttpCloseHandle($hOpen)
      Exit
    EndIf
  Else
    Msgbox(0, "Fehlermeldung", "Fehlercode: " & @error)
    ; Close handles
    _WinHttpCloseHandle($hConnect)
    _WinHttpCloseHandle($hOpen)
    Exit
  EndIf

  ; Now I want to request a product page, but even though I receive HTML-Code that implies I am still logged in (username is visible), prices and stock information are still hidden.
  ; How do you request another page and send credentials & token correctly?
  ; My attempt at a request
  Local $sURL = "/de/rillenkugellager-63/p-G1112003141"
  Local $sRequest = _WinHttpSimpleSSLRequest($hConnect, Default, $sURL, Default, "Authorization: Token " & $aCSRF[0]) ; "POST" gives me a 405 error

  If @error Then
    Msgbox(0, "Fehlermeldung", "Fehlercode: " & @error)
    ; Close handles
    _WinHttpCloseHandle($hConnect)
    _WinHttpCloseHandle($hOpen)
    Exit
  Else
    Msgbox(0, "Die Website hat geantwortet", "Die Website hat geantwortet!")
    $hFileHTM = FileOpen($sFileHTM, 2)
    FileWrite($hFileHTM, $sRequest)
    FileClose($hFileHTM)
    ShellExecuteWait($sFileHTM)
  EndIf

  ; Close handles
  _WinHttpCloseHandle($hConnect)
  _WinHttpCloseHandle($hOpen)
EndFunc ;==> LogIn

I'm quite a newbie. What I don't understand is: How do I figure out how to phrase the content of a header in a post? Like, is it always "username=user&password=pass&token=token" or something else?

Thanks for reading,

Flieder

[Edit]

After staying away from this problem for a few days, I found out that everything is working as intended. I don't have to extract and add the CSRF-Token to my request. The cookie returned with _WinHttpSimpleFormFill() includes every information needed and is stored in the cache as long as the Internethandle stays open.

My problem seems to stem from the getPrice.json! I have no idea how to solve this yet, but I wanted to keep everyone updated who stumbles across this post and faces similiar issues.

Kind regards,

Flieder

And before I forget: Thank you, trancexx and ProgAndy, for creating this UDF. Your work leads to people like me, with little to no network understanding, getting interested in subjects they never would have thought about on their own

Edited July 12, 2019 by Flieder

kaximbeider · August 12, 2019

Hello everybody!

I’m getting stuck with deflate response from a Http request.

I have used _WinHttpSetOption for enabling but no success. Here is my function:

Func HTTP_PUT ($IP, $URI, ByRef $iStatusCode, $sPostData, $type=-1, $QueryData=True, $verb="PUT")

Local $hOpen = _WinHttpOpen()
_WinHttpSetTimeouts($hOpen, 10000)
Local $hConnect = _WinHttpConnect($hOpen, "http://"&$IP)
Local $hRequest = _WinHttpOpenRequest($hConnect, $verb, $URI)
Local $user,$password
Local $sChunk, $sOut

    Switch $type
        Case 0; setup.html credentials
            $user=$SystemCredentials[0]
            $password=$SystemCredentials[1]
        Case 1 ;Client credentials
            $user=$SystemCredentials[2]
            $password=$SystemCredentials[3]
    EndSwitch

                _WinHttpSetOption($hRequest, $WINHTTP_OPTION_DECOMPRESSION, $WINHTTP_DECOMPRESSION_FLAG_DEFLATE)

                ; Send the request
                _WinHttpSendRequest($hRequest, Default, Default, StringLen($sPostData))

                _WinHttpWriteData($hRequest, $sPostData)

                ; Wait for the response
                _WinHttpReceiveResponse($hRequest)

                Sleep(100)

                If $QueryData==True Then

                    ; Check if there is any data available and read if yes

                    If _WinHttpQueryDataAvailable($hRequest) Then
                        While 1
                            $sChunk = _WinHttpReadData($hRequest)
                            If @error Then ExitLoop
                            $sOut &= $sChunk
                        WEnd
                    EndIf

                EndIf

                ; Query status code
                $iStatusCode = _WinHttpQueryHeaders($hRequest, $WINHTTP_QUERY_STATUS_CODE)


                ; Check status code
                If $iStatusCode == $HTTP_STATUS_DENIED Or $iStatusCode == $HTTP_STATUS_PROXY_AUTH_REQ Then
                    $sOut=""
                    $sChunk=""
                    ; Query Authorization scheme
                    Local $iSupportedSchemes, $iFirstScheme, $iAuthTarget


                    If _WinHttpQueryAuthSchemes($hRequest, $iSupportedSchemes, $iFirstScheme, $iAuthTarget) Then
                        ; Set passed credentials

                        _WinHttpSetCredentials($hRequest, $iAuthTarget, $iFirstScheme, $user, $password)


                        ; Send request again now

                           _WinHttpSendRequest($hRequest, Default, Default, StringLen($sPostData))

                            _WinHttpWriteData($hRequest, $sPostData)



                        ; And wait for the response again
                        _WinHttpReceiveResponse($hRequest)

                        Sleep (100)
                        $iStatusCode = _WinHttpQueryHeaders($hRequest, $WINHTTP_QUERY_STATUS_CODE)


                    EndIf

                    If $QueryData==True Then


                    ; Check if there is any data available and read if yes

                        If _WinHttpQueryDataAvailable($hRequest) Then
                            While 1
                                $sChunk = _WinHttpReadData($hRequest)
                                If @error Then ExitLoop
                                $sOut &= $sChunk
                            WEnd
                        EndIf


                    EndIf

                EndIf


    ; Close handles
    _WinHttpCloseHandle($hRequest)
    _WinHttpCloseHandle($hConnect)
    _WinHttpCloseHandle($hOpen)

Return $sOut

EndFunc

Thanks in advance.

Edited August 14, 2019 by kaximbeider

trancexx · August 17, 2019

On 8/12/2019 at 4:52 PM, kaximbeider said:

Hello everybody!

I’m getting stuck with deflate response from a Http request.

I have used _WinHttpSetOption for enabling but no success. Here is my function:

Func HTTP_PUT ($IP, $URI, ByRef $iStatusCode, $sPostData, $type=-1, $QueryData=True, $verb="PUT")

Local $hOpen = _WinHttpOpen()
_WinHttpSetTimeouts($hOpen, 10000)
Local $hConnect = _WinHttpConnect($hOpen, "http://"&$IP)
Local $hRequest = _WinHttpOpenRequest($hConnect, $verb, $URI)
Local $user,$password
Local $sChunk, $sOut

    Switch $type
        Case 0; setup.html credentials
            $user=$SystemCredentials[0]
            $password=$SystemCredentials[1]
        Case 1 ;Client credentials
            $user=$SystemCredentials[2]
            $password=$SystemCredentials[3]
    EndSwitch

                _WinHttpSetOption($hRequest, $WINHTTP_OPTION_DECOMPRESSION, $WINHTTP_DECOMPRESSION_FLAG_DEFLATE)

                ; Send the request
                _WinHttpSendRequest($hRequest, Default, Default, StringLen($sPostData))

                _WinHttpWriteData($hRequest, $sPostData)

                ; Wait for the response
                _WinHttpReceiveResponse($hRequest)

                Sleep(100)

                If $QueryData==True Then

                    ; Check if there is any data available and read if yes

                    If _WinHttpQueryDataAvailable($hRequest) Then
                        While 1
                            $sChunk = _WinHttpReadData($hRequest)
                            If @error Then ExitLoop
                            $sOut &= $sChunk
                        WEnd
                    EndIf

                EndIf

                ; Query status code
                $iStatusCode = _WinHttpQueryHeaders($hRequest, $WINHTTP_QUERY_STATUS_CODE)


                ; Check status code
                If $iStatusCode == $HTTP_STATUS_DENIED Or $iStatusCode == $HTTP_STATUS_PROXY_AUTH_REQ Then
                    $sOut=""
                    $sChunk=""
                    ; Query Authorization scheme
                    Local $iSupportedSchemes, $iFirstScheme, $iAuthTarget


                    If _WinHttpQueryAuthSchemes($hRequest, $iSupportedSchemes, $iFirstScheme, $iAuthTarget) Then
                        ; Set passed credentials

                        _WinHttpSetCredentials($hRequest, $iAuthTarget, $iFirstScheme, $user, $password)


                        ; Send request again now

                           _WinHttpSendRequest($hRequest, Default, Default, StringLen($sPostData))

                            _WinHttpWriteData($hRequest, $sPostData)



                        ; And wait for the response again
                        _WinHttpReceiveResponse($hRequest)

                        Sleep (100)
                        $iStatusCode = _WinHttpQueryHeaders($hRequest, $WINHTTP_QUERY_STATUS_CODE)


                    EndIf

                    If $QueryData==True Then


                    ; Check if there is any data available and read if yes

                        If _WinHttpQueryDataAvailable($hRequest) Then
                            While 1
                                $sChunk = _WinHttpReadData($hRequest)
                                If @error Then ExitLoop
                                $sOut &= $sChunk
                            WEnd
                        EndIf


                    EndIf

                EndIf


    ; Close handles
    _WinHttpCloseHandle($hRequest)
    _WinHttpCloseHandle($hConnect)
    _WinHttpCloseHandle($hOpen)

Return $sOut

EndFunc

Thanks in advance.

I believe that the data served by the server is compressed by deflate inside a zlib stream (RFC1951 + RFC1950), and WinHttp probably works with only former. When you find server that delivers raw deflate (RFC1951) this can be confirmed.

Sorry if I'm no help.

trancexx · August 17, 2019

Yes, that appears to be the issue. So, only raw deflate works. For example bing.com will deliver data in raw deflate form if $WINHTTP_DECOMPRESSION_FLAG_DEFLATE is used and that will be processed correctly. However processing postman-echo.com/deflate will give error.

kaximbeider · August 19, 2019

Brilliant! Thanks for the response.

I have made some testing with curl and it was working. I was able to read the response. But as I understood I cannot do anything else with winhttp?

Do I need an extra UDF for using zlib? Or do you have something that I could test?

trancexx · August 19, 2019

10 hours ago, kaximbeider said:

Brilliant! Thanks for the response.

I have made some testing with curl and it was working. I was able to read the response. But as I understood I cannot do anything else with winhttp?

Do I need an extra UDF for using zlib? Or do you have something that I could test?

Out of curiosity, why would you force deflate response and not go with gzip?

kaximbeider · August 21, 2019

The good point is the server who is responding as deflate.

I have tried to add on request headers to respond as plain text but without success.

I’m not an expert about http requests, how can I use gzip with winhttp?

trancexx · August 22, 2019

18 hours ago, kaximbeider said:

The good point is the server who is responding as deflate.

I have tried to add on request headers to respond as plain text but without success.

I’m not an expert about http requests, how can I use gzip with winhttp?

Instead of $WINHTTP_DECOMPRESSION_FLAG_DEFLATE use $WINHTTP_DECOMPRESSION_FLAG_GZIP and if server accepts your wish that'd be all.

kaximbeider · August 22, 2019

Ok! I will try that but I assume that the server will answer anyway as deflate.

fingers crossed! And thanks for your help. It is very appreciate.

kaximbeider · September 2, 2019

As I expected, server still answering as a deflated compressed.

It was a good try...

JoeBar · November 7, 2019

Hi, thank you for your nice UDF, i have a problem.

Here is the help in the SDK page of my app :

Quote
To request access tokens, an application must make a POST request with the following multipart form data to the token URI: grant_type=client_credentials

The application must pass basic HTTP auth credentials using the client_id as the user and client_secret as the password.

Example in curl :
curl -u {ClientID}:{ClientSct} -d grant_type=client_credentials https:/url/domain

I tried with curl, of course it's working.

I'm trying to do the same request in Autoit and my current "don't working" request is :

Global $hRequestSSL = _WinHttpSimpleSSLRequest($hConnect, "POST", $sDomain, Default, "grant_type=client_credentials", Default, Default, Default, $sClientID, $sClientSct)

I don't know which parameter to change to put grant_type=client_credentials (it's the only body of the request) or if i need to switch function for another one.

Thanks for your time !

Edit : I found how to proceed :

Global $hRequestSSL = _WinHttpSimpleSSLRequest($hConnect, "POST", $sDomain, Default, "client_id=" & $sClientID & "&" & "client_secret=" & $sClientSct & "&" & "grant_type=client_credentials")

Edited November 8, 2019 by JoeBar

seadoggie01 · November 15, 2019

I'm unable to get the example on the first page to work. _WinHttpSendRequest returns @error = 1. I split the function out and the DLL call doesn't error out, but it fails to return an array. I've tried changing to use TLS 1.2 and SSL 2.0 through the registry and using ObjCreate("winhttp.WinHttpRequest.5.1") and using the object instead, but it fails on .Send saying that I sent an incorrect parameter. I've tried running as an admin in case it's related to my work computer, but nothing gets through. Does anyone know what I can try next?

JoeBar · December 3, 2019

There is a problem with _WinHttpSimpleFormFill_SetUploadCallback : The $iPercent when upload is completed remains at 99, so when updating a progressbar, it's freezed at 99% at the end of the operation.

Obliged to do this to fullfill the Progressbar :

If $iPercent = 100 Then
    GUICtrlSetData($hprogress, 100)

Is it a known bug ?

Edited December 3, 2019 by JoeBar

HamidZaeri · December 17, 2019

There is problem with "_WinHttpSimpleSSLRequest" function and Persian characters in data.

When I use Persian characters in data to POST, the function trim some character from the end of data!

And I should add some space to right hand side to be trimmed instead.

Try this data: "Hello پک و چیز Hello"

trancexx · December 18, 2019

23 hours ago, HamidZaeri said:

There is problem with "_WinHttpSimpleSSLRequest" function and Persian characters in data.

When I use Persian characters in data to POST, the function trim some character from the end of data!

And I should add some space to right hand side to be trimmed instead.

Try this data: "Hello پک و چیز Hello"

It's hard to figure out what user wants to send. It should be up to you to feed the function with data properly formatted. In your case this should work correctly:

$sPostData = "Hello پک و چیز Hello"
$sPostData = BinaryToString(StringToBinary($sPostData, 4))
;...

trancexx · December 18, 2019

On 12/3/2019 at 2:59 PM, JoeBar said:
There is a problem with _WinHttpSimpleFormFill_SetUploadCallback : The $iPercent when upload is completed remains at 99, so when updating a progressbar, it's freezed at 99% at the end of the operation.

Obliged to do this to fullfill the Progressbar :
If $iPercent = 100 Then
    GUICtrlSetData($hprogress, 100)
Is it a known bug ?

Could you write a reproducer for me to run?

JoeBar · December 18, 2019

On 12/18/2019 at 1:09 PM, trancexx said:

Could you write a reproducer for me to run?

#include "WinHTTP.au3"
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#Include <GuiButton.au3>
#include <ProgressConstants.au3>

Global $hGui = GUICreate("",600,315,-1,-1,-1,-1)
Global $hprogress = GUICtrlCreateProgress(20,289,197,20,-1,-1)
Global $Upload = GUICtrlCreateButton("Upload",445,20,132,50,-1,-1)

GUISetState(@SW_SHOW, $hGui)

While True
    Global $msg = GUIGetMsg()
    Switch $msg
        Case $Upload
            Upload()
        Case $GUI_EVENT_CLOSE
            Exit
    EndSwitch
WEnd

Func Upload()
$sAddress = "https://paratus.hr/software/testing/fileupload/" ; the address of the target (https or http, makes no difference - handled automatically)

; Select some file
$sFileToUpload = FileOpenDialog("Select file to upload...", "", "All Files (*)")
If Not $sFileToUpload Then Exit 5 ; check if the file is selected and exit if not

$sForm = _
        '<form action="' & $sAddress & '" method="post" enctype="multipart/form-data">' & _
        ' <input type="file" name="upload"/>' & _
        '</form>'

; Initialize and get session handle
$hOpen = _WinHttpOpen()

$hConnect = $sForm ; will pass form as string so this is for coding correctness because $hConnect goes in byref

; Register callback function
_WinHttpSimpleFormFill_SetUploadCallback(UploadCallback)

; Fill form
$sHTML = _WinHttpSimpleFormFill($hConnect, $hOpen, _
        Default, _
        "name:upload", $sFileToUpload)
; Collect error number
$iErr = @error

; Unregister callback function
_WinHttpSimpleFormFill_SetUploadCallback(0)

; Close handles
_WinHttpCloseHandle($hConnect)
_WinHttpCloseHandle($hOpen)

If $iErr Then
    MsgBox(4096, "Error", "Error number = " & $iErr)
Else
    ConsoleWrite($sHTML & @CRLF)
    MsgBox(4096, "Success", $sHTML)
EndIf
EndFunc

; Callback function. Update progress control
Func UploadCallback($iPercent)
    If $iPercent = 100 Then
        MsgBox(4096, "Done", "Complete")
    Else
        GUICtrlSetData($hprogress, $iPercent)
    EndIf
EndFunc   ;==>UploadCallback

To simplify, i took your modified example ( from Github) of the func to display the progress on a Progress.

The problem is solved when you remove the If statement in the callback function :

Func UploadCallback($iPercent)
        GUICtrlSetData($hprogress, $iPercent)
EndFunc   ;==>UploadCallback

-----> Ok i understood, it's the last GUICtrlSetData($hprogress, $iPercent) not done so the $iPercent is at 99 when we get out of the callback ...

~~It's your example that just have to be changed to not have the same problem 😊~~

Edited December 20, 2019 by JoeBar

HamidZaeri · December 19, 2019

19 hours ago, trancexx said:
It's hard to figure out what user wants to send. It should be up to you to feed the function with data properly formatted. In your case this should work correctly:
$sPostData = "Hello پک و چیز Hello"
$sPostData = BinaryToString(StringToBinary($sPostData, 4))
;...

I used that before but not worked,

but StringToBinary($sPostData, 4) did the job!

trancexx · December 19, 2019

23 hours ago, JoeBar said:

It's your example that just have to be changed to not have the same problem 😊

But that's not true. You can check my code again

JoeBar · December 20, 2019

13 hours ago, trancexx said:

But that's not true. You can check my code again

You're right, i dind't even see you set Progress at 100 😓

So my intervention is useless 😒

WinHTTP functions

Recommended Posts

Link to comment

Share on other sites

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

trancexx

jvanegmond

trancexx

Posted Images

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members