WinHTTP functions

[ParoXsitiC]

Here's an example how to upload file using WinHttp.au3. I've added event monitoring to (obvious follows) monitor events.

expandcollapse popup

#AutoIt3Wrapper_AU3Check_Parameters=-d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6

#include "WinHttp.au3"

Opt("MustDeclareVars", 1)

;=====================================================================
; Page and Form info
; http://www.cs.tut.fi/~jkorpela/forms/file.html
Global Const $sAddress = "www.cs.tut.fi"
Global Const $sPage = "~jkorpela/forms/file.html"
Global Const $sFormId = "index:0" ;<- meaning first form on the page
;=====================================================================

; Register Callback function to monitor events
Global $hWINHTTP_STATUS_CALLBACK = DllCallbackRegister("__WINHTTP_STATUS_CALLBACK", "none", "handle;dword_ptr;dword;ptr;dword")

; Initialize and get session handle (impersonating Chrome browser)
Global $hOpen = _WinHttpOpen("Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/533.4 (KHTML, like Gecko) Chrome/5.0.375.127 Safari/533.4")

; Associate callback function with this handle
_WinHttpSetStatusCallback($hOpen, $hWINHTTP_STATUS_CALLBACK)

; Get connection handle
Global $hConnect = _WinHttpConnect($hOpen, $sAddress)
Global $sHTM = _WinHttpSimpleFormFill($hConnect, $sPage, $sFormId, _
        "name:datafile", @ScriptFullPath, _
        "txt", "Private dancer.")

ConsoleWrite(">Server response:" & @CRLF & $sHTM & @CRLF)

_WinHttpCloseHandle($hConnect)
_WinHttpCloseHandle($hOpen)



; Define callback function
Func __WINHTTP_STATUS_CALLBACK($hInternet, $iContext, $iInternetStatus, $pStatusInformation, $iStatusInformationLength)
    #forceref $hInternet, $iContext, $pStatusInformation, $iStatusInformationLength
    ConsoleWrite("!-> ")
    ; Interpret the status
    Local $sStatus
    Switch $iInternetStatus
        Case $WINHTTP_CALLBACK_STATUS_CLOSING_CONNECTION
            $sStatus = "Closing the connection to the server"
        Case $WINHTTP_CALLBACK_STATUS_CONNECTED_TO_SERVER
            $sStatus = "Successfully connected to the server."
        Case $WINHTTP_CALLBACK_STATUS_CONNECTING_TO_SERVER
            $sStatus = "Connecting to the server."
        Case $WINHTTP_CALLBACK_STATUS_CONNECTION_CLOSED
            $sStatus = "Successfully closed the connection to the server."
        Case $WINHTTP_CALLBACK_STATUS_DATA_AVAILABLE
            $sStatus = "Data is available to be retrieved with WinHttpReadData."
        Case $WINHTTP_CALLBACK_STATUS_HANDLE_CREATED
            $sStatus = "An HINTERNET handle has been created."
        Case $WINHTTP_CALLBACK_STATUS_HANDLE_CLOSING
            $sStatus = "This handle value has been terminated."
        Case $WINHTTP_CALLBACK_STATUS_HEADERS_AVAILABLE
            $sStatus = "The response header has been received and is available with WinHttpQueryHeaders."
        Case $WINHTTP_CALLBACK_STATUS_INTERMEDIATE_RESPONSE
            $sStatus = "Received an intermediate (100 level) status code message from the server."
        Case $WINHTTP_CALLBACK_STATUS_NAME_RESOLVED
            $sStatus = "Successfully found the IP address of the server."
        Case $WINHTTP_CALLBACK_STATUS_READ_COMPLETE
            $sStatus = "Data was successfully read from the server."
        Case $WINHTTP_CALLBACK_STATUS_RECEIVING_RESPONSE
            $sStatus = "Waiting for the server to respond to a request."
        Case $WINHTTP_CALLBACK_STATUS_REDIRECT
            $sStatus = "An HTTP request is about to automatically redirect the request."
        Case $WINHTTP_CALLBACK_STATUS_REQUEST_ERROR
            $sStatus = "An error occurred while sending an HTTP request."
        Case $WINHTTP_CALLBACK_STATUS_REQUEST_SENT
            $sStatus = "Successfully sent the information request to the server."
        Case $WINHTTP_CALLBACK_STATUS_RESOLVING_NAME
            $sStatus = "Looking up the IP address of a server name."
        Case $WINHTTP_CALLBACK_STATUS_RESPONSE_RECEIVED
            $sStatus = "Successfully received a response from the server."
        Case $WINHTTP_CALLBACK_STATUS_SECURE_FAILURE
            $sStatus = "One or more errors were encountered while retrieving a Secure Sockets Layer (SSL) certificate from the server."
        Case $WINHTTP_CALLBACK_STATUS_SENDING_REQUEST
            $sStatus = "Sending the information request to the server."
        Case $WINHTTP_CALLBACK_STATUS_SENDREQUEST_COMPLETE
            $sStatus = "The request completed successfully."
        Case $WINHTTP_CALLBACK_STATUS_WRITE_COMPLETE
            $sStatus = "Data was successfully written to the server."
    EndSwitch
    ; Print it
    ConsoleWrite($sStatus & @CRLF)
EndFunc   ;==>__WINHTTP_STATUS_CALLBACK

You need WinHttp.au3 1.6.1.6. for that.

Is there anyway to send files without the use of a form?

I am migrating all my TCP code to your WinHTTP functions and I use this UDF with the TCP: #645071

Then use this code to do what I need:

Dim  $data[3]
Dim  $filenames[1]

$CLOCKED_IN_USERNAME = _URLEncode($CLOCKED_IN_USERNAME, 1)
$CLUB_DIR = _URLEncode($CLUB_DIR, 1)
$Error = _URLEncode($Error, 1)
$data[0] = "User="&$CLOCKED_IN_USERNAME
$data[1] = "Club="&$CLUB_DIR
$data[2] = "Error="&$Error

$filenames[0] = $Zip_File

_HTTPPost_files($host, $page, $socket, $data, $filenames)

[trancexx]

Where do you upload to?

Edited August 31, 2010 by trancexx

[ParoXsitiC]

Where do you upload to?

Normally:

http://www.imvu-e.com/api/email_error.php

I have created a test form to test it out, as this method doesnt not require me to use the base64 UDF, so it saves some space. I just didnt really want people to find the form and mess with it, but not a huge deal. I can't get the form to work using your UDF.

If I manually select a zip file it works fine, the HTTP information shows it as Content-Type: application/octet-stream.

When I use your UDF it shows as Content-Type: application/zip

I am not sure if that is causing it not to process correctly or not, but it seems to be the only thing I find different about manually using the form and autoit.

My test URLS for WinHTTP

http://www.imvu-e.com/api/email_error_form.php

http://www.imvu-e.com/api/email_error_http.php

Edit:

Here are the 2 HTTP POSTs side by side.

Your UDF:

POST /api/email_error_form.php/email_error_http.php HTTP/1.1
Connection: Keep-Alive
Content-Type: multipart/form-data; boundary=----WinHttpBoundaryLine_91944.13181
Accept: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,*/*;q=0.5
Accept-Charset: utf-8;q=0.7
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/533.4 (KHTML, like Gecko) Chrome/5.0.375.127 Safari/533.4
Content-Length: 914
Host: www.imvu-e.com

------WinHttpBoundaryLine_91944.13181
Content-Disposition: form-data; name="User"

bri
------WinHttpBoundaryLine_91944.13181
Content-Disposition: form-data; name="Club"

test
------WinHttpBoundaryLine_91944.13181
Content-Disposition: form-data; name="Error"

errrr
------WinHttpBoundaryLine_91944.13181
Content-Disposition: form-data; name="filename"; filename="songs.zip"
Content-Type: application/zip

PK

The manual form submittion using chrome:

POST /api/email_error_http.php HTTP/1.1
Host: www.imvu-e.com
Connection: keep-alive
Referer: http://www.imvu-e.com/api/email_error_form.php
Content-Length: 938
Cache-Control: max-age=0
Origin: http://www.imvu-e.com
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryIU0xOIIjBKBSK4vS
Accept: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.3 (KHTML, like Gecko) Chrome/6.0.472.51 Safari/534.3
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3

------WebKitFormBoundaryIU0xOIIjBKBSK4vS
Content-Disposition: form-data; name="User"

bri
------WebKitFormBoundaryIU0xOIIjBKBSK4vS
Content-Disposition: form-data; name="Club"

test
------WebKitFormBoundaryIU0xOIIjBKBSK4vS
Content-Disposition: form-data; name="Error"

errrr
------WebKitFormBoundaryIU0xOIIjBKBSK4vS
Content-Disposition: form-data; name="filename"; filename="songs.zip"
Content-Type: application/octet-stream

PK

The biggest thing that strikes me as odd is the content length.

Edit 2:

I have tested it with forcing application/octet-stream...Still nothing. Also I don't think content length is a big deal because of the different sizes in the Boundary is probably contributing to that.

Edited August 31, 2010 by ParoXsitiC

[trancexx]

There was an issue in WinHttp_1.6.1.6.

Funny thing is you found it right away. It took me some thinking on my side yesterday to find it. I have yesterday. It was Revision 58.

Can you try the latest file (rev 60). Link to that.

If that works I will make new release right away.

edit:

The issue is seen in the first line of what you posted as Your UDF. It's:

POST /api/email_error_form.php/email_error_http.php HTTP/1.1

and should be:

POST /api/email_error_http.php HTTP/1.1

Edited August 31, 2010 by trancexx

[ParoXsitiC]

There was an issue in WinHttp_1.6.1.6.

Funny thing is you found it right away. It took me some thinking on my side yesterday to find it. I have yesterday. It was Revision 58.

Can you try the latest file (rev 60). Link to that.

If that works I will make new release right away.

I tried with rev 60 and it works perfect. No errors.

[trancexx]

I tried with rev 60 and it works perfect. No errors.

Good.

Anything else you can think of?

edit:

Yes or no, new release!

Edited August 31, 2010 by trancexx

[ParoXsitiC]

Good.

Anything else you can think of?

edit:

Yes or no, new release!

I am noticing something else; This could be as a result of the zip UDF, but it seems 1 out of every 10 post to the form seems to cause autoit.exe to hard crash.

[trancexx]

I am noticing something else; This could be as a result of the zip UDF, but it seems 1 out of every 10 post to the form seems to cause autoit.exe to hard crash.

With callback? If yes then try without it.

[ParoXsitiC]

With callback? If yes then try without it.

Yes with the callback.

I noticed it happens way more often if I don't close the handles. What I am doing is sending a debug log file to my email when there is an AutoIT error. Basically a second script runs side by side the main script. When it realizes the PID wants to terminate, it reads the console and finds the error, then it takes my temporary debug file in %TEMP% and creates a zip file and posts it to a PHP file that sends it to my email.

The problem I had is that if there is an unexpected error, I didnt account that I should close the handle so what I did was close all WinHTTP handles using OnAutoItExitRegister. This seems to cut down the AutoIt hard crashes ten fold.

Edit:

If I disable the callback, I can't get it to crash even with not closing the handle. It definitely seems the instability of using the callback has to deal with closing handles properly.

Edited August 31, 2010 by ParoXsitiC

[trancexx]

Edit:

If I disable the callback, I can't get it to crash even with not closing the handle. It definitely seems the instability of using the callback has to deal with closing handles properly.

I actually expected something like that. I think It's related to lifetime of the callbacks in AutoIt. WinHttp keeps calling function that no longer exists.

This is old AutoIt's issue. It can be worked around by smarter coding (closing handles in this case).

Edited August 31, 2010 by trancexx

[notsure]

The helpfile isn't working, i only get Page cannot be displayed in it.

Is there somewhere a descent example for using WINhttp with Google as example?

Like why isn't this working?

#include<WinHTTP.au3>


$LocalIP = "www.google.com"
$hw_open = _WinHttpOpen()
$hw_connect = _WinHttpConnect($hw_open, $LocalIP)

$h_openRequest = _WinHttpOpenRequest($hw_connect, "GET", "","","","")
$data = ""

local $SendREQ = _WinHTTPSendRequest($h_openRequest) ; send request
_WinHttpReceiveResponse($h_openRequest) ; wait for response

$sHeaders = _WinHttpQueryHeaders($h_openRequest)


$sData = ""
If _WinHttpQueryDataAvailable($h_openRequest) Then
    Do
        $sData &= _WinHttpReadData($h_openRequest)
    Until @error<>0
EndIf

_WinHttpCloseHandle($h_openRequest)

_WinHttpCloseHandle($hw_connect)

_WinHttpCloseHandle($hw_open)


MsgBox(0, 'Header', $sHeaders)
MsgBox(0, 'Data', $sData)

[ProgAndy]

Hi,

The helpfile is OK. This behaviour is a "feature" of Windows. You have to open the file properties for the chm-file (Contextmenu -> Properties) and click on "Unblock" and then "Apply". If you need more help look here

Edit: You should have a look at the _WinHttpSimple...-functions. Those functions are designed to simplify some parts of a request.

Edited September 6, 2010 by ProgAndy

[notsure]

It doesn't say its blocked. Also the "Unblock" button is missing there.

Edit: The AUTOIT helpfile .CHM is working like a charm... so i'm not sure why this aint workin.

Edited September 6, 2010 by notsure

[trancexx]

It doesn't say its blocked. Also the "Unblock" button is missing there.

Edit: The AUTOIT helpfile .CHM is working like a charm... so i'm not sure why this aint workin.

Maybe because you are running it from within zip package. Unzip it if yes.

edit: and there is "Home" button available that would allow you to view online help from the help file as well. That works?

Edited September 6, 2010 by trancexx

[notsure]

Its unzipped, however i can browse through a documentation website if i choose "HOME" indeed. Whats the URL of that website?

Thanks, atleast i have some documentation of it.

[trancexx]

It's in the first post. Probably (hopefully) the one saying Online help.

Interesting you aren't able to run help file properly. Hm.

[ProgAndy]

What is the path of your chm-file? It won't work with UNC paths, only local diectories are supported. Try to copy it it the same location as your AutoIt help. What happens then?

[notsure]

What is the path of your chm-file? It won't work with UNC paths, only local diectories are supported. Try to copy it it the same location as your AutoIt help. What happens then?

Thats it! No UNC path's. It works when i copy it to local drives. Thanks.

[marko001]

Hi Trancexx,

I'll move the post here since it seems the right place where I can get your help:

I developed a tool to automatically log into a web page:

expandcollapse popup

#include "WinHttp.au3"
#Include <String.au3>
#include <Array.au3>

Opt("MustDeclareVars", 1)

Global $sFile = @DesktopDir & "\Skills.xml"
Global $sFile2 = @DesktopDir & "\Skills2.xml"
Global $hFile, $hfile2
Global $temp_file, $A_token, $P_token

global $p_csrfToken, $p_id



Global $hConnect, $sRead, $hrequest
Global $sContent = "Content-Type: application/x-www-form-urlencoded; charset=UTF-8"
global $sAddress = "mydomain.com"
global $spage = "login"
global $spage_2= "2ndpage?p_id="
Global $hOpen
global $sChunk, $sData


$hOpen = _WinHttpOpen()
$hConnect = _WinHttpConnect($hOpen, $sAddress)
_WinHttpSetOption($hOpen, $WINHTTP_OPTION_USER_AGENT, $sContent)
$sRead = _WinHttpSimpleFormFill($hConnect, $spage ,"logindata","name", "myname", "password", "mypw")

If @error Then
    msgbox(0,"","Error")
    Exit
Else
    $hFile = FileOpen($sFile, 2)
    FileWrite($hFile, $sRead)
    FileClose($hFile)
EndIf

Into this page i can find a unique ID I need to go further:

FileOpen($sfile)
$temp_file = FileRead($sfile)
$P_token = _StringBetween($temp_file,"p._id = '","';",-1)
$p_id = _ArrayToString($P_token)
fileclose($sfile)
$spage_2= $spage_2& $p_id

Now i need to go further, second page, using the url:

http://mydomain.com/2ndpage?p_id=$p_id (the one i got into previous lines)

I tried

expandcollapse popup

$hRequest = _WinHttpOpenRequest($hConnect, _
        "GET", _ ; verb
        $spage_2=, _    ; target
        Default, _ ; version
        $sAddress, _    ; referer
        "*/*") ; accept

_WinHttpAddRequestHeaders($hRequest, "User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3")
_WinHttpAddRequestHeaders($hRequest, "Accept-Language: it-it,it;q=0.8,en-us;q=0.5,en;q=0.3")
_WinHttpAddRequestHeaders($hRequest, "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
_WinHttpAddRequestHeaders($hRequest, "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7")
_WinHttpAddRequestHeaders($hRequest, "Keep-Alive: 115")
_WinHttpAddRequestHeaders($hRequest, "Connection: keep-alive")
_WinHttpAddRequestHeaders($hRequest, "Accept: application/json, text/javascript, */*")
_WinHttpAddRequestHeaders($hRequest, "Content-Type: application/x-www-form-urlencoded; charset=UTF-8")
_WinHttpAddRequestHeaders($hRequest, "Cache-Control: no-cache")

global $answer = _WinHttpSendRequest($hRequest)

;msgbox(0,"answer",$answer)

global $answer2 = _WinHttpReceiveResponse($hrequest)
If @error Then
    MsgBox(48, "Error", "Error ocurred for WinHttpReceiveResponse, Error number is " & @error)
Else

EndIf

global $data_avail = _WinHttpQueryDataAvailable($hRequest)
;msgbox(0,"data",$data_avail)

If _WinHttpQueryDataAvailable($hRequest) Then
    ; Read
    $hFile2 = FileOpen($sFile2, 2)
    FileWrite($hFile2, _WinHttpReadData($hRequest))
    FileClose($hFile2)
Else
    MsgBox(48, "Error", "Site is experiencing problems.")
EndIf

I always get no errors into msgboxes but always a 0 bytes file.

I think problem is due to cookies, since it seems he can't remember previous credentials.

I tried, using Mozilla Live Headers, to catch them logging from Mozilla itself, and I added them to _WinHttpAddRequestHeaders:

_WinHttpAddRequestHeaders($hRequest, 'Cookie: portal_tid=1286813622690-82399; __utma.....')
_WinHttpAddRequestHeaders($hRequest, 'Cookie: chtOpt=0; cid=661578042; __utma=2148334....')

But notihng change.

It seems that once i close the first call it forgets login datas so when I try to go to inner page he can't find it.

How can it be solved?

Thanks in advance,

M.

[trancexx]

It's not cookies. They are properties of session handle. To be sure just add something like:

_WinHttpSetOption($hOpen, $WINHTTP_OPTION_DISABLE_FEATURE, $WINHTTP_DISABLE_COOKIES)

Data is always read in a loop. Fix that then ask again.

edit: it's not clear to me if you get to close $hOpen after logging in. Don't, just continue with that handle; pass it to the next request.

Edited October 11, 2010 by trancexx