_INetGetSourcePro() UDF

[CatchFish]

Download from a URL without writing a temporary file. Enhanced version with functionalities of authentication(basic), proxy and even uploading.

Syntax:

$content = _INetGetSourcePro("http://some.place.com")

- Usual download

$content = _INetGetSourcePro("http://need.password.com", "GET", "", 1, "Username", "Password")

- Download using a username/password pair for the web page authentication

$content = _INetGetSourcePro("http://through.proxy.com", "GET", "", 1, "", "", _
"http=http://proxyname:port", "ProxyUsername", "ProxyPassword")

- Download through a common HTTP proxy

$content = _INetGetSourcePro("http://upload.something.com", "POST", "information to upload", 0)

- Upload to a web server & don't need the result

$content = _INetGetSourcePro("http://upload.something.com")
If @error = 9 Then $HttpStatusCode = @extended

- Get the HTTP reponse code, if not "HTTP 200 OK"

Download the .au3: INetPro.au3

Notice: you need AutoIt beta 3.1.1.93 or above in order to make it work correctly.

[rambo3889]

Nice

Edited February 14, 2006 by rambo3889

[w0uter]

Local $ai_IO, $i_Type = $INTERNET_OPEN_TYPE_DIRECT

should be changed too

Local $ai_IO, $i_Type = $INTERNET_OPEN_TYPE_PRECONFIG

other then that it certainly looks good

[raquien]

Hi guys, does anyone tested this? Where will the downloaded file is saved?

[mary]

good job ! thinks

Just a question: how to handle headers:

-like cookies (send a request with a cookie's value) ?

-Content Encoding (gzip or deflate)

-gzip, deflate decompression

thinks

Edited May 18, 2007 by mary

[drlava]

Added ability to download binary files and use headers like Accept-encoding, for eg:

$ReceiveBuffer=_InetGetSourcePro("http://www.redlinenetworks.com/", 'GET', "Accept-Encoding: gzip, deflate"&@CR&@LF)

The headers var pushes vars after $s_Verb back one position.

Func _INetGetSourcePro($s_URL, $s_Verb = 'GET', $s_Header = '', $s_Submits = '', $i_ReadFile = 1, $s_UrlUsername = '', $s_UrlPassword = '', $s_Proxy = '', $s_ProxyUsername = '', $s_ProxyPassword = '')

InetPro.zip

for dealing with gzipped data, see here.

Edited July 17, 2007 by drlava

[MrSpacely]

Can Something be added?

I have seen on msdn that als the returned headers can be retrieved with this dll but I am not really into structs so I someone can add that it would be great. it uses queryinfo but it would be handy if it would just send all headers to @extended (or something else)

Edit: It seems I found a way to make it work (only did something wrong first)

So I added some things

First I changed the function info comments so the header var was explained (I did not make that but is was from the previous addition)

Second I added a dllcall so it would retrieve all headers sent back by the server

Third I changed the way the function would return the source. It uses a array as return value why? Well that works easy and no need for a byref this way.

Return value change! What?

Previous you get a string as return value.

In this version you get a array with 3 items

[0] is the source of the page (actually wich is send after all the headers have been sent)

[1] are all the headers wich are sent by the server (wich are actually raw wich means they are exactly the same structure as they are sent over tcp)

[2] A cut out of the Set-Cookie: header only the string after Set-Cookie is inside here

Small fact the http server sents the headers (just one per line) and after al headers it sends a extra CRLF wich just looks like a blank line (wich it is)

after this the page is sent or anything you download like images flash movies program or etc. So the header and data are just devided by 2 CRLFs nothing more quite simple. Weird thing is that wininet.dll needs a extra call to get them while its just one string of data wich anyone can cut into 2 pieces (just look for the first double CRLF)

Why did I make the addition to get server headers? Well to automate actions on websites its really handy to get the sessionid cookie sometimes otherthings found in the header are also really handy.

I had it working on autoitscript using a windows port of Wget wich is used on unix/linux. But thats a exe file and wich had to be included in the compilation making the autoit compiled script larger and also its not the greatest way because I had to remove the file on execution wich could make garbage on someones pc if something crashed. Using autoit to automate website actions (like voting logging on etc) is really quick this way because you donot need to load all the images flash and other thing you donot see anyway when you want to automate this. But it does need some good reseach of the website each time the webmaster changes something using a sniffer and reading the source helps but its also fun researching this.

I also had this same thing working using the tcp socket functions in autoitscript but there is something bad about that it was quite slow. Also you had to make the redirect function also (wich wininet en wget can do automatic). Something else also happened it had trouble receiving everything so I had to make a flow control wich made it slower (wich is weird tcp should take care of that) But i tested that few months ago maybe the tcp functions of autoitscript have improved

I hope that someone can also use this addition

Edit 2: Added a useragent var for easy changing. Why it seems websites sent differend websites when its not mozilla or internet explorer (stopping spiders or bots)

so I changed so its sends a internet explorer user-agent standard

Maybe something can be done about the wait time with large file the script just waits using adlibenable it might be possible to get the handle and read the status of the connection using

void CALLBACK InternetStatusCallback(
  HINTERNET hInternet,
  DWORD_PTR dwContext,
  DWORD dwInternetStatus,
  LPVOID lpvStatusInformation,
  DWORD dwStatusInformationLength
);

http://msdn2.microsoft.com/en-us/library/aa385121.aspx

This might make it really handy automating something and seeing the progress (wich I had working using Wget.exe reading the console with adlib enable wich worked good but should also be able to work using this way)

Edit3: Problem found it seems I cannot send a cookie (wich should be simple with headers)

It just sends its own cookie even with INTERNET_FLAG_NO_COOKIES option set to HttpOpenRequest it still sends the cookie wich I donot want

and it will not send my own cookie how stupid its so simple wininet.dll just send it its just a string why filter it out

Edit4: It was my mistake I just added the const's at the top and copy pasted the ;~ at the beginning of the line with it (lol)

So it does work now and added HTTPaddRequestHeaders wich can force replacement of headers (could be handy so I kept it in)

updated INetPro.au3

Edited July 30, 2007 by MrSpacely

[EvAsion]

can you explain how this can be used to upload to a web server?

[raquien]

Got this error..

C:\Documents and Settings\Desktop\INetPro.au3(279,148) : ERROR: Binary(): undefined function.
    Local $ai_IO, $i_Type = $INTERNET_OPEN_TYPE_PRECONFIG, $ai_IC, $ai_HOR, $ai_HSR, $ai_HQI, $s_StatusCode, $strc_Length, $ai_IRF, $s_Buf = Binary('')
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
C:\Documents and Settings\Desktop\INetPro.au3(426,77) : ERROR: BinaryMid(): undefined function.
                                        $s_Buf &= BinaryMid(DllStructGetData($BinaryBuffer,1),1,$ai_IRF[4])
                                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
C:\Documents and Settings\Desktop\INetPro.au3(432,45) : ERROR: BinaryToString(): undefined function.
                                        $ReturnVal = BinaryToString($s_Buf)
                                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^

[Sypher]

Same here

[kanumi]

I found an error in the latest InetPro.au:

Else ;Everything's OK

Local $ReturnVal[8]

$ReturnVal[0] = DllStructGetData($strc_UC, 4) ;INTERNET_SCHEME value

$ReturnVal[1] = _GetDataFromPtr(DllStructGetData($strc_UC, 2 ), 'char[' & DllStructGetData($strc_UC, 3 ) & ']') ;Protocol scheme name

$ReturnVal[2] = _GetDataFromPtr(DllStructGetData($strc_UC, 5 ), 'char[' & DllStructGetData($strc_UC, 5 ) & ']') ;Host name

$ReturnVal[3] = DllStructGetData($strc_UC, 7) ;Port number

$ReturnVal[4] = _GetDataFromPtr(DllStructGetData($strc_UC, 8 ), 'char[' & DllStructGetData($strc_UC, 8 ) & ']') ;User name

$ReturnVal[5] = _GetDataFromPtr(DllStructGetData($strc_UC, 10), 'char[' & DllStructGetData($strc_UC, 10) & ']') ;Password

$ReturnVal[6] = _GetDataFromPtr(DllStructGetData($strc_UC, 12), 'char[' & DllStructGetData($strc_UC, 12) & ']') ;URL path

$ReturnVal[7] = _GetDataFromPtr(DllStructGetData($strc_UC, 14), 'char[' & DllStructGetData($strc_UC, 14) & ']') ;Extra information

The indices to access the sizes elements within the struct must be incremented by 1:

Else ;Everything's OK

Local $ReturnVal[8]

$ReturnVal[0] = DllStructGetData($strc_UC, 4) ;INTERNET_SCHEME value

$ReturnVal[1] = _GetDataFromPtr(DllStructGetData($strc_UC, 2 ), 'char[' & DllStructGetData($strc_UC, 3 ) & ']') ;Protocol scheme name

$ReturnVal[2] = _GetDataFromPtr(DllStructGetData($strc_UC, 5 ), 'char[' & DllStructGetData($strc_UC, 6 ) & ']') ;Host name

$ReturnVal[3] = DllStructGetData($strc_UC, 7) ;Port number

$ReturnVal[4] = _GetDataFromPtr(DllStructGetData($strc_UC, 8 ), 'char[' & DllStructGetData($strc_UC, 9 ) & ']') ;User name

$ReturnVal[5] = _GetDataFromPtr(DllStructGetData($strc_UC, 10), 'char[' & DllStructGetData($strc_UC, 11) & ']') ;Password

$ReturnVal[6] = _GetDataFromPtr(DllStructGetData($strc_UC, 12), 'char[' & DllStructGetData($strc_UC, 13) & ']') ;URL path

$ReturnVal[7] = _GetDataFromPtr(DllStructGetData($strc_UC, 14), 'char[' & DllStructGetData($strc_UC, 15) & ']') ;Extra information