HTTP UDF's

[OverloadUT]

I am working on a fairly large project and a big part of it is an AutoIt application that uploads a lot of data to a PHP script.

For a while I was using INetGet and submitting all my data as GET variables in the URL. However, the amount of data to upload got too big (over 1000 characters) and the INetGet function started to fail.

I also hated that INetGet ONLY works if the webserver responds with "200 OK" - There is no way to tell the difference between a 404 error and simply not being able to connect to the server in the first place.

So I write this library of UDF's. Basically, it's a fully compliant HTTP/1.1 client. It uses only the TCP functions; no DllCall needed here! I had a blast learning all about the HTTP protocol and how to use it.

Advantages over INetGet:

The data is downloaded right in to variables instead of to files. You can of course then write this data to a file if you wish.

You can read all of the headers supplied by the webserver.

You can get the HTTP Response Code from the webserver.

When it failes, you know exactly what failed instead of having to guess.

; ===================================================================
; HTTP UDF's
; v0.5
;
; By: Greg "Overload" Laabs
; Last Updated: 07-22-06
; Tested with AutoIt Version 3.1.1.131
; Extra requirements: Nothing!
;
; A set of functions that allow you to download webpages and submit
; POST requests.
;
; Main functions:
; _HTTPConnect - Connects to a webserver
; _HTTPGet - Submits a GET request to a webserver
; _HTTPPost - Submits a POST request to a webserver
; _HTTPRead - Reads the response from a webserver
; ===================================================================

I consider this UDF package to be a "beta" and that's why I call it version 0.5. However, I would love people to give it a try and tell me how it works for them. I have done extensive testing and I think I have the parser working perfectly.

I plan on improving this library. It's possible that I might change the way the functions work (I don't think the _HTTPConnect function is necessary; I could move that functionality right in to the Get or Post functions.)

To Do:

• Add a function that downloads the data right to a file. You might not want to download a 10 meg file in to memory before saving it to a file.
• Add a method to get the progress of the download. This will probably be in the form of a callback function, if that's possible in AutoIt.
• Add the ability to automatically follow "Location:" headers. Currently you'd have to do it manually.
• Other things I can't think of!

Post #25 For Updated functions to work with current AutoIt versions.

Edited November 15, 2007 by SmOke_N
Redirect to correct post for download

[rakudave]

sounds very promising, keep going!

[Wus]

What resource did you find most helpful in learning the http protocol?

Looks nice though and it gives me a few ideas.

[rbhkamal]

@OverloadUT

Thanks for sharing, maybe when you are done you can add it as part of the Include package.\

You are the best!

RK

[OverloadUT]

Thank you everyone for the kind words. If anyone actually gives it a try, I'd love to hear how it works for them.

What resource did you find most helpful in learning the http protocol?

This website was my main source. Well, that and a LOT of trial and error.

[rbhkamal]

Thank you everyone for the kind words. If anyone actually gives it a try, I'd love to hear how it works for them.

Edit: Since $socket is optional and $data is more likely to be used, shouldn't it be at the end? Like(_HTTPPost($host, $page, $data = "" , $socket = -1).

I was woundering how can I submit a search on google.

I realy don't understand how to use the _HTTPPost() function.

here is what I did:

#include <HTTP.au3>
#include <IE.au3>
$host = "www.google.com"
$page = "/imghp?hl=en&tab=wi&q="


ConsoleWrite("Example GET Request:"&@CRLF)
$socket = _HTTPConnect($host)
ConsoleWrite("Socket Created: "&$socket&@CRLF)
$get = _HTTPGet($host, $page, $socket)
ConsoleWrite("Bytes sent: "&$get&@CRLF)
$recv = _HTTPRead($socket,1)
If @error Then
    ConsoleWrite("_HTTPRead Error: "&@error&@CRLF)
    ConsoleWrite("_HTTPRead Return Value: "&$recv &@CRLF)
Else
    ConsoleWrite("HTTP Return Code: "&$recv[0]&@CRLF)
    ConsoleWrite("HTTP Return Response: "&$recv[1]&@CRLF)
    ConsoleWrite("Number of headers: "&UBound($recv[3])&@CRLF)
    ConsoleWrite("Size of data downloaded: "&StringLen($recv[4])&" bytes"&@CRLF)
    ConsoleWrite("Page downloaded: "&@CRLF&$recv[4]&@CRLF)
    $O_IE = _IECreate()
    _IEBodyWriteHTML( $O_IE , $recv[4] )
EndIf
_HTTPPost($host, $page, $socket, "What should I put here??????" );I want to search for "Prosche"
_HTTPClose($socket)

RK

Edited July 24, 2006 by rbhkamal

[Swimming_Bird]

wewt i've been after an HTTPPost command

tahnks i'll play with it a bit.

[CoePSX]

:"> You know what would be REALLY nice --> something to handle cookies.

Althouhg i have 0 knowledge on cookies, they're still necessary for pages that require

logins

Cheers!

[OverloadUT]

Google actually does all of its searches as GET requests, not POST requests. This is so that you can link a page of results and actually have it work.

In that case, to do a google search, you would want to do this:

$host = "www.google.com"
$page = "/imghp"
$vars = "hl=en&tab=wi&q="

$searchterm = "porche"

$url = $page&"?"&_HTTPEncodeString($vars&$searchstring)

$socket = _HTTPConnect($host)
$get = _HTTPGet($host,$url,$socket)
$recv = _HTTPRead($socket,1)
ConsoleWrite("Data received:"&@CRLF$recv[4]&@CRLF)

Any variables after the question mark need to be HTTP Encoded, which is why I put the "vars" in a separate variable from the page.

[OverloadUT]

:"> You know what would be REALLY nice --> something to handle cookies.

Althouhg i have 0 knowledge on cookies, they're still necessary for pages that require

logins

Cheers!

Well, you can read the cookies sent by the server using these functions, but I currently don't have a way of sending custom headers with a get or post request - that's something I'll need to add. I might create special cookie handling functions as well.

[martijn]

I've just begun playing around with your script and have a few suggestions:

1. Change "Dim $recv = TCPRecv($socket,16)" into "Dim $recv = TCPRecv($socket,4096)" to speed things up (especially when receiving larger files)

2. I'm unable to use this at work due to proxy settings (TCPConnect doesn't like the proxy). Is it possible to add such settings?

3. How about adding a referrer to the _HTTPPost and _HTTPGet

4. It would be nice if the _HTTPSetUserAgent would accept an array of values (*)

(*) My useragent is built out of 3 products and 1 comment: "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7". Here's a link to the 2616RFC

Other than that it's excellent and perfectly suits my needs

[ConsultingJoe]

This is a great UDF, I was wondering if it can be used with HTTPS.

it only works if I use www. not http. is there a reason for this?

Thanks

[martijn]

For https you have to add an extra parameter to the _HTTPConnect-function call. Instead of _HTTPConnect("www.yourserver.com"), make it _HTTPConnect("www.yourserver.com",443)

443 is the default port number for ssl

[ConsultingJoe]

For https you have to add an extra parameter to the _HTTPConnect-function call. Instead of _HTTPConnect("www.yourserver.com"), make it _HTTPConnect("www.yourserver.com",443)

443 is the default port number for ssl

Really, Thanks. I was trying 80.

So that means that instead of https://www.site.com its just www.site.com:443?

[ConsultingJoe]

I tried that and it didn't work.

Here is the Script I am using:

#include <HTTP.au3>
$from = "JOE"
$number = "***-***-****"
$msg = "hi"

$host = "wmg.tmomail.net"
$page = "/keys/write.php"
$vars = "trackResponses=No&Send.x=Yes&DOMAIN_NAME=@tmomail.com&min=" & $number & "&require_sender=" & $from & "&text=" & $msg & "count=" & 140-StringLen($msg) & "&msgTermsUse=1"

$data = _HTTPEncodeString($vars)
ConsoleWrite($data & @CRLF)
$socket = _HTTPConnect($host, 443)
$post = _HTTPPost($host, $page, $socket, $data)
ConsoleWrite(@error & @CRLF)
$recv = _HTTPRead($socket,1)
FileWrite( @ScriptDir & "\result.html", $recv[4] )

[martijn]

I tried that and it didn't work.

I've tried the url in a browser from above code example, but it gives me a 404-not found-message.

So HTTP UDF *does* seem to work, but you are probably using the wrong url.

Edit: the 404-error could be generated by the php-file

Edited October 4, 2006 by martijn

[ConsultingJoe]

I've tried the url in a browser from above code example, but it gives me a 404-not found-message.

So HTTP UDF *does* seem to work, but you are probably using the wrong url.

Edit: the 404-error could be generated by the php-file

The address should be: https://wmg.tmomail.net/customer_site/jsp/messaging_lo.jsp

So how do I format it?

Thanks

[martijn]

It's a shame I didn't see that this script does allow to connect to a server with a ssl-portnumber (443), but that it doesn't support ssl-communication. In other words: it's sending a http request over port 443, where in fact it should send a https-request (ssl encrypted) over port 443. That is not implemented. I suggest to use something like Automation. I've illustrated with a little example:

#include <IE.au3>

Dim $oIE, $o_form, $o_from, $o_to, $o_msg, $o_terms, $b_showbrowser = true
Dim $from = "Joe", $to = "3334445555", $msg = "Hi again"

; Create a browser window and navigate to tool
    $url = "https://wmg.tmomail.net/customer_site/jsp/messaging_lo.jsp"
    $oIE = _IECreate ($url, 0, $b_showbrowser)
; get pointers to the form and from, to, message and terms fields
    $o_form = _IEFormGetObjByName ($oIE, "message_form")
    $o_from = _IEFormElementGetObjByName ($o_form, "require_sender")
    $o_to = _IEFormElementGetObjByName ($o_form, "min")
    $o_msg = _IEFormElementGetObjByName ($o_form, "text")

    _IEFormElementCheckBoxSelect($o_form, 0, "", 1, "byIndex")

; Set field values and submit the form
    _IEFormElementSetValue ($o_from, $from)
    _IEFormElementSetValue ($o_to, $to)
    _IEFormElementSetValue ($o_msg, $msg)
    _IEFormSubmit ($o_form)
    _IELoadWait($oIE)
    _IEQuit ($oIE)

Ofcourse, uncomment the IEQuit to see the result of this code ;-)

Works for me

Edited October 5, 2006 by martijn

[jazo10]

Hey To start of great script!

just when im using the HTTP example, im constatly getting a

_HTTPRead Error: 5

_HTTPRead Return Value: HTTP/1.1 200 OK

Reply. ive tried heaps of diffrent addresses,

any clue whats going on?

it may be a proxy server, but IE is set for it, and one of the address i am using is localhost anyway.

any help would be awsome.

[faldo]

Hey To start of great script!

just when im using the HTTP example, im constatly getting a

_HTTPRead Error: 5

_HTTPRead Return Value: HTTP/1.1 200 OK

Reply. ive tried heaps of diffrent addresses,

any clue whats going on?

it may be a proxy server, but IE is set for it, and one of the address i am using is localhost anyway.

any help would be awsome.

It because this script wasn't intended for AutoIT v3.2.2.0, i tried it with an earlier version of AU3 and it works... could someone please update it?