Problem with _WinHttpSimpleFormFill in action page

[Mithrandir]

I am experiencing problems with this WinHttp.au3 method. Here is the code -note: the forum I'm using has a section called 'Pruebas' (Testing) to test things so that's why I chose it- As you will see in the code I tried both the log in that section and in the main page and in the html code returned in $sRead the forum tells me of wrong user or pass but I tried manually and both are ok. What's wrong? I tried in another forum that is located at the main page (like www.forums.site.com) and it did it well because I left $sActionPage as default from _WinHttpSimpleFormFill(ByRef $hInternet [, $sActionPage = Default [, $sFormId = Default [, $sFieldId1 = Default [, $sData1 = Default [, (...)]]]]])

but when I tried to post in a section of that forum it also told me the user or password were wrong. The problem seems to be in the $sActionPage parameter. I appreciate any help because I'm lost. Thanks for your help!!

#include <WinHttp.au3>

$host = 'www.dreamsprojects.com'

$hOpen = _WinHttpOpen("Mozilla/5.0 (Windows; U; Windows NT 5.1; es-ES; rv:1.9.2.10) Gecko/20100914 Firefox/3.6.13 (.NET CLR 3.5.30729)")

$hConnect = _WinHttpConnect($hOpen, $host)

_WinHttpSetOption($hConnect, $WINHTTP_OPTION_DISABLE_FEATURE, $WINHTTP_DISABLE_COOKIES)

#cs
Here is the two tries to log in www.dreamsprojects.com/foro/newthread.php?do=newthread&f=104 both in the navbar (index 0) and in the central log in form (index 3) 
;$sRead = _WinHttpSimpleFormFill($hConnect,"foro/newthread.php?do=newthread&f=104","index:0","navbar_username","user","navbar_password","pass")
;$sRead =_WinHttpSimpleFormFill($hConnect,"foro/newthread.php?do=newthread&=104","index:3","vb_login_username","user","vb_login_password","pass")

The tries to log in the main page of the forum (www.dreamsprojects.com/foro/)
;$sRead = _WinHttpSimpleFormFill($hConnect,"foro/login.php?do=login","index:0","vb_login_username","user","vb_login_password","pass")
;$sRead = _WinHttpSimpleFormFill($hConnect,"foro/","index:0","vb_login_username","user","vb_login_password","pass")
#ce

FileWrite(@ScriptDir&'/htmlofsubmittedform.html',$sRead)

Here it is WinHttp.au3 UDF: http://www.mediafire.com/?ug21cgccn1nsj6g

Edited December 31, 2010 by Mithrandir

[Mithrandir]

I found in msdn documentation that

WINHTTP_OPTION_DISABLE_FEATURE

Sets an unsigned long integer value that specifies which features are disabled with one or more of the following flags. Be aware that this feature should only be passed to WinHttpSetOption on request handles after the request handle is created with WinHttpOpenRequest, and before the request is sent with WinHttpSendRequest.

So is useless to disable the feature of disabling cookies (thus enabling them for sure) because I performed a winhttpopen not a winhttpopenrequest. Anyway, I think cookies are managed by WinHttpSimpleFormFill and that the problem is in the actionpage but I'm still clueless about that

Edit: Another thing that I found was that in the case of the forum which I succeded in login in, the returned HTML was from a page that redirected to the one which I want the HTML. Is there a way to get that HTML using WinHttpSimpleFormFill?

Edited January 1, 2011 by Mithrandir

[trancexx]

Your $sActionPage should be left default, after quick look. And your form should include md5 of the password. Something like this:

$sRead = _WinHttpSimpleFormFill($hConnect, Default, "index:0", _
        "name:vb_login_username", "user", _
        "name:vb_login_password", "pass", _
        "name:vb_login_md5password", _MD5("pass"))

There is also vb_login_md5password_utf input but that's probably for passwords that include characters requiring UTF encoding (if yours is then use that).

_MD5() is some function that will return md5 checksum for the password.

[Mithrandir]

Your $sActionPage should be left default, after quick look. And your form should include md5 of the password. Something like this:

$sRead = _WinHttpSimpleFormFill($hConnect, Default, "index:0", _
        "name:vb_login_username", "user", _
        "name:vb_login_password", "pass", _
        "name:vb_login_md5password", _MD5("pass"))

There is also vb_login_md5password_utf input but that's probably for passwords that include characters requiring UTF encoding (if yours is then use that).

_MD5() is some function that will return md5 checksum for the password.

Thanks! That functioned well! :

#include <WinHttp.au3>
#include <Crypt.au3>

$host = 'www.dreamsprojects.com'
$hOpen = _WinHttpOpen("Mozilla/5.0 (Windows; U; Windows NT 5.1; es-ES; rv:1.9.2.10) Gecko/20100914 Firefox/3.6.13 (.NET CLR 3.5.30729)")
$hConnect = _WinHttpConnect($hOpen, $host)
_WinHttpSetOption($hConnect, $WINHTTP_OPTION_DISABLE_FEATURE, $WINHTTP_DISABLE_COOKIES)
_WinHttpSetOption($hConnect,$WINHTTP_OPTION_DISABLE_FEATURE,$WINHTTP_DISABLE_REDIRECTS)
$sPassword = "mypassword"
$MD5pass = StringLower(StringTrimLeft(_Crypt_HashData($sPassword,$CALG_MD5),2))
$sRead = _WinHttpSimpleFormFill($hConnect, Default, "index:0", _
"name:vb_login_username", "username", _
"name:vb_login_password", $sPassword, _
"name:vb_login_md5password", $MD5pass)
FileWrite(@ScriptDir&'/htmldesubmittedform.html',$sRead)

I had to use stringtrimleft because _Crypt_HasData returns the md5 with a 0x before and in the request that is not send. Also, the characters must be sent in lowercase and the function returned it all upercase.

As you can see I left the _WinHttpSetOption functions but are they useful? because as I posted before I read in msdn documentation that you have to pass the handle from _WinHttpOpenRequest and not _WinHttpConnect. Anyway, I can solve the issue of getting the following page by using regexp but concerning the issue of cookies:

are the cookies stored in $hConnect or $hOpen?

are all the cookies stored? or only the first line of the 'set-cookie:' response?

HTTP/1.1 200 OK
Date: Fri, 31 Dec 2010 22:18:24 GMT
Server: Apache
Cache-Control: private
Pragma: private
X-UA-Compatible: IE=7
Content-Encoding: gzip
Set-Cookie: bblastactivity=0; expires=Sat, 31-Dec-2011 22:18:24 GMT; path=/; domain=.dreamsprojects.com
Set-Cookie: bbsessionhash=0856598f6d9dsdfsaewaaaaaaa; path=/; domain=.dreamsprojects.com; HttpOnly
Set-Cookie: vbseo_loggedin=yes; expires=Fri, 31-Dec-2010 23:18:24 GMT; path=/
Content-Length: 3448
Keep-Alive: timeout=15, max=150
Connection: Keep-Alive
Content-Type: text/html; charset=ISO-8859-1

Because I read in another thread that with a winhttp function (I don't remember now what exactly the function was) only the first one is returned.

Anyway I'm going to do some testing to see if I can answer the above doubts by myself and report back either the solution or the failure

Another thing that I would like to ask to trancexx is: is it possible to modify your UDF in line 1293

$hRequest = _WinHttpOpenRequest($hInternet, $sMethod, $sAction)
        If $fMuftiPart Then

To add a refferrer instead of living it blank? It could be an optional parameter in _WinHttpSimpleFormFill and to the best of my knowledge after reading your code, it wouldn't mess anything.

[trancexx]

Cookies are properties of sessions. That would be $hOpen. Every connection in that session get its cookies unless disabled. You read cookies with _WinHttpQueryHeaders(). You do that by reading the full header and then extract cookies data, or specifying $WINHTTP_QUERY_COOKIE and calling in a loop while @extended is set (last param for _WinHttpQueryHeaders should then be set to @extended value of the previous call). That way you get all cookies read.

Why would you want to specify referrer? Is that a requirement of some sort of the page that you automate? It can be added, but is it really necessary?

Edited January 1, 2011 by trancexx

[Mithrandir]

Cookies are properties of sessions. That would be $hOpen. Every connection in that session get its cookies unless disabled. You read cookies with _WinHttpQueryHeaders(). You do that by reading the full header and then extract cookies data, or specifying $WINHTTP_QUERY_COOKIE and calling in a loop while @extended is set (last param for _WinHttpQueryHeaders should then be set to @extended value of the previous call). That way you get all cookies read.

Why would you want to specify referrer? Is that a requirement of some sort of the page that you automate? It can be added, but is it really necessary?

I was told servers don't usually deny access based on referrers so I don't think it would be a feature of utmost importance. In any case it would be easy to edit the udf as I said. Don't take me wrong, I am in not way critizicing your udf, it's great . In particular _WinHttpSimpleFormFill saved me from having to code all the process using the other functions from the udf and analyzing all the requests with live http headers

What I did not understand was the employ of @extended in the code you told me. In the help file it says that macro is used in functions like StringReplace -in which it shows the number of replacements performed- . Another thing is that the handle passed should be the one returned by _WinHttpOpenRequest() but is it the same to use the one returned by _WinHttpOpen in this case? (sorry for my ignorance, I don't know much about handles although I have the idea that they are like pointers to abstract data types).

Edit: sorry, I read another time the documentation and it clearly states that @extended shows the index of the next header so here is my guess:

$index = 0
Do
_WinHttpQueryHeaders( $hOpen, Default, $WINHTTP_QUERY_COOKIE, $index)
$index = @extended
Until @error = 1

I'm going to try it tomorrow and report back. Thanks again for your help!

[trancexx]

I meant this (said wrong option obviously):

#include "WinHttp.au3"
#include <Array.au3>

Global $hOpen = _WinHttpOpen()
Global $hConnect = _WinHttpConnect($hOpen, "google.com")
Global $hRequest = _WinHttpOpenRequest($hConnect)
_WinHttpSendRequest($hRequest)
_WinHttpReceiveResponse($hRequest)

;~ If _WinHttpQueryDataAvailable($hRequest)
Global $aCookies[100]
Global $iIndex = 0
While 1
    $aCookies[$iIndex] = _WinHttpQueryHeaders($hRequest, $WINHTTP_QUERY_SET_COOKIE, Default, $iIndex)
    If @extended = 0 Then ExitLoop
    $iIndex = @extended
WEnd
If $iIndex > 0 Then
    ReDim $aCookies[$iIndex]
    _ArrayDisplay($aCookies, "Cookie")
Else
    MsgBox(0, "Cookie", "No cookies for you here.")
EndIf

_WinHttpCloseHandle($hRequest)
_WinHttpCloseHandle($hConnect)
_WinHttpCloseHandle($hOpen)

[Mithrandir]

I tried the last code you posted but for some reason @extended kept stuck at 0 and so I was shown "no cookies for you here". Anyway, I modified it a bit and this worked:

#include <WinHttp.au3>
        #include <Crypt.au3>
        #include <Array.au3>

        Func _GetCookies(ByRef $hRequest)
    ;function that receives the handle returned by _WinHttpOpenRequest after using _WinHttpSendRequest and returns the cookies from the server's 
        ;response in an array which starts at index 0
        
    Local $aCookies[100]
    Local $iIndex = 0
    While 1
        $aCookies[$iIndex] = _WinHttpQueryHeaders($hRequest, $WINHTTP_QUERY_SET_COOKIE, Default, $iIndex)
        MsgBox(0, "Cookie: " & $iIndex, $aCookies[$iIndex])
        If StringCompare($aCookies[$iIndex], "") = 0 Then ExitLoop
        $array = StringRegExp($aCookies[$iIndex], "\w+?=[[:alnum:]]+?;", 3)
        If IsArray($array) Then
            $aCookies[$iIndex] = $array[0]
        EndIf
        $iIndex += 1
    WEnd
    If $iIndex > 0 Then
        ReDim $aCookies[$iIndex]
        _ArrayDisplay($aCookies, "Cookie")
    Else
        MsgBox(0, "Cookie", "No cookies for you here.")
    EndIf
    Return $aCookies
EndFunc   ;==>_GetCookies

But now I'm facing a problem with filling another field. I read the function _WinHttpSimpleFormFill again and I guess it is because the field is not an <input> field but a <textarea> one. Here's the code I tried:

#include <WinHttp.au3>
#include <Crypt.au3>
#include <Array.au3>

        $host = 'www.dreamsprojects.com'
    $hOpen = _WinHttpOpen("Mozilla/5.0 (Windows; U; Windows NT 5.1; es-ES; rv:1.9.2.10) Gecko/20100914 Firefox/3.6.13 (.NET CLR 3.5.30729)")
    $hConnect = _WinHttpConnect($hOpen, $host)
    _WinHttpSetOption($hConnect, $WINHTTP_OPTION_DISABLE_FEATURE, $WINHTTP_DISABLE_COOKIES)
    _WinHttpSetOption($hConnect,$WINHTTP_OPTION_DISABLE_FEATURE,$WINHTTP_DISABLE_REDIRECTS)
    $sUserName = "myusername"
    $sPassword = "mypassword"
    $MD5pass = StringLower(StringTrimLeft(_Crypt_HashData($sPassword,$CALG_MD5),2))
    $sRead = _WinHttpSimpleFormFill($hConnect, Default, "index:0", _
    "name:vb_login_username", $sUserName, _
    "name:vb_login_password", $sPassword, _
    "name:vb_login_md5password", $MD5pass)
    FileWrite(@ScriptDir&'/htmlofsubmittedform.html',$sRead)
    $sRead = _WinHttpSimpleFormFill($hConnect, "/foro/newthread.php?do=newthread&f=104", "name:vbform", _
    "subjecttitle", "prueba4", _
    "name:message", "cuarta prueba de hacer un post");I also tried the id: "vB_Editor_001_textarea", "cuarta prueba de hacer un post")
    FileWrite(@ScriptDir&'/htmlofsubmittedpost2.html',$sRead)
        _WinHttpCloseHandle($hConnect)
    _WinHttpCloseHandle($hOpen)

In the file 'htmlofsubmittedpost2.html' I saw that 'subjecttitle' field was filled but not 'name:message' one so I modified the function _WinHttpSimpleFormFill by adding the parameter $TextArea to the function (I did it quite in a hurry,I know it isn't very 'tidy' ):

_WinHttpSimpleFormFill(ByRef $hInternet, $sActionPage = Default, $sFormId = Default, $TextArea = Default, $sFieldId1 = Default, $sData1 = Default,

and I add some code at line 1172 so that it can recognize >textarea> fields:

Switch $sEnctype
            Case "", "application/x-www-form-urlencoded"
                $aInput = StringRegExp($sForm, "(?si)<\h*input\h*(.*?)/*\h*>", 3)
                If @error Then Return SetError(2, 0, "") ; invalid form
                #cs
                    This part is added so that <textarea can be filled
                #ce
                If $TextArea <> Default Then
                    $aInput2 = StringRegExp($sForm, "(?si)<\h*textarea\h*(.*?)/*\h*>", 3)
                    If @error Then Return SetError(2, 0, "") ; invalid form
                    _ArrayConcatenate($aInput, $aInput2)
                EndIf

By the way, I mimicked the conditional regular expression made by trancexx because in the documentation about regular expresions of autoit I read here which I get from the helpfile it isn't clear the sintax for me. In autoit I guess the condition starts by (?si) and the yes-pattern is after the '/' but I would appreciate any documentation or thread about conditional regex in autoit.

Now ontopic: With the modified _WinHttpSimpleFormFill I mentioned above I tried this code:

$host = 'www.dreamsprojects.com'
    $hOpen = _WinHttpOpen("Mozilla/5.0 (Windows; U; Windows NT 5.1; es-ES; rv:1.9.2.10) Gecko/20100914 Firefox/3.6.13 (.NET CLR 3.5.30729)")
    $hConnect = _WinHttpConnect($hOpen, $host)
    _WinHttpSetOption($hConnect, $WINHTTP_OPTION_DISABLE_FEATURE, $WINHTTP_DISABLE_COOKIES)
    _WinHttpSetOption($hConnect,$WINHTTP_OPTION_DISABLE_FEATURE,$WINHTTP_DISABLE_REDIRECTS)
    $sUserName = "myusername"
    $sPassword = "mypassword"
    $MD5pass = StringLower(StringTrimLeft(_Crypt_HashData($sPassword,$CALG_MD5),2))
    $sRead = _WinHttpSimpleFormFill($hConnect, Default, "index:0", Default, _
    "name:vb_login_username", $sUserName, _
    "name:vb_login_password", $sPassword, _
    "name:vb_login_md5password", $MD5pass)
    FileWrite(@ScriptDir&'/htmlofsubmittedform3.html',$sRead)
    $sRead = _WinHttpSimpleFormFill($hConnect, "/foro/newthread.php?do=newthread&f=104", "name:vbform", "textarea", _ ;I used "textarea" but could be any string
    "subjecttitle", "prueba4", _
    "name:message", "cuarta prueba de hacer un post")   
    FileWrite(@ScriptDir&'/htmlofsubmittedpost3.html',$sRead)
    _WinHttpCloseHandle($hConnect)
    _WinHttpCloseHandle($hOpen)

and now it filled the textarea field with "cuarta prueba de hacer un post" but it didn't send the form. Instead it seems to have clicked the 'preview post button' because in the $sRead it shows the preview of the message :S I appreciate any help, maybe I mess with modyfing _WinHttpSimpleFormFill as I didn't understand completely the For at line 1223 (of the original WinHttp.au3) that comes after what I added:

For $i = 0 To UBound($aInput) - 1
                    $aArray = StringRegExp($aInput[$i], '(?i).*?id\h*=(\h*"(.*?)"|' & "\h*'(.*?)'|" & '(.*?)(?: |\Z))', 3) ; e.g. id="abc" or id='abc' or id=abc
                    If Not @error Then $sInputId = $aArray[UBound($aArray) - 1]

But I don't think it has something to do with that because the field was filled but the submit button wasn't correctly pressed unlike in the first try where the submit button must have been pressed because I got a 'message too short' message in $sRead because the textarea field of the message had not been filled.

Thanks for your help!

EDIT: Sorry now I found this post: and I'll try it. Also, I noticed I had version 1.6.1.7 and not 1.6.1.8 of WinHttp.au3 so I will try also with the latest one

Edited January 5, 2011 by Mithrandir

[Mithrandir]

I tried with version 1.6.1.8 and it neither clicks the submit button but apparently the preview button. Also I found that as you can see in the file that I attached called 'htmlofsubmittedpost2.html' the notification that the message is too short ('El mensaje que has ingresado es muy corto. Por favor alarga tu mensaje a por lo menos 8 caracteres.') was in the page that returned _WinHttpSimpleFormFill while if I submit the post manually the notification is in a popup. Anyway that happened when the <textarea> field was not filled but now it is but it seems the submit button is not correctly identified as you can see in the file I attached called 'htmlofsubmittedpost4.html'.

I think that if I knew how is the button to submit the form identified in _WinHttpSimpleFormFill I could add a parameter to force the detection of the button to submit by looking at the source page. Or maybe I'm too used to IE.au3 udf and I'm mixing things. I will re-read the function to see what can I do and as always, I appreciate any help

htmlofsubmittedpost2.html

htmlofsubmittedpost4.html

[trancexx]

You are right about buttons. Every button is clicked with the current code.

I'll think of something for the next time UDF is updated.

[Mithrandir]

You are right about buttons. Every button is clicked with the current code.

I'll think of something for the next time UDF is updated.

Yes I debugged the function by doing this at line 1310 aprox:

_WinHttpAddRequestHeaders($hRequest, "Accept: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,*/*;q=0.5")
_WinHttpAddRequestHeaders($hRequest, "Accept-Charset: utf-8;q=0.7")
#cs
   The following is to debug
#ce
MsgBox(0, "Data to be sent: ", $sAddData)       
#cs
#ce
_WinHttpSendRequest($hRequest, Default, $sAddData)
_WinHttpReceiveResponse($hRequest)

And I saw that it was being sent

subject=prueba4&threadjaq=&description=&message=cuarta+prueba+de+hacer+un+post&wysiwyg=1&taglist=prueba4%2C&sendtrackbacks=&iconid=0&iconid=20&iconid=15&iconid=21&iconid=22&iconid=41&iconid=37&iconid=19&iconid=56&iconid=55&iconid=54&iconid=98&iconid=58&iconid=59&iconid=45&iconid=46&iconid=82&iconid=83&iconid=97&iconid=96&iconid=95&iconid=94&iconid=93&iconid=92&iconid=91&iconid=90&iconid=89&iconid=88&iconid=87&iconid=86&iconid=85&iconid=84&iconid=47&iconid=48&iconid=61&iconid=60&iconid=79&iconid=78&iconid=77&iconid=76&iconid=75&iconid=74&iconid=73&iconid=72&iconid=71&iconid=62&iconid=63&iconid=64&iconid=50&iconid=51&iconid=52&iconid=53&iconid=57&iconid=70&iconid=69&iconid=68&iconid=67&iconid=66&iconid=65&iconid=1&s=&securitytoken=1294289670-fa929fa371902afed8e853b000754abcaaaaaaaaaaa&f=104&do=postthread&posthash=d4f39e1c5371cf797e130aaaaaaaaaaa&poststarttime=1294289670&loggedinuser=32163&sbutton=Enviar+Nuevo+Tema&preview=Vista+Previa+de+Mensaje&parseurl=1&vbseo_retrtitle=1&vbseo_is_retrtitle=1&disablesmilies=1&emailupdate=&postpoll=yes&polloptions=4&sbutton=Enviar+Nuevo+Tema&preview=Vista+Previa+de+Mensaje

As you can see it sends all the radio buttons for the smilies as well as both the preview buttons and submit button. These last ones are sent twice because the form actually has two of each of these buttons.

Anyway I found a workaround by adding this code at line 1310 aprox:

_WinHttpAddRequestHeaders($hRequest, "Accept-Charset: utf-8;q=0.7")
        #cs
            The following is to solve the issue of submitting the wrong submit button and also other &data that was being sent when they shouldn't
        #ce
        ;MsgBox(0, "Data to be sent: ", $sAddData)
        ;FileWrite(@ScriptDir & "/databeforereplace.txt", $sAddData)
        $sAddData = StringReplace($sAddData, "&preview=Vista+Previa+de+Mensaje", "")
        $sAddData = StringReplace($sAddData, "&postpoll=yes", "")
        $sAddData = StringRegExpReplace($sAddData, "&iconid=[^0^\W]{1,2}", "")
        $sAddData = StringReplace($sAddData, "&iconid=000000", "&iconid=0");Added this because I don't know why after all the replacements it left &iconid=000000 
        If StringInStr($sAddData, "&sbutton=Enviar+Nuevo+Tema") Then
            $sAddData = StringReplace($sAddData, "&sbutton=Enviar+Nuevo+Tema", "")
            $sAddData = $sAddData & "&sbutton=Enviar+Nuevo+Tema"
        EndIf
        ;FileWrite(@ScriptDir & "/dataafterreplace.txt", $sAddData)
        MsgBox(0, "Data to be sent: ", $sAddData)       
        #cs
        #ce
        _WinHttpSendRequest($hRequest, Default, $sAddData)

Anyway your _WinHttpSimpleFormFill does what it is supposed to do: It's a function to fill simple forms and it's great! Maybe I/you can make another function that deals with complex forms. The important part is to get the posthash and securitytoken from the html code of the form and then the submit or whatever button that the user wants to be clicked that can be passed as a parameter like in IE.au3 _IEForm functions. Now I understand (a bit) better the regexp that you used but I have to learn more about them still mainly conditional regexp.

Edited January 7, 2011 by Mithrandir

[trancexx]

It doesn't matter how complex the form is, it's always the same procedure. I've already committed updates for the function to be able to properly process "submit" control(s). You/anyone can test it with WinHttp.au3 from here. That's a dev version.

I plan to add proper "radio" and "checkbox" input types handling and then make new release. It shouldn't take long.

Proper is proper.