IE11 "Save as" for file download

[NicSol]

I know this issue has been brought up a few times, but most of the solutions I've found in the forum involve mouse clicks and key sending that I'd like to avoid if possible. Basically, I am using the "_IE" UDFs to automate some very repetitive tasks on a website that runs scientific calculations. When the calculations are done, a CSV file is offered for download. The problem is, a box like this pops up: and I have not been able to find a reliable way of selecting "Save as". I read that the UI Automate scripts can be used for this, but for the life of me, I cannot find the example scripts to which people keep referring. Can anyone provide me a link? Or does anyone have another reliable way of doing this?

Thanks in advance!

Just to save time on discussion:
- I can't use InetGet() because I have to log in to the site, and it's not a simple HTTP AUTH
- I've tried using cURL and wget with cookies, but the site uses an apparently complex CSRF scheme (it runs the Yii framework), and I'm having trouble getting cURL and wget to play nicely with it. I've also tried WinHTTP, but I seem to be running into the same problems. That's why I've gone back to trying to use IE directly.

[JohnOne]

It would greatly help your cause if you were to post the code you tried with each of the methods you noted.

[NicSol]

It would greatly help your cause if you were to post the code you tried with each of the methods you noted.

Thanks, John. I was actually trying to keep the scope of the question narrow: specifically in IE11, how do you "Save as", and is there a way to use UI Auotmate? (I've seen a couple references to an example in the forum, but after hours of searching, I cannot find it).

Here's an example of my attempt with WinHttp, based on code I found elsewhere on the forum:

expandcollapse popup

#include "WinHttp.au3"

Global Const $sUserName = "name"
Global Const $sPassword = "pass"

; Initialize and get session handle
$hOpen = _WinHttpOpen()
; Get connection handle
$hConnect = _WinHttpConnect($hOpen, "site.com", $INTERNET_DEFAULT_HTTPS_PORT)

; Fill login form:
$sRead = _WinHttpSimpleFormFill($hConnect, _
        "basic/web/index.php\?r=site%2Flogin", _
        "login-form", _
        "name:_csrf", "akQtTExDbUUJHmMrKgdYIw0TbDsYDiYdKQZdA2EQCy4FLBopOjYALA%3D%3D", _
        "name:LoginForm[username]", $sUserName, _
        "name:LoginForm[password]", $sPassword, _
        "name:LoginForm[rememberMe]", 1, _
        "name:login-button", "submit" _
        )

; You can check both @error and inspect $sRead to see if login was good
ConsoleWrite($sRead & @CRLF)
ConsoleWrite(@error & @CRLF)

; Close connection handle
_WinHttpCloseHandle($hConnect)

; Open new connection handle
$hConnect = _WinHttpConnect($hOpen, "mysite.com", $INTERNET_DEFAULT_HTTPS_PORT)

$vXLS = _WinHttpSimpleSSLRequest($hConnect, _
        Default, _
        "basic/web/index.php\?r=calculation%2Fdownload&id=5900", _
        Default, _
        Default, _
        Default, _
        Default, _
        Default) ; BINARY

; Close connection handle
_WinHttpCloseHandle($hConnect)
; Close session handle
_WinHttpCloseHandle($hOpen)

; Do whatever with this data, save to file or just print to console like I'm doing here
ConsoleWrite($vXLS & @CRLF)

The _csrf value was obtained by visiting the login page in a browser and finding the value in the hidden form field in the page  This takes me either to the site's home page or to an error 400 page (saying "Unable to verify your data submission.") . I can get into the wget and cURL alternatives I've tried, but the idea was essentially the same (i.e. load the page, get the csrf and cookies, and then make another request [sending the cookies back, along with the csrf token] to login). So far, I have not even been able to log in, let alone download a file with either of these methods. But this really isn't an AutoIT question at that point, it's a cURL and wget issue...

Regarding my attempts with IE, I don't think they'd add much to the discussion. Basically, I have been clicking at the coordinates of the button, but it's a terrible hack. The problem is, I am not able to find any way using the autoIT window info tool to get a handle to the control.

[mLipok]

Is this file have always the same name ?
Or do you can check the file name before you download the file ?

 

[NicSol]

Hi mLipok

No, the file name offered by the server depends on a number of things and is different each time. I check it before download based on the link offered by the webpage...

[trancexx]

I can help you with WinHttp.au3, but that would be off topic. Create new thread if you want. _csrf isn't static value, it's generated per session, that's why your code doesn't work, I assume.

[LarsJ]

I think it's this UI Automation example, you're looking for.

Edited July 8, 2015 by LarsJ

[PACaleala]

You could ask Windows to help. Read this : 

http://helpdeskgeek.com/how-to/re-enable-always-ask-before-opening-this-type-of-file-check-box-in-windows/

 

[NicSol]

I can help you with WinHttp.au3, but that would be off topic. Create new thread if you want. _csrf isn't static value, it's generated per session, that's why your code doesn't work, I assume.

Thank you for offering. I would be interested to know regardless, so if you'd be willing to work on it with me i would be highly appreciative. I'll open a new thread.

I think it's this UI Automation example, you're looking for.

Thanks, this indeed seems to be the thread I could not find (links in old posts were broken, and somehow I did not come across this as I combed through the pages). It seems to work perfectly for the "Save" function. I will try to better understand how it works and see if I can modify/extend it to work on the "Save as" option.

You could ask Windows to help. Read this : 

http://helpdeskgeek.com/how-to/re-enable-always-ask-before-opening-this-type-of-file-check-box-in-windows/

 

Thanks. I'd considered going that route at one point also, but it seems that it was made much more difficult in IE11, and then outright impossible. See this page, and the comments on it saying it no longer works (which was also my experience):

http://jwcooney.com/2014/03/31/remove-internet-explorer-open-or-save-popup/

Edited July 9, 2015 by NicSol

[NicSol]

Thank you for offering. I would be interested to know regardless, so if you'd be willing to work on it with me i would be highly appreciative. I'll open a new thread.

On second thought, discussing how to login to a site with csrf tokens (which would have to be part of the discussion) seems to violate the forum rules...or am I misinterpreting?

[trancexx]

You are misinterpreting, because you are overthinking. Your code is like that too.
Things are often simple.

Edited July 9, 2015 by trancexx

[NicSol]

Things are often simple.

Heheh, you inspired me to try again, and I think I finally got it working. I'm not 100% certain what my problem was before (it was not that the token was static -- I was just manually inserting it from loading the page in a browser right before running the script), but the solutions below seem to work at least for downloading a file requiring login (in my case it also must be SSL). I wouldn't say this is terribly original, but hopefully this will help someone out.

So far I have not been able to wrap my head around the code for IE, but I'm interested in figuring that out at some point, so I will post here if I figure it out.

Thanks to everyone for your help, and thanks to trancexx for the WinHTTP UDF.

----

For me, the key to getting this working was using the Live HTTP Headers plugin for Firefox (http://livehttpheaders.mozdev.org/). Go to the options menu and delete all the cookies from the site so you're starting with a clean slate. Then visit the login page and log in. Then very carefully examine what info is being sent, how it is encoded, etc. Using the two methods below (cURL command line and WinHTTP), you don't really need to worry about the cookies, but it's helpful to at least see what's going so you can see how far you're getting during trouble shooting. Note that many of the cookies set may be "HTTPOnly" cookies that are short-lived and will not show up in your browser's list of cookies. So this is really the only way you can see them.

Method 1 - cURL

• Get the cURL command line exe (http://curl.haxx.se/download.html); there is also an autoIT interface to the cURL libraries you could use (https://www.autoitscript.com/forum/topic/137456-curl-udf-a-udf-for-transferring-data-with-url-syntax/) with the same strategy as below, but you'd obviously have to code it yourself. Put curl.exe on your PATH or in the same folder as your script.
• The basic idea is to go to the login page, get the CSRF token, login, and then get the file you need
• command line options: -c saves the cookies to a file. -b loads the cookies from a file. -o writes the page you visit to a file. -D writes the response headers to a file (for debugging). -d specifies the POST data for the login form
• This is just to give you the concept. You will have to modify how you parse the CSRF token and the POST data for your particular case (based on the headers you find with the Live HTTP Headers plugin).

;get the csrf token by visiting the login page
RunWait("curl -o initrequest.txt -D headers1.txt -c cookies.txt https://site.com/login")
$fh = FileOpen("initrequest.txt", 0);
$s_page1 = FileRead($fh);
FileClose($fh);

;parse the token from the page
$aMatch = StringRegExp($s_page1, 'name="_csrf" value="([^"]+)"', 3)
$s_csrf = StringReplace($aMatch[0], '=', '%3D')
$s_csrf = StringReplace($s_csrf, '/', '%2F')

ConsoleWrite("CSRF = " & $s_csrf & @CRLF)

;build the post data string -- this can basically be copied from the Live HTTP Headers window
$s_post_data = "_csrf=" & $s_csrf & "&LoginForm%5Busername%5D=USERNAME&LoginForm%5Bpassword%5D=PASSWORD" _
               & "&LoginForm%5BrememberMe%5D=0&LoginForm%5BrememberMe%5D=1&login-button=";

;log in
RunWait('curl -o loginpage.txt -D headers2.txt -b cookies.txt -c cookies.txt -d "' & $s_post_data & '"  https://site.com/login')
;download
RunWait('curl -o download.txt -D headers3.txt -b cookies.txt -c cookies.txt  https://site.com/file')

Method 2 - WinHTTP

• This method only requires WinHTTP.au3 (https://github.com/dragana-r/autoit-winhttp/releases)
• Same idea as above, but it handles all the cookies internally and you don't need to create files to hold them. However, this may make it a little harder to debug (I'm sure there's a way to look at the cookies, but I don't know it). I found it helpful to get things working with cURL and then translate it to WinHTTP for my actual script.
• Again this is just showing you the principle. You'll need to modify this for your particular case, and include error checking as necessary

#include "WinHttp.au3"

$s_uname = 'user'
$s_pword = 'pass'
$s_site = 'site.com'
$s_login_path = 'path/to/login'
$s_file_path = 'path/to/file'

; Initialize and get session handle
$hOpen = _WinHttpOpen()
; Get connection handle
$hConnect = _WinHttpConnect($hOpen, $s_site)

; Navigate to the page and get the CSRF token
$s_page1 = _WinHttpSimpleSSLRequest($hConnect, 'GET', $s_login_path)
; will depend on the site what the regexp should be...
$a_match = StringRegExp($s_page1, 'name="_csrf" value="([^"]+)"', 3)
;URL encode the token (this simple method works for my site, you may need to be more thorough)
$s_csrf = StringReplace($a_match[0], '=', '%3D')
$s_csrf = StringReplace($s_csrf, '/', '%2F')

; log In (your POST parameters will depend on your specific website)
$s_post_data = '_csrf=' & $s_csrf & '&LoginForm%5Busername%5D=' & $s_uname & '&LoginForm%5Bpassword%5D=' & $s_pword _
               & '&LoginForm%5BrememberMe%5D=0&LoginForm%5BrememberMe%5D=1&login-button=';
$s_page2 = _WinHttpSimpleSSLRequest($hConnect, 'POST', $s_login_path, Default, $s_post_data)


;get file data
$s_data = _WinHttpSimpleSSLRequest($hConnect, 'GET', $s_file_path)
;write the data to the file
FileWrite('Data.csv', $s_data)

; Close handles
_WinHttpCloseHandle($hConnect)
_WinHttpCloseHandle($hOpen)

 

Edited July 10, 2015 by NicSol
Forgot to include link to cURL

[trancexx]

That's great. I'm glad you did it without the need to use half of the system's resources :D.

However it's even simpler than that. I bet this works too:

#include "WinHttp.au3"

$s_uname = 'user'
$s_pword = 'pass'
$s_site = 'https://site.com'
$s_login_path = 'path/to/login'
$s_file_path = 'path/to/file'

; Initialize and get session handle
$hOpen = _WinHttpOpen()
; Get connection handle
$hConnect = _WinHttpConnect($hOpen, $s_site)

; Fill the login form:
_WinHttpSimpleFormFill($hConnect, _
        $s_login_path, _
        "login-form", _
        "name:LoginForm[username]", $s_uname, _
        "name:LoginForm[password]", $s_pword)

$s_data = _WinHttpSimpleSSLRequest($hConnect, Default, $s_file_path)

; Close connection handle
_WinHttpCloseHandle($hConnect)
; Close session handle
_WinHttpCloseHandle($hOpen)

; Write the data to the file
FileWrite('Data.csv', $s_data)

 

 

Edited July 10, 2015 by trancexx

[NicSol]

However it's even simpler than that. I bet this works too:

No kiddin' -- it also works, thanks!

How does it work without explicitly sending the csrf token? I guess it's sent in the cookie automatically? From reading about the website's content-management system, I was under the impression that you had to send the proper token that matches the csrf value in the cookie, but I guess not. Anyway, thanks again for the help!

[trancexx]

No kiddin' -- it also works, thanks!

How does it work without explicitly sending the csrf token? I guess it's sent in the cookie automatically? From reading about the website's content-management system, I was under the impression that you had to send the proper token that matches the csrf value in the cookie, but I guess not. Anyway, thanks again for the help!

Token is send, no worries. _WinHttpSimpleFormFill does it internally for you.
_WinHttpSimpleFormFill is super advanced function.

[NicSol]

_WinHttpSimpleFormFill is super advanced function.

Ah yes I see that now looking through the code. Very handy. Many thanks again.