Url to download file is different than website url

[BrianTheLibrarian]

Hello I have a question. I am trying to download set of files in IE using InetGet to download them:

On the website the url to download is:

https://www.sitenamehere.com/Download/Solid-Blue-Theme

But whenever I used that url with InetGet the file it downloaded is a 50-60k file. The file that downloads is corrupted / won't open.  If I manually click on the download link and look at the download URL in IE it looks like:

https://downloads.sitenamehere.com/1/1060/1060405/1471022748_original_blue_theme_v2.pdf?CuL76J813d59M63Tzj4hasfwMK4B_H7uHqT0F6vEZ2VQxcQy_LFh_AHe7Zzu2U6z&file_name=SolidBlueTheme.pdf

Is there anything I can do to download the second url / find out what the second url is? I've got about 800 files to download and the url is different for every one of them.  I have been using sleep and using tabs / keystrokes to use the IE download popup at the bottom of the page, but that doesn't work consistently 100% of the time.

Any help would be greatly appreciated.  
 

[Danp2]

I'm sure it's possible, Are you able to share the actually site? If not, find a public one that can be used to demonstrate the issue.

[BrianTheLibrarian]

I was hesitant to share the url because its something I'm selling, but here is a free example

Website download url:

https://www.teacherspayteachers.com/Download/Gamification-Assessments-3465703

Actual file url:

https://downloads.teacherspayteachers.com/3/3465/3465703/fd8841b303a3705d56cfedd3278bc561?Sqio3qRJOWKrNllCbXeE-OD38_Djua8yrcAaEERXJKCnVRtrNT74JYbXn5TsTYIX&file_name=GamificationAssessments.pdf

 

[careca]

For 800 files, maybe download the whole domain. Else, scan every page for the download button and click it, move to next page and so on.

 

[Danp2]

What's the URL for the page containing the initial download link?

[BrianTheLibrarian]

23 hours ago, careca said:

For 800 files, maybe download the whole domain. Else, scan every page for the download button and click it, move to next page and so on.

 

There are hundreds of thousands of pages, it would be like downloading eBay. I have a list of URL of the pages I need to download from.  I've been trying to do the download button, but it skips files every once in a while.   

Here is the url for the download page:

https://www.teacherspayteachers.com/Product/Gamification-Assessments-3465703

 

[Danyfirex]

Seems to be you need credentials/cookies for this. Probably you can handle using WinHttp UDF or curl(commandline) or UDF.

Saludos

[careca]

before the firefox update i'd use mozrepl and the firefox udf, but now, im not sure.

[genius257]

Hi @BrianTheLibrarian.

Your question is more JS related than AutoIt.

The page https://www.teacherspayteachers.com/Product/Gamification-Assessments-3465703

contains the download button with the url: https://www.teacherspayteachers.com/FreeDownload/Gamification-Assessments-3465703

that page loads the script: https://static-assets.teacherspayteachers.com/js-min/customer/items/free_download.js?rand=1515580196 and one of the last lines contains: window.location=cfg.free_download.link

if you open the developer console in the browser of your coosing on page: https://www.teacherspayteachers.com/FreeDownload/Gamification-Assessments-3465703 and run the line cfg.free_download.link you will get the link: https://www.teacherspayteachers.com/Download/Classroom-Gamification-Assessments-3465703 that link will send status code 302 Moved Temporarily and send your browser to the link: https://downloads.teacherspayteachers.com/3/3465/3465703/fd8841b303a3705d56cfedd3278bc561?Sqio3qRJOWKrNllCbXeE-OD38_Djua8yrcAaEERXJKCnVRtrNT74JYbXn5TsTYIX&file_name=GamificationAssessments.pdf

So you can either get the JS variable from the download site, or just take the URL https://www.teacherspayteachers.com/Download/<The string after /FreeDownload/ on first link> if it works as i think.

Edited January 10, 2018 by genius257