Downloading files from https sites using inetget

Dante_t · July 25, 2017

Hi Guys, I need help. I have searched the forum before posting and i couldn't find anything. The code below works fine when downloading files from "http" sites, but when trying to download from "https" sites, no files are downloaded. I tried different sites and I experience the same problem everywhere. Is there something I'm missing or doing wrong? Please note that I'm not a programmer and I'm new to this. I'm just using logic wherever i can to get things done. your help will be highly appreciated.

#include <InetConstants.au3>
#include <MsgBoxConstants.au3>
#include <WinAPIFiles.au3>

; Download a file in the background.
; Wait for the download to complete.

Example()

Func Example()
; Save the downloaded file to the temporary folder.
Local $sFilePath = "d:\"

; Download the file in the background with the selected option of 'force a reload from the remote site.'
Local $hDownload = InetGet("https://en.wikipedia.org/wiki/HTTPS#/media/File:Internet2.jpg", $sFilePath& "Internet2.jpg", $INET_FORCERELOAD, $INET_DOWNLOADBACKGROUND)

    ; Wait for the download to complete by monitoring when the 2nd index value of InetGetInfo returns True.
    Do
        Sleep(250)
    Until InetGetInfo($hDownload, $INET_DOWNLOADCOMPLETE)

    ; Retrieve the number of total bytes received and the filesize.
    Local $iBytesSize = InetGetInfo($hDownload, $INET_DOWNLOADREAD)
    Local $iFileSize = FileGetSize($sFilePath&"Internet2.jpg")

; Close the handle returned by InetGet.
InetClose($hDownload)

    ; Display details about the total number of bytes read and the filesize.
    MsgBox($MB_SYSTEMMODAL, "", "The total download size: " & $iBytesSize & @CRLF & _
            "The total filesize: " & $iFileSize)

    ; Delete the file.
    ;FileDelete($sFilePath)
EndFunc   ;==>Example

Jos · July 25, 2017

Shouldn't that URL be: "https://upload.wikimedia.org/wikipedia/commons/d/da/Internet2.jpg" ?

Jos

MarxBros · July 25, 2017

Hi,

the problem imo is your link ..../File:Internet2.jpg (the File:)

Correct Downloadlink for this file seems to be:

https://upload.wikimedia.org/wikipedia/commons/thumb/d/da/Internet2.jpg/1024px-Internet2.jpg

I changed your script a little bit with error message, etc...

hth

sample_https_download.au3

Dante_t · July 25, 2017

I was just using Wikipedia as an example because to actually get the files from the site i want i have to login. the logging in and navigating to the correct web page part is sorted. I'm actually trying to download files from https://rda.ucar.edu/. file sizes are between 17 and 20 MB. I tried using your code but still no luck. I've seen people mentioning things like cookies for sites that require authentication, etc. I don't understand what they were saying but maybe that's where my problem is.

My login in code is attached. i will pm my credentials. link to the file i want to download is on the code.

login&navigate.au3

Danp2 · July 25, 2017

Maybe this will work for you --

MarxBros · July 25, 2017

Hi again,

just found this tip elsewhere here in the forum...

inetget ("https://" & $user & ":" & $pass & "@www.mywebsite.com)

can you try something like...

Local $hDownload = InetGet("https://" & $User & ":" & $Pwd & "@rda.ucar.edu/data/ds083.2/grib2/2017/2017.07/fnl_20170724_06_00.grib2", $sFilePath & "fnl_20170724_06_00.grib2", $INET_DOWNLOADWAIT);, $INET_FORCERELOAD)

...with your username and password?

good luck!

Dante_t · July 26, 2017

22 hours ago, Danp2 said:

Maybe this will work for you --

Hi Danp2, which method are you referring to? can you please share your code if ystill have?

Edited July 26, 2017 by Dante_t
spelling error

Dante_t · July 26, 2017

20 hours ago, MarxBros said:

Hi again,

just found this tip elsewhere here in the forum...

inetget ("https://" & $user & ":" & $pass & "@www.mywebsite.com)

can you try something like...

Local $hDownload = InetGet("https://" & $User & ":" & $Pwd & "@rda.ucar.edu/data/ds083.2/grib2/2017/2017.07/fnl_20170724_06_00.grib2", $sFilePath & "fnl_20170724_06_00.grib2", $INET_DOWNLOADWAIT);, $INET_FORCERELOAD)

...with your username and password?

good luck!

Will try this when i get home, thank you MarxBros

Dante_t · July 26, 2017

21 hours ago, MarxBros said:

Hi again,

just found this tip elsewhere here in the forum...

inetget ("https://" & $user & ":" & $pass & "@www.mywebsite.com)

can you try something like...

Local $hDownload = InetGet("https://" & $User & ":" & $Pwd & "@rda.ucar.edu/data/ds083.2/grib2/2017/2017.07/fnl_20170724_06_00.grib2", $sFilePath & "fnl_20170724_06_00.grib2", $INET_DOWNLOADWAIT);, $INET_FORCERELOAD)

...with your username and password?

good luck!

Hi MarxBros,

I tried this method and it doesnt seem to work. the code just runs continually without any file being downloaded.. how can i use IE object as Danp2 suggested above?

Regards

Danp2 · July 26, 2017

1 hour ago, Dante_t said:

Hi Danp2, which method are you referring to? can you please share your code if ystill have?

You create a hidden GUI containing an embedded IE object. Use this object to log into the website and navigate to the correct page. Retrieve links to be downloaded and then use InetGet to perform the download.

Like this --

; *******************************************************
; Download files from website
; *******************************************************
Func RetrieveFiles()
Local $oButton, $oElement, $oForm, $oFrame, $oTable, $aTableData, $oLinks, $oLink, $iNumLinks
Local $filename, $i, $trs, $href, $created, $array

    $IsDownloaded = False

    _FileWriteLog($LogFile, "Logging into website")

    ; Create embedded IE object
    $oIE = _IECreateEmbedded()

    ; Dummy GUI
    GUICreate("Dummy", 640, 580, _
        (@DesktopWidth - 640) / 2, (@DesktopHeight - 580) / 2)
    GUICtrlCreateObj($oIE, 10, 40, 600, 360)
    GUISetState(@SW_SHOW)

    ; Open instance of website
    _IENavigate ($oIE, $Url)

    If @error Then
        ErrNotify("_IENavigate error. Errorcode = " & @error)
        Return
    EndIf

    ; Log into website
    $oForm = _IEFormGetCollection($oIE,0)

    If @error Then
        ErrNotify("_IEFormGetCollection error. Errorcode = " & @error)
        Return
    EndIf

    $oElement = _IEFormElementGetObjByName($oForm, "user")

    If @error Then
        ErrNotify("_IEFormElementGetObjByName (user) error. Errorcode = " & @error)
        Return
    EndIf

    _IEFormElementSetValue($oElement,$User)

    $oElement = _IEFormElementGetObjByName($oForm, "password")

    If @error Then
        ErrNotify("_IEFormElementGetObjByName (password) error. Errorcode = " & @error)
        Return
    EndIf

    _IEFormElementSetValue($oElement,$Pass)

    _IEFormSubmit($oForm)
    _IELoadWait ($oIE)

    _FileWriteLog($LogFile, "Switching to Downloads tab")

    ; Switch to Downloads tab
    _IELinkClickByText($oIE, "Downloads")

    If @error Then
        ErrNotify("_IELinkClickByText (Downloads) error. Errorcode = " & @error)
        Return
    EndIf

    $oForm = _IEFormGetObjByName($oIE, "frmSearch")

    If @error Then
        ErrNotify("_IEFormGetObjByName (Downloads) error. Errorcode = " & @error)
        Return
    EndIf

    $oElement = _IEFormElementGetObjByName($oForm, "lstboxes")

    If @error Then
        ErrNotify("_IEFormElementGetObjByName (Downloads) error. Errorcode = " & @error)
        Return
    EndIf

    For $i = 0 To $oElement.Length - 1
        _IEFormElementOptionSelect($oElement, $i, 1, "byIndex", 1)
        _IELoadWait($oIE)

        _FileWriteLog($LogFile, "Retrieving links (" & $i & ")")

        $oLinks = _IEGetObjByName($oIE, "file_list_link", -1)

        If @error Then
            ErrNotify("_IEGetObjByName (Downloads) error. Errorcode = " & @error)
            Return
        EndIf

        $iNumLinks = @extended

        For $oLink In $oLinks
            $href = $oLink.href

            $filename = StringTrimLeft($href, StringInStr($href, "*", 0, -1))

            If StringRegExp($filename, "(?i)\.(?:tif|txt)$") Then
                ; Make sure file wasn't already processed
                If Not FileExists($ProcessedDir & $filename) Then
                    $IsDownloaded = True

                    _FileWriteLog($LogFile, "Retrieving file: " & $filename)

                    ; Retrieve file

                    ConsoleWrite($InDir & $filename & @CRLF)

                    InetGet($href, $InDir & $filename, 1)

                    If @error Then
                        ErrNotify("InetGet (Downloads) error. Errorcode = " & @error)
                        Return
                    EndIf
                EndIf
            EndIf
        Next

        ; Reselect the desired objects due to prior page reload
        $oForm = _IEFormGetObjByName($oIE, "frmSearch")

        If @error Then
            ErrNotify("_IEFormGetObjByName (Downloads) error. Errorcode = " & @error)
            Return
        EndIf

        $oElement = _IEFormElementGetObjByName($oForm, "lstboxes")

        If @error Then
            ErrNotify("_IEFormElementGetObjByName (Downloads) error. Errorcode = " & @error)
            Return
        EndIf
    Next

    _IELinkClickByText($oIE, "Log off")

    ; Release GUI
    GUIDelete()

    _FileWriteLog($LogFile, "Completed File Retrieval")
EndFunc

mikell · July 29, 2017

Huge credits to trancexx for the wonderful WinHttp.au3 , this works nice - using the *appropriate* credentials

#include "WinHttp.au3"

$sServerAddress = "https://rda.ucar.edu"
$sGeneratorLocation = "/cgi-bin/login"

; sorry, these are private
$sEmail = "xxx@xxx"
$sPassword = "xxxxxxxx"

$hOpen = _WinHttpOpen()

; get access cookie
$hConnect = _WinHttpConnect($hOpen, $sServerAddress)
_WinHttpSimpleSSLRequest($hConnect)
_WinHttpCloseHandle($hConnect) 

; build and fill the login form
$sForm = _
        '<form action="' & $sServerAddress & $sGeneratorLocation & '" method="post">' & _
        '   <input name="email" />' & _
        '   <input name="passwd" />' & _
        '   <input name="remember" />' & _
        '   <input name="do" />' & _
        '   <input name="url" />' & _
        '</form>'
$hConnect = $sForm 

$sReturned = _WinHttpSimpleFormFill($hConnect, $hOpen, _
        Default, _
        "name:email", $sEmail, _
        "name:passwd", $sPassword, _ 
        "name:remember", "on", _
        "name:do", "login", _
        "name:url", "/")
If @error Then
    MsgBox(4096, "Error", @error) ; ref. WinHttp.chm
Else
    MsgBox(4096, "OK", "login successful, now download")

; download
  $sFile = "fnl_20170724_06_00.grib2"
  $sTarget = "data/ds083.2/grib2/2017/2017.07/" & $sFile

  $hConnect = _WinHttpConnect($hOpen, $sServerAddress)
  $hRequest = _WinHttpOpenRequest($hConnect, "GET", $sTarget)  
  _WinHttpSendRequest($hRequest)
  _WinHttpReceiveResponse($hRequest)
  If _WinHttpQueryDataAvailable($hRequest) Then

    $headers = _WinHttpQueryHeaders($hRequest)
    $Length = StringRegExpReplace($headers, '(?s).*Content-Length:\h*(\d+).*', "$1") 
    Local $bChunk, $bData = Binary("")

    ProgressOn($sFile, "Downloading", "0 %")
    While 1
         $bChunk = _WinHttpReadData($hRequest, 2, 8192) ; binary
         If @error Then ExitLoop
         $bData &= $bChunk
         $percent = Int((BinaryLen($bData)/$Length)*100)
         ProgressSet($percent, "Downloading", $percent & " %")
     WEnd
     ProgressSet(100, "Done !", "100 %")
     FileWrite(@scriptdir & "\" & $sFile, $bData)
     Sleep(1000)
     ProgressOff()
  Else
      MsgBox(48, "", "connection error")
  EndIf

; close handles
_WinHttpCloseHandle($hRequest)
_WinHttpCloseHandle($hConnect)
_WinHttpCloseHandle($hOpen)
EndIf

Dante_t · July 31, 2017

Hi Danp2, i wanted to use your code but i just didnt know where to edit it for it to do what i require it to do. Thank you

Hi Mikell, I used this code. the logging in was successful. the file i would like to download is 17MB in size but this code only downloads 440bytes file. the content of the file downloaded is as follows:

"Your browser sent a request that this server could not understand.
Reason: You're speaking plain HTTP to an SSL-enabled server port.
Instead use the HTTPS scheme to access this URL, please"

Is there something im not doing right?

Regards

Dante

Edited July 31, 2017 by Dante_t
big qoutes

Dante_t · July 31, 2017

#include "WinHttp.au3"

$sServerAddress = "https://rda.ucar.edu"
$sGeneratorLocation = "/cgi-bin/login"

$sEmail = "xxxxxxx@xxxxxx"
$sPassword = "xxxxxxx"

$hOpen = _WinHttpOpen()

; collect access cookie first
$hConnect = _WinHttpConnect($hOpen, $sServerAddress)
_WinHttpSimpleSSLRequest($hConnect)
_WinHttpCloseHandle($hConnect) 

; build and fill the login form
$sForm = _
        '<form action="' & $sServerAddress & $sGeneratorLocation & '" method="post">' & _
        '   <input name="email" />' & _
        '   <input name="passwd" />' & _
        '   <input name="remember" />' & _
        '   <input name="do" />' & _
        '   <input name="url" />' & _
        '</form>'
$hConnect = $sForm 

$sReturned = _WinHttpSimpleFormFill($hConnect, $hOpen, _
        Default, _
        "name:email", $sEmail, _
        "name:passwd", $sPassword, _ 
        "name:remember", "on", _
        "name:do", "login", _
        "name:url", "/")
If @error Then
    MsgBox(4096, "Error", @error)
Else
    MsgBox(4096, "OK", "login successful, now download")





; download
  $sFile = "fnl_20170724_06_00.grib2"
  $sTarget = "data/ds083.2/grib2/2017/2017.07/" & $sFile

  _WinHttpCloseHandle($hConnect)
  $hConnect = _WinHttpConnect($hOpen, $sServerAddress)
  $hRequest = _WinHttpSimpleSendSSLRequest($hConnect, "GET", $sTarget)
  If _WinHttpQueryDataAvailable($hRequest) Then

    $headers = _WinHttpQueryHeaders($hRequest)
    $Length = StringRegExpReplace($headers, '(?s).*Content-Length:\h*(\d+).*', "$1") 
    Local $bChunk, $bData = Binary("")

    ProgressOn($sFile, "Downloading", "0 %")
    While 1
         $bChunk = _WinHttpReadData($hRequest, 2, 8192) ; binary
         If @error Then ExitLoop
         $bData &= $bChunk
         $percent = Int((BinaryLen($bData)/$Length)*100)
         ProgressSet($percent, "Downloading", $percent & " %")
     WEnd
     ProgressSet(100, "Done !", "100 %")
     FileWrite(@scriptdir & "\" & $sFile, $bData)
     Sleep(1000)
     ProgressOff()
  Else
      MsgBox(48, "", "connection error")
  EndIf

; close handles
_WinHttpCloseHandle($hRequest)
_WinHttpCloseHandle($hConnect)
_WinHttpCloseHandle($hOpen)
EndIf

Dante_t · July 31, 2017

Just now, Dante_t said:

#include "WinHttp.au3"

$sServerAddress = "https://rda.ucar.edu"
$sGeneratorLocation = "/cgi-bin/login"

$sEmail = "xxxxxxx@xxxxxx"
$sPassword = "xxxxxxx"

$hOpen = _WinHttpOpen()

; collect access cookie first
$hConnect = _WinHttpConnect($hOpen, $sServerAddress)
_WinHttpSimpleSSLRequest($hConnect)
_WinHttpCloseHandle($hConnect) 

; build and fill the login form
$sForm = _
        '<form action="' & $sServerAddress & $sGeneratorLocation & '" method="post">' & _
        '   <input name="email" />' & _
        '   <input name="passwd" />' & _
        '   <input name="remember" />' & _
        '   <input name="do" />' & _
        '   <input name="url" />' & _
        '</form>'
$hConnect = $sForm 

$sReturned = _WinHttpSimpleFormFill($hConnect, $hOpen, _
        Default, _
        "name:email", $sEmail, _
        "name:passwd", $sPassword, _ 
        "name:remember", "on", _
        "name:do", "login", _
        "name:url", "/")
If @error Then
    MsgBox(4096, "Error", @error)
Else
    MsgBox(4096, "OK", "login successful, now download")





; download
  $sFile = "fnl_20170724_06_00.grib2"
  $sTarget = "data/ds083.2/grib2/2017/2017.07/" & $sFile

  _WinHttpCloseHandle($hConnect)
  $hConnect = _WinHttpConnect($hOpen, $sServerAddress)
  $hRequest = _WinHttpSimpleSendSSLRequest($hConnect, "GET", $sTarget)
  If _WinHttpQueryDataAvailable($hRequest) Then

    $headers = _WinHttpQueryHeaders($hRequest)
    $Length = StringRegExpReplace($headers, '(?s).*Content-Length:\h*(\d+).*', "$1") 
    Local $bChunk, $bData = Binary("")

    ProgressOn($sFile, "Downloading", "0 %")
    While 1
         $bChunk = _WinHttpReadData($hRequest, 2, 8192) ; binary
         If @error Then ExitLoop
         $bData &= $bChunk
         $percent = Int((BinaryLen($bData)/$Length)*100)
         ProgressSet($percent, "Downloading", $percent & " %")
     WEnd
     ProgressSet(100, "Done !", "100 %")
     FileWrite(@scriptdir & "\" & $sFile, $bData)
     Sleep(1000)
     ProgressOff()
  Else
      MsgBox(48, "", "connection error")
  EndIf

; close handles
_WinHttpCloseHandle($hRequest)
_WinHttpCloseHandle($hConnect)
_WinHttpCloseHandle($hOpen)
EndIf

This code worked for me. Thank you Mikell for the solution and thank you everyone who tried to help. much appreciated.

Downloading files from https sites using inetget

Recommended Posts

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members

Similar Content