Multiple threads achieved with curl

[TheSaint]

The free curl.exe program is a good downloader program, but alas it does not do multi-threading.

However, you can with a bit of effort achieve the next best thing. That is to say, you can download multiple parts of a file simultaneously, and then join them together afterward.

The following is a script I have been testing with the GOG games store, using elements available through their API/SDK downloaded to a manifest file.

NOTE - A BIG THANKS to @TheDcoder for the '-L -#' command-line switches, that enabled me to finally be able to download files from GOG using curl.exe.

expandcollapse popup

#cs ----------------------------------------------------------------------------

 AutoIt Version: 3.3.14.2
 Author:         TheSaint

 Script Function:  Download a game file from GOG using curl.exe
    Template AutoIt script.  (multiple threads)

#ce ----------------------------------------------------------------------------

#include <Constants.au3>
#include <WindowsConstants.au3>
#include <Misc.au3>
#include <File.au3>
#include <Crypt.au3>

Global $begin, $bytes, $checksum, $curl, $divnext, $divone, $divthree, $divtwo, $downfold, $download, $downone
Global $downthree, $downtwo, $file, $fileone, $filesize, $filethree, $filetwo, $found, $gotten, $growth, $handle, $hash
Global $link, $params, $percent, $pid, $pid2, $pid3, $progress, $range, $secs, $speed, $taken, $tempfold, $URL

$bytes = "437597744"
$checksum = "571df424a0df6bfdbd805198cbe100de"
$curl = @ScriptDir & "\curl.exe"
$downfold = @ScriptDir & "\Files"
$file = "setup_crystal_project_demo_1.5.4.1_(64bit)_(72187).exe"
$filesize = "417 MB"
$tempfold = @ScriptDir & "\temp"
$URL = "/downloads/crystal_project_demo/en1installer0"

If Not FileExists($downfold) Then DirCreate($downfold)
If Not FileExists($tempfold) Then DirCreate($tempfold)

FileDelete($tempfold & "\*.*")

_Singleton("down-in-parts-thesaint", 0)

$divone = Ceiling($bytes / 3)
$divtwo = $divone + 1
$divnext = $divtwo + $divone
$range = $divtwo & '-' & $divnext
$divthree = $divnext + 1
$fileone = "part1.exe"
$filetwo = "part2.exe"
$filethree = "part3.exe"

If FileExists($curl) Then
    $downone = $tempfold & "\" & $fileone
    $link = "https://www.gog.com" & $URL
    $params = $curl & ' -L -# -r 0-' & $divone & ' -o "' & $downone & '" --cookie gogcom_cookies.txt "' & $link & '"'
    FileChangeDir(@ScriptDir)
    $pid = Run($params, "", @SW_SHOW)
    If @error = 0 Then
        $downtwo = $tempfold & "\" & $filetwo
        $params = $curl & ' -L -#  -r ' & $range & ' -o "' & $downtwo & '" --cookie gogcom_cookies.txt "' & $link & '"'
        $pid2 = Run($params, "", @SW_SHOW)
        $downthree = $tempfold & "\" & $filethree
        $params = $curl & ' -L -# -C ' & $divthree & ' -o "' & $downthree & '" --cookie gogcom_cookies.txt "' & $link & '"'
        $pid3 = Run($params, "", @SW_SHOW)
        ProgressOn("Download Progress Meter", $file, "0%", Default, Default, 16)
        While 1
            If ProcessExists($pid) = 0 And ProcessExists($pid2) = 0 And ProcessExists($pid3) = 0 Then ExitLoop
            Sleep(500)
            If $bytes <> 0 Then
                If $found = "" Then $found = FileExists($downone)
                If $found = 1 Then
                    ; Get the file sizes as they download
                    $handle = FileOpen($downone, 0)
                    DllCall("Kernel32.dll", "BOOLEAN", "FlushFileBuffers", "HANDLE", $handle)
                    FileClose($handle)
                    $handle = FileOpen($downtwo, 0)
                    DllCall("Kernel32.dll", "BOOLEAN", "FlushFileBuffers", "HANDLE", $handle)
                    FileClose($handle)
                    $handle = FileOpen($downthree, 0)
                    DllCall("Kernel32.dll", "BOOLEAN", "FlushFileBuffers", "HANDLE", $handle)
                    FileClose($handle)
                    ; Get the folder size as the files download
                    $growth = DirGetSize($tempfold)
                    If $growth <> 0 Then
                        If $begin = "" Then $begin = TimerInit()
                        $progress = $growth
                        $progress = ($progress / $bytes) * 100
                        $percent = Floor($progress) & "%"
                        $taken = TimerDiff($begin)
                        $secs = $taken / 1000
                        ; Folder Size Growth
                        $gotten = $growth / 1024
                        If $gotten < 1024 Then
                            ; Kilkobytes downloaded
                            $growth = Floor($gotten) & " Kb"
                        Else
                            ; Downloaded
                            $growth = $gotten / 1024
                            If $growth < 1024 Then
                                ; Megabytes downloaded
                                $growth = Round($growth, 1) & " Mb"
                            Else
                                ; Gigabytes downloaded
                                $growth = Round($growth / 1024, 2) & " Gb"
                            EndIf
                        EndIf
                        ; MBs
                        $gotten = $gotten / 1024
                        $speed = Round($gotten / $secs, 1) & " Mb/s"
                        If $speed < 1 Then $speed = Floor($speed * 1024) & " Kb/s"
                        ProgressSet($progress, $filesize & "  --  " & $percent & "  --  " & $growth & "  --  " & $speed)
                    EndIf
                EndIf
            EndIf
        WEnd
        ProgressSet(100, $filesize & "  --  " & $percent & "  --  " & $growth & "  --  " & $speed, "Complete")
        Sleep(2000)
        ProgressSet(100, "Done", "Complete")
        Sleep(2000)
        If FileExists($downone) And FileExists($downtwo) And FileExists($downthree) Then
            ProgressSet(100, "Done", "Joining")
            $download = $downfold & "\" & $file
            RunWait(@ComSpec & ' /c COPY /B "' & $downone & '" + "' & $downtwo & '" + "' & $downthree & '" "' & $download & '"', "", @SW_SHOW)
            Sleep(1000)
            ProgressSet(100, "Done", "Validating")
            _Crypt_Startup()
            $hash = _Crypt_HashFile($download, $CALG_MD5)
            $hash = StringTrimLeft($hash, 2)
            If $hash = $checksum Then
                ProgressSet(100, "Download Result Checksum", "Passed")
            Else
                ProgressSet(100, "Download Result Checksum", "Failed")
            EndIf
            _Crypt_Shutdown()
        Else
            ProgressSet(100, "Download Result", "Missing File(s)")
        EndIf
        Sleep(3000)
        ProgressOff()
    Else
        MsgBox(262192, "Downloader Alert", "The process failed (i.e. no connection)!", 0)
    EndIf
Else
    MsgBox(262192, "Downloader Alert", "The 'curl.exe' file could not be found!", 0)
EndIf

Exit

For my script to work for you as is, you will need a cookie text file, plus have the Crystal Project Demo in your GOG account. But the principle shown, can be used beyond GOG of course, and have less requirements. To download other games in your account with GOG, you will need the manifest file data.

The easiest way to get a manifest file, is to use one of the third party downloaders for GOG ... gogcli.exe or gogrepo.py

A browser addon can get you a cookie text file.

PLEASE FEEL FREE TO SUGGEST IMPROVEMENTS  

i.e. There is likely a better way to refresh the download folder content without having to flush file buffers for each file.

Edited July 5 by TheSaint

[TheSaint]

My code (last post) is pretty basic, and you could certainly clean it up and make a function, maybe UDF out of it, where you specify a few things like the number of parts to download, etc. The totality of my code, is very specific to the GOG games store and games in your account with them. However, much of it could be tailored to other downloads from other sites.

My code is using data derived from a manifest file ... file name, file size (bytes and estimated), download link (referred) and an MD5 checksum value. The code also checks the resulting file against that MD5 checksum value. The script presents a floating progress bar, which also displays download speed and file size growth, etc.

Anyway, if nothing else, it was an interesting exercise, and at this point I am not utilizing it for anything more than an experiment ... though I have managed to download quite a few updates during my testing ... mostly for free demos.

I have been thinking about implementing it, because of the poor download speeds I currently get with GOG for a single thread. It wasn't that way for me for most of the last 7 years with GOG, but has been since their CDN crash mid last year (2023). Some days that speed is as low as 512 Kilobytes a second. On good days, or maybe certain times of the day, I can get a steady 1.3 Megabytes a second. I used to always get around 5 Megabytes a second. That was doable because I had a 50 Megabit connection, which has been upgraded in the last week to a 97 Megabit connection, so virtually double now. However, that hasn't helped with my download speed from GOG, with a single thread. With multiple threads it is a different story.

[TheDcoder]

@TheSaint You forgot to mention your bud who gave you the idea of using curl with byte ranges 😢

In any case, the vast majority of people won't need to use curl to do this sort of crippled multi-threaded downloading, there's a much better program for that called aria2 and it's a dedicated CLI downloader program and supports multi-threaded downloading natively... so it doesn't have to combine the separate parts later on and cause double the wear and tear as it would with curl

[TheSaint]

4 hours ago, TheDcoder said:

You forgot to mention your bud who gave you the idea of using curl with byte ranges

Gawd bud, your memory really is failing. I well remember the thoughts in my head, when checking the available command-line options, and I first realized 'I might' be able to alternatively download a file in portions and then join them together. I had been trying to find an option that would reveal the real download URL with curl.

So you never said anything to me about byte ranges, I discovered I could do that on my own, after you provided the command-line for using aria2 and curl together, which failed for me with the aria2 program, so I experimented further just with curl. So thanks for the curl portion of the command-line, which was the tricky bit I'd not been able to work out previously.

And in fact, you warned me against using byte ranges after I told you I had a good speed improvement doing that, as you said it would thrash my hard drive too much, though I can't see much difference myself between simultaneously downloading multiple files at a time. Sure I am creating and deleting more files, but I reckon you would need to do an awful lot of those to have any significant impact.

P.S. Anyway, I'll let you know how that wear and tear goes, mr. memory man.  

 

Edited July 4 by TheSaint

[TheSaint]

Today I decided to flesh out my code some more, adding improvements. Part of that is now specifying how many threads (parts) you want to download of a file. Anyway, I have added a bunch of helpful things. Most of my recent tests have been using 6 threads, which has worked real well with a great improvement to download speed. If you have a manifest file, you can also copy the data for each file, one at a time, to the clipboard, and the required values will be extracted from that automatically.

Here's some screenshots.

 

Here's the updated script

Download In Parts.au3

(1 prior download)

 

Edited July 5 by TheSaint

[TheSaint]

I have probably been remiss in not mentioning the following help I got from my good buddy TheDcoder, and now stated in my first post in this topic.

A BIG THANKS to @TheDcoder for the '-L -#' command-line switches, that enabled me to finally be able to download files from GOG using curl.exe.

They provided curl with the much needed redirect command for a path like the following.

https://www.gog.com/downloads/brok_the_investigator_prologue/en1installer0

P.S. I've also amended my script, by adding a thank you line to TheDcoder there, near the start.  

EDIT

By the way bud, I seem to recall you mentioning in the past, that there is a better way to get the growing file size, that didn't use "FlushFileBuffers". Or were you just referring to the _WinAPI_FlushFileBuffers command in the Help file? ... which unfortunately doesn't have an example, but I imagine does the same basic thing as the DLL command I use.

$handle = FileOpen($filepth, 0)
DllCall("Kernel32.dll", "BOOLEAN", "FlushFileBuffers", "HANDLE", $handle)
FileClose($handle)

Also ... I never did work out how to get curl.exe to return a value for me, of the full download URL from the redirect, so I could use it with some other downloader, that would let me gain the benefit of multi-streaming. Though it has now just occurred to me, that I might just be able to replace aria2 in the command-line example you gave me, with another program (i.e. Free Download Manager 5). The issue I had with aria2, according to my investigations, is possibly to do with Win 7 and 32 bit, and many other folk have the same issue from what I have discovered.

I will eventually of course, not be using Win 7, but it is just a convenience for me right now, until I get some space etc sorted.

Edited July 5 by TheSaint

[TheDcoder]

23 hours ago, TheSaint said:

Gawd bud, your memory really is failing. I well remember the thoughts in my head, when checking the available command-line options, and I first realized 'I might' be able to alternatively download a file in portions and then join them together.

Ah okay, I thought you got the idea after looking at the example I provided where I used the byte ranges trick to actually download 0 bytes while getting the final download URL

23 hours ago, TheSaint said:

Sure I am creating and deleting more files, but I reckon you would need to do an awful lot of those to have any significant impact.

The equation is pretty simple actually, you get double to wear and tear since you're writing double the data (once for downloading and once for combining). So if you download a large file the impact would be a lot more compared to downloading a small file.

10 hours ago, TheSaint said:

P.S. I've also amended my script, by adding a thank you line to TheDcoder there, near the start.  

Awww bud it's no biggie, don't mention it

10 hours ago, TheSaint said:

By the way bud, I seem to recall you mentioning in the past, that there is a better way to get the growing file size, that didn't use "FlushFileBuffers". Or were you just referring to the _WinAPI_FlushFileBuffers command in the Help file? ... which unfortunately doesn't have an example, but I imagine does the same basic thing as the DLL command I use.

It's been so long ago that I forgot the entire context of that bud...

10 hours ago, TheSaint said:

Also ... I never did work out how to get curl.exe to return a value for me, of the full download URL from the redirect

You couldn't get it to work? The first aria2 example command I gave you uses curl to fetch the final download URL. This is the command:

curl -L -s -w '%{url_effective}' -r 0-0 -o /dev/null --cookie $cookies_file $url

It will send the final URL to standard output while downloading 0 bytes (you can adjust value for the output (-o) switch since /dev/null doesn't exist in Windows)

[TheSaint]

On 7/5/2024 at 8:33 PM, TheDcoder said:

Ah okay, I thought you got the idea after looking at the example I provided where I used the byte ranges trick to actually download 0 bytes while getting the final download URL

To be honest bud, I don't recall that, possibly because I did not fully understand what was going on.

I knew it was getting the URL and passing it to aria2, and that was about it.

On 7/5/2024 at 8:33 PM, TheDcoder said:

You couldn't get it to work? The first aria2 example command I gave you uses curl to fetch the final download URL. This is the command:

curl -L -s -w '%{url_effective}' -r 0-0 -o /dev/null --cookie $cookies_file $url

It will send the final URL to standard output while downloading 0 bytes (you can adjust value for the output (-o) switch since /dev/null doesn't exist in Windows)

Okay, I am a bit clearer on things now, so I should hopefully be able to get that working for me. THANKS.

And no I couldn't get it working, but my memories are all tied up in all the attempts at getting aria2 to work anyway. And when that failed still after many tries, I then thought about just retrieving the true URL, but couldn't get a result for that either ... or maybe that is where it failed with aria2 ... so maybe we have been here before, and so I won't get it working.

P.S. By the way, in trying to get aria2 working I went and downloaded many prior versions of aria2 and tried them, hoping one would work.

On 7/5/2024 at 8:33 PM, TheDcoder said:

It's been so long ago that I forgot the entire context of that bud...

Basically, to get the growing size of a file being downloaded, so that we can show that in the progress bar, we need something to refresh the file regularly. If you don't refresh it, then it remains zero bytes until the download is finished, after which time the system updates things.

Some time back, years now probably, I asked in AutoIt Help, and someone provided me with the 'FlushFileBuffers' code I am using. Some time later, more than a year I am guessing, you happened to see that code in one of my scripts and replied that there was a better way.

In this latest case, i am monitoring more than one file as it downloads, so the folder content really, which is just limited to the parts of a file being downloaded. Currently for that to work properly, I need to refresh each part file as it downloads. So I am guessing there is probably a better way of just refreshing the folder content as a whole, rather than each individual file part?

On 7/5/2024 at 8:33 PM, TheDcoder said:

The equation is pretty simple actually, you get double to wear and tear since you're writing double the data (once for downloading and once for combining). So if you download a large file the impact would be a lot more compared to downloading a small file.

Of course. So in other words the same as downloading two similar files. So twice the impact on the drive. But in the bigger picture of things, that is not a lot really. Yes it would be better if I didn't have to do it, but the impact is not anywhere near enough for me to lose sleep over or worry terribly about how much quicker my drive will die. So I deem the impact as rather minor overall.

I've always done a lot of downloading and deleting and moving files around etc, and most of it won't involve this sort of thing. It's not like I download many files every day from GOG. Not that I have actually implemented it yet, and may never do so, just solely been testing things. It might be a good temporary thing though, until my PC situation improves. But if I can get that URL via curl and pass that to Free Download Manager for instance, which is a multi-thread program, then have it successfully download my GOG game files, then that would be the better solution, until I am able to take advantage of aria2. The problem with FDM5 is that it has extremely limited command-line support ... just IN (and maybe OUT) as I recall. If I can control OUT, then I am cooking with gas, as that means I can automate file validating etc. EVen if I cannot control where OUT will be, I know where the default is currently set, and can ultilise that. The other significant benefit, will be not having to grab every URL one by one like I currently have to, from my GOG library web page for a game. So even just that last benefit will be reward enough.

Edited July 6 by TheSaint

[TheSaint]

47 minutes ago, TheSaint said:

so maybe we have been here before, and so I won't get it working.

And so it seems. As I now recall all my experiments with the command-line you gave me.

The following is what I have just tried, in a BAT file, and it returns an error.

D:\Projects\GOG-CLI\Curl\curl.exe -L -s -w '%{url_effective}' -r 0-0 -o /dev/null --cookie gogcom_cookies.txt "https://www.gog.com/downloads/underground_blossom_demo/en1installer0"

So it shows the following in the console window.

Quote

curl: (2) no URL specified
curl: try 'curl --help' for more information
Press any key to continue . . .

And then and old memory surfaced about % being used in a BAT file. That sometimes you need to use %% instead of %.

And bugger me, that worked.  

Quote

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3590  100  3590    0     0   4184      0 --:--:-- --:--:-- --:--:--  4278
100  1072  100  1072    0     0    772      0  0:00:01  0:00:01 --:--:--     0
Warning: Failed to open the file /dev/null: No such file or directory
  0     1    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
curl: (23) Failure writing output to destination
'https://gog-cdn-fastly.gog.com/token=nva=blahblahblah~dirs=6~token=blahblahblahblah/secure/offline/blah/blah/blah/blah/setup_brok_t
he_investigator_-_prologue_1.3.5_%2864046%29.exe'

And so maybe now I can get aria2 to work ... if that was the issue.

Edited July 6 by TheSaint

[TheSaint]

No luck with aria2, and no luck with FDM5. Not even any luck with curl as just a single thread, as it complains about a missing port number. WTF

Testing with a BAT file only at this point. The following is my attempt with curl as the downloader, and where I get that port number error.

@echo off

"D:\Projects\GOG-CLI\Curl\curl.exe" -L -s -w '%%{url_effective}' -r 0-0 -o /dev/null --cookie gogcom_cookies.txt "https://www.gog.com/downloads/underground_blossom_demo/en1installer0" > Address.txt
set /p url=<Address.txt
echo %url%
D:\Projects\GOG-CLI\Curl\curl.exe -# -o "D:\Projects\GOG-CLI\Curl\Downloads\game1.exe" %url%
pause
cls
exit

I'd also tried piping the result as follows, without using a variable, but that gives the following error message.

@echo off

"D:\Projects\GOG-CLI\Curl\curl.exe" -L -s -w '%%{url_effective}' -r 0-0 -o /dev/null --cookie gogcom_cookies.txt "https://www.gog.com/downloads/underground_blossom_demo/en1installer0" | D:\Projects\GOG-CLI\Curl\curl.exe -# -o "D:\Projects\GOG-CLI\Curl\Downloads\game1.exe"
pause
cls
exit

Quote

curl: (2) no URL specified
curl: try 'curl --help' for more information
Press any key to continue . . .

 

[TheSaint]

Okay I had to give up on BATCH files and just use AutoIt, and now I am getting good results with the following.

#include <Constants.au3>
#include <WindowsConstants.au3>
#include <Misc.au3>
#include <File.au3>

Global $curl, $data, $downfold, $download, $file, $link, $params, $pid, $URL

_Singleton("curl-get-thesaint", 0)

$curl = @ScriptDir & "\curl.exe"
$downfold = @ScriptDir & "\Files"
$file = "setup_underground_blossom_demo_1.0_(67138).exe"
$download = $downfold & "\" & $file

FileChangeDir(@ScriptDir)

$URL = "/downloads/underground_blossom_demo/en1installer0"
$link = "https://www.gog.com" & $URL
$params = $curl & ' -L -s -w "%{url_effective}" -r 0-0 -o /dev/null --cookie gogcom_cookies.txt "' & $link & '"'
$pid = Run($params, "", @SW_SHOW, $STDOUT_CHILD)
ProcessWaitClose($pid)
$data = StdoutRead($pid)
$data = StringReplace($data, "%28", "(")
$URL = StringReplace($data, "%29", ")")
MsgBox(262144 + 64, "CURL Result", $URL)
$params = $curl & ' -# -o "' & $download & '" "' & $URL & '"'
$pid = Run($params, "", @SW_SHOW)

Exit

 

[TheSaint]

Still no luck with aria2. I keep getting the following error message.

Quote

07/07 00:37:56 [NOTICE] Downloading 1 item(s)

07/07 00:37:56 [ERROR] CUID#7 - Download aborted. URI=https://gog-cdn-fastly.gog.com/token=nva=blahblahblah~dirs=6~token=blahblahblahblah/secure/offline/blah/blah/blah/blah/setup_underground_blossom_demo_1.0_(671
38).exe
Exception: [AbstractCommand.cc:351] errorCode=1 URI=https://gog-cdn-fastly.gog.com/token=nva=blahblahblah~dirs=6~token=blahblahblahblah/secure/offline/blah/blah/blah/blah/setup_underground_blossom_demo_1.0_(67138
).exe
  -> [SocketCore.cc:1019] errorCode=1 SSL/TLS handshake failure: Error: The mess
age received was unexpected or badly formatted.
(80090326)

07/07 00:37:56 [NOTICE] Download GID#75ee81048a193795 not complete: D:/Projects/
GOG-CLI/Curl/Files/setup_underground_blossom_demo_1.0_(67138).exe

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
75ee81|ERR |       0B/s|D:/Projects/GOG-CLI/Curl/Files/setup_underground_blossom
_demo_1.0_(67138).exe

Status Legend:
(ERR):error occurred.

aria2 will resume download if the transfer is restarted.
If there are any errors, then see the log file. See '-l' option in help/man page
 for details.

D:\Projects\GOG-CLI\Curl>

The command I am using is the following.

$pid = Run(@ComSpec & " /k " & 'aria2c.exe -c -d "' & $downfold & '" -o "' & $file & '" -x 4 "' & $URL & '"')

And so we appear to be back at why I could never get aria2 to work for me ... not on this PC anyway.

[TheSaint]

But on the good result side, I can now successfully pass the URL to Free Download Manager 5 on the command-line, even if it is already running.

Alas though, I cannot specify the output folder via command-line.

So now I need to look at how I can leverage this to my best advantage.

EDIT

I just discovered that every download with FDM5 using this method (at least), gets a different PID. So now it looks like we might be starting to cook with gas, as that gives monitoring options. But alas, those PID are short lived. Even using RunWait does not wait until a download completes. So now I am stuck at needing some way to know when a download has completed. I could do the 'FlushFileBuffers' thing, but that seems to me it would have an element of presume. I need to know when FDM5 has finished with a download, so it can be relocated and then validated.

Edited July 6 by TheSaint

[TheDcoder]

On 7/5/2024 at 4:33 PM, TheDcoder said:

(you can adjust value for the output (-o) switch since /dev/null doesn't exist in Windows)

May I should've used "must" instead of "can" in this sentence

Either remove it entirely or change /dev/null to a proper file path of the file to be downloaded.

[TheSaint]

8 hours ago, TheDcoder said:

May I should've used "must" instead of "can" in this sentence

Either remove it entirely or change /dev/null to a proper file path of the file to be downloaded.

Well I never got to the point of doing that this time around.

And as I got the curl side of things working anyway, probably not an issue anyway. Though maybe I will try it to see if it makes a difference to aria2 ... probably not.

[TheSaint]

As I was waking today, and cogitating over things, I suddenly remembered that FDM5, like many downloaders, downloads to a temporary file first until the download has finished, after which I presume it just renames the temp file to the desired file name.

So it would appear I can just rely on file exists and byte size to determine when a download has completed.

Anyway, that is what I am testing right now, among other things.

EDIT

While that works, I am still needing to tweak things due to timing issues etc.

EDIT 2

Okay, I appear to have that sussed now. Time to investigate other options, etc.

1. See if I can determine via code, what download folder FDM5 is currently set to use. ONGOING - Seems it is set on a file by file basis, with there being a recorded history and the last folder used is the current one, and from what I can tell is only listed in a sqlite database entry, which one might be able to modify to get a changed result.

2. Download latest aria2 and see if they have fixed anything for my use. DONE - Already using the latest version from November 2023.

3. Look again, for another command-line multi-thread downloader program. DISCOVERED that wget2 now exists, and is multi-threaded, but only 64 bit. I was also reminded of httrack, which I might take another look at now.

Edited July 7 by TheSaint

[TheSaint]

Well I can be like a dog with a bone sometimes. I worry at it until I succeed.

I finally got aria2 to work, and yes it was Win 7 32 bit related.

Another search on my socket error, eventually brought me to the following posted issue.

https://github.com/aria2/aria2/issues/1801

So going by that I grabbed a static version of aria2, and tried that. That failed with a certificate error.

So as I had also read how to provide the certificate on the command-line, I did that ... and whamo it worked.

$params = $aria2 & ' -c -d "' & $downfold & '" -o "' & $file & '" -x ' & $threads & ' --ca-certificate="' & $cert & '" "' & $URL & '"'
$pid = RunWait($params, "", @SW_SHOW)

I'll provide the full script, once I have tidied a few things up, but could not contain my excitement right now, so had to share.  

Anyway, once again a BIG THANKS to my buddy @TheDcoder for his original suggestion and command-line and helpful comments along the way at various stages.

We got there in the end ... what a journey.  

Edited July 7 by TheSaint

[TheSaint]

On 7/7/2024 at 7:06 PM, TheSaint said:

I'll provide the full script, once I have tidied a few things up

Well, I've not done that yet, due to wanting to import the code to my GOGcli GUI program, which I have now done and tweaked and tested a lot, and it works well.

That is, the aria2 with curl portion of my code, and it took some tweaking to get it exactly how I wanted ... progress feedback and all.

At this stage, I have only done the preliminary work for Free Download Manager support, so not available for use yet. In a way, I felt I no longer need it, so I just did some elements as a start in case it is ever needed. I no longer feel quite that way, as I have come across a limitation with aria2 ... the numbers of threads you can use with GOG, which appears to be a maximum of five. Five threads will give me somewhere around 5 Megabytes a second, but usually less on average ... some days and times are clearly better than others.

On the other hand, I could get a fair bit more with Free Download Manager 5, which must be doing something tricky to get around that seeming 5 thread limitation. I know I can bump the threads in FDM5 up to about 11 and get close to 7 Megabytes a second at times.

The aria2 program reports on how many threads is in use, and I have never seen it above 5, and the speed indicates that too. So it is very clear when using FDM5, that it can be faster.

Here's the latest screenshot of the downloading window of my GOGcli GUI program, after I reworked it a bit to fit some new or changed elements in.

I've grabbed over 74 game updates in the last few days, most of them either demos or prologues ... 118 GB worth.

So finally I have made a start and dent on all the updates I need to get at GOG ... more than 100 still to go, of mostly full games ... so a lot of gigabytes.

Anyway ... I will eventually get back to tidying up that code example and providing here.

Edited July 11 by TheSaint

[TheSaint]

23 hours ago, TheSaint said:

I have only done the preliminary work for Free Download Manager support, so not available for use yet. In a way, I felt I no longer need it, so I just did some elements as a start in case it is ever needed.

No longer the case, as I couldn't resist completing the code today, and trying it out successfully with several game downloads. And after tweaking it a bit it works well. One of the files was a 9GB Linux version game file, which like the rest downloaded in a satisfying portion of time.

So now my program offers 4 different methods to download your purchases from GOG.

1. The original single thread gogcli.exe.
2. Single thread curl.exe.
3. Multi-thread aria2.exe with curl.exe.
4. Multi-thread fdm.exe  with curl.exe.

[TheDcoder]

On 7/12/2024 at 3:48 PM, TheSaint said:

So now my program offers 4 different methods to download your purchases from GOG.

Well bud guess what... do you know that there is a wget2 now? That's right, they made a literal 2nd part of wget, just like in the movies

It's a completely new program that was written from the ground up and it has many neat features:

Quote

# Features

A non-exhaustive list of features

- Support for HTTP/1.1 and HTTP/2.0 protocol
- [brotli](https://github.com/google/brotli) decompression support (Accept-Encoding: br)
- [zstandard](https://github.com/facebook/zstd) decompression support, RFC8478 (Accept-Encoding: zstd)
- [lzip](https://www.nongnu.org/lzip/) decompression support, (Accept-Encoding: lzip)
- HPKP - HTTP Public Key Pinning (RFC7469) with persistent database
- TCP Fast Open for plain text *and* for HTTPS
- TLS Session Resumption including persistent session data cache
- TLS False Start (with GnuTLS >= 3.5.0)
- HTTP2 support via nghttp2 and GnuTLS ALPN including streaming/pipelining
- OCSP stapling + OCSP server querying as a fallback (experimental, needs GnuTLS >= 3.3.11)
- Use [libpsl](https://github.com/rockdaboot/libpsl) for cookie domain checking (using Public Suffix List)
- Support link conversion (-k/--convert-links and -K/--backup-converted)
- Support for RFC 6266 compliant Content-Disposition
- RFC 6797 HSTS (HTTP Strict Transport Security)
- Support for bzip2 Content-Encoding / Accept-Encoding compression type
- New Year 2014 gimmick: added support for XZ Content-Encoding / Accept-Encoding compression type
- Character encoding of input files may be specified despite from local and remote encoding (--input-encoding)
- Support scanning RSS 2.0 feeds from local files (`--force-rss -i <filename>`)
- Support scanning RSS 2.0 feeds.
- Support scanning Atom 1.0 feeds from local files (`--force-atom -i <filename>`)
- Support scanning Atom 1.0 feeds.
- Support scanning URLs from local Sitemap XML file (`--force-sitemap -i <filename>`)
- Support scanning sitemap files given in robots.txt (Sitemap XML, gzipped Sitemap XML, plain text) including
sitemap index files.
- Support arbitrary number of proxies for parallel downloads
- Multithreaded download of single files (option --chunk-size)
- Internationalized Domain Names in Applications (compile-selectable IDNA2008 or IDNA2003)
- ICEcast / SHOUTcast support via library (see examples/getstream.c)
- respect /robots.txt "Robot Exclusion Standard" and `<META name="robots" ...>`
- new option --secure-protocol=PFS to have TLS only plus forcing Perfect Forward Secrecy (PFS)
- IDN support for international domains
- autotools support
- proxy support
- cookies (session/non-session), detection of supercookies via Mozilla Public Suffix List
  (use the new option `--cookie-suffixes <filename>`, better: put it into ~/.wgetrc)
- recursive download of websites with or without spanning hosts
- download of single web pages / resources
- zlib/gzip compressed HTTP/HTTPS downloads (gzip, deflate)
- number of parallel download threads is adjustable
- include directive for config files (wildcards allowed)
- support for keep-alive connections
- included CSS, HTML, XML parser needed for recursive downloads
- gettext support
- HTTPS via libgnutls (and basic WolfSSL support)
- support for Metalink RFC 6249 (Metalink/HTTP: Mirrors and Hashes)
- support for Metalink RFC 5854 (Metalink Download Description Format / .meta4 files)
- support for Metalink 3
- Metalink checksumming via libgnutls
- DNS lookup cache
- IPv4 and IPv6 support
- built and tested on Linux, OSX, OpenBSD, FreeBSD, Solaris, Windows

So it might be able to replace aria2 for you, give it a go. You can find the prebuilt EXE here: https://github.com/rockdaboot/wget2/releases/latest

I had completely forgotten about this until just a few days ago