Curl UDF - libcurl with x64 support

[Beege]

I modified the example above to also get the date,from,to in addition to the subject only. 

For anyone who has some subject lines that look like this below, I found this article: https://dmorgan.info/posts/encoded-word-syntax/

Subject: =?UTF-8?B?VG9wIFdhdGNoYXRob24gcGlja3Mgd2UgdGhpbmsgeW91J2xsIA==?=

[Beege]

This works for the base64 encoded subjects, but I'm not finding much info for the quopri encoding that article talks about (outside of python). I dont think any of the emails im working with are encoded with quopri so not sure its that common.

Func _encoded_word_syntax_test()
    $sTest = "=?UTF-8?B?VG9wIFdhdGNoYXRob24gcGlja3Mgd2UgdGhpbmsgeW91J2xsIA==?="
    $aEWS = StringRegExp($sTest, '=\?(.*)\?(.)\?(.*)\?=', 3); seperates charset, encoding, encoded-text
    ;_ArrayDisplay($aEWS)
    If $aEWS[1] = "B" Then; base64 encoded
        $sDecode = BinaryToString(_Base64Decode($aEWS[2]))
        ConsoleWrite($sDecode & @CRLF)
    Else
        ;quopri encoded
    EndIf
EndFunc

Func _Base64Decode($sB64)
    Local $iLen = DllCall("Crypt32.dll", "bool", "CryptStringToBinaryA", "str", $sB64, "dword", 0, "dword", 1, "ptr", NULL, "dword*", 0, "ptr", 0, "ptr", 0)[5]
    Local $tDecoded = DllCall("Crypt32.dll", "bool", "CryptStringToBinaryA", "str", $sB64, "dword", 0, "dword", 1, "struct*", DllStructCreate("byte[" & $iLen & "]"), "dword*", $iLen, "ptr", 0, "ptr", 0)[4]
    Return DllStructGetData($tDecoded, 1)
EndFunc   ;==>_Base64Decode

 

edit:

Actually this is better for the multi-line subjects extracting only base64 encoded text.

Also noticing there's something weird about the single quote character (') being used in a subject. It translates like its a special character or something
 

$sTest = "Subject: =?utf-8?B?V2VsY29tZSB0byBYRklOSVRZIENvbm5lY3QgRW1haWwhIExldA==?=" & @CRLF & _
        @TAB & "=?utf-8?B?4oCZcyBnZXQgc3RhcnRlZA==?="

$aEWS = StringRegExp($sTest, '(?m)^(?:Subject\:\h|\h)?=\?.*\?B\?(.*)\?=$', 3) ; grabs only the base64 portion from each line
If Not @error Then
    _ArrayDisplay($aEWS)
    Local $sDecode
    For $sEnc In $aEWS
        $sDecode &= BinaryToString(_Base64Decode($sEnc))
    Next
    ConsoleWrite($sDecode & @CRLF)
EndIf

 

Edit:

Fix for the single quote weirdness is to use the charset specified (utf-8 for this one) in the binarytostring() call options
 

$sDecode &= BinaryToString(_Base64Decode($sEnc), 4)

 

Edited June 21, 2022 by Beege

[Beege]

I was able to get the imap append example working to save sent items. This was a real pain in the ass due to that example being wrong in multiple ways so I got to rant a little bit. 

For starters it states twice that we are sending an email when we are not, we are just appending one.

/* <DESC>
 * IMAP example showing how to send emails
 * </DESC>
 
 /* This is a simple example showing how to send mail using libcurl's IMAP
 * capabilities.

They clarify this in the RFC

Note: The APPEND command is not used for message delivery,
      because it does not provide a mechanism to transfer [SMTP]
      envelope information.

Then we get to the guts of the example. This is wrong

curl_easy_setopt(curl, CURLOPT_URL, "imap://imap.example.com/100");

And will give you errors like this

;< A002 OK Logged in
;> A003 APPEND Sent/3 (\Seen) {125}
;< A003 NO [TRYCREATE] Mailbox doesn't exist: Sent/3 (0.001 + 0.000 secs).
;> A004 LOGOUT
;< * BYE Logging out

I ended up down a rabbit hole with TRYCREATE. If you read the rfc for that, it seems like everything is kinda doing what its suppose to be only the email client isn't re-asking to create the email.. 

If the destination mailbox does not exist, a server MUST return an
 error, and MUST NOT automatically create the mailbox.  Unless it
 is certain that the destination mailbox can not be created, the
 server MUST send the response code "[TRYCREATE]" as the prefix of
 the text of the tagged NO response.  This gives a hint to the
 client that it can attempt a CREATE command and retry the APPEND
 if the CREATE is successful.

 

Finally I found this link from 2016 reporting basically the same problem I'm seeing and got the syntax right. Same syntax I got in all the other examples. I should have just guessed. 

expandcollapse popup

; #FUNCTION# ====================================================================================================================
; Name ..........: Example_Email_Append
; Description ...: Appends an email to a directory. (Use to save sent emails in sent folder)
; Parameters ....: $sIMAPServer         - imap server address. ex: imap.comcast.net
;                  $sUser               - username/email address
;                  $sPass               - password
;                  $sTo                 - recepient address
;                  $sFrom               - sender address
;                  $sSubject            - email subject
;                  $sBody               - email body
;                  $sDirName            - Directory Name ex: Outbox, Sent Items, Sent
; Remarks .......: This does not send an email. Just creates a copy
; ===============================================================================================================================
Func Example_Email_Append($sIMAPServer, $sUser, $sPass, $sTo, $sFrom, $sSubject, $sBody, $sDirName = 'Sent')

    ;Build email
    Local $sHeaders = "To: <" & $sTo & ">" & @CRLF
    $sHeaders &= "From: <" & $sFrom & ">" & @CRLF
;~  $sHeaders &= "CC: <" & $sCC & ">" & @CRLF
;~  $sHeaders &= "BCC: <" & $sBCC & ">" & @CRLF
    $sHeaders &= "Subject: " & $sSubject & @CRLF

    Local $sEmail = $sHeaders & @CRLF & $sBody

    Local $Curl = Curl_Easy_Init()
    If Not $Curl Then Return

    Curl_Easy_Setopt($Curl, $CURLOPT_VERBOSE, 1) ;
    Curl_Easy_Setopt($Curl, $CURLOPT_USERNAME, $sUser) ;
    Curl_Easy_Setopt($Curl, $CURLOPT_PASSWORD, $sPass) ;
    Curl_Easy_Setopt($Curl, $CURLOPT_CAINFO, @ScriptDir & '\curl-ca-bundle.crt') ;
    Curl_Easy_Setopt($Curl, $CURLOPT_WRITEFUNCTION, Curl_DataWriteCallback())
    Curl_Easy_Setopt($Curl, $CURLOPT_WRITEDATA, $Curl)

    ;request next uid
    Curl_Easy_Setopt($Curl, $CURLOPT_CUSTOMREQUEST, 'STATUS "' & $sDirName & '" (UIDNEXT)')
    Curl_Easy_Setopt($Curl, $CURLOPT_URL, "imaps://" & $sIMAPServer)
    Local $Code = Curl_Easy_Perform($Curl)
    If $Code = $CURLE_OK Then

        Local $aNextUID = StringRegExp(BinaryToString(Curl_Data_Get($Curl)), 'UIDNEXT\h(\d+)', 3)
        If Not @error Then
            ConsoleWrite('next uid = ' & $aNextUID[0] & @CRLF)

            Curl_Easy_Reset($Curl) ; clear all previous options

            Curl_Easy_Setopt($Curl, $CURLOPT_VERBOSE, 1) ;
            Curl_Easy_Setopt($Curl, $CURLOPT_USERNAME, $sUser) ;
            Curl_Easy_Setopt($Curl, $CURLOPT_PASSWORD, $sPass) ;
            Curl_Easy_Setopt($Curl, $CURLOPT_CAINFO, @ScriptDir & '\curl-ca-bundle.crt') ;

            ;Set email data and callback
            Curl_Data_Put($Curl, $sEmail)
            Curl_Easy_Setopt($Curl, $CURLOPT_UPLOAD, 1)
            Curl_Easy_Setopt($Curl, $CURLOPT_READFUNCTION, Curl_DataReadCallback())
            Curl_Easy_Setopt($Curl, $CURLOPT_READDATA, $Curl)
            Curl_Easy_Setopt($Curl, $CURLOPT_INFILESIZE, StringLen($sEmail)) ;set filesize
            ;Curl_Easy_Setopt($Curl, $CURLOPT_CONNECTTIMEOUT,20); very helpful when troubleshooting. default is 300 secs

            $sDirName = StringReplace($sDirName, ' ', '%20') ; replace spaces
            Curl_Easy_Setopt($Curl, $CURLOPT_URL, "imaps://" & $sIMAPServer & "/" & $sDirName & ";UID=" & $aNextUID[0]) ;append email
            Local $Code = Curl_Easy_Perform($Curl)
            If $Code = $CURLE_OK Then
                ConsoleWrite('append OK' & @CRLF)
            Else
                ConsoleWrite(Curl_Easy_StrError($Code) & @LF)
                ConsoleWrite('append failed' & @CRLF)
            EndIf
        Else
            ConsoleWrite('extract uidnext failed' & @CRLF)
        EndIf
    Else
        ConsoleWrite('request uidnext failed' & @CRLF)
    EndIf

    Curl_Data_Cleanup($Curl)
    Curl_Easy_Cleanup($Curl) ;
EndFunc   ;==>Example_Email_Append

 

Edited June 23, 2022 by Beege

[lbsl]

I have downloaded your library, to circumvent the issues of a certified Windows 7 environment that has outdated TLS1.0 and SSL2.0 support only.
It works pretty great.
One remark though, it might be the API has been altered and extended during the releaseperiod of this library and today, but you have build the post mechanism based on the formadd() function while the developers highly advise to use the Mime functions instead, as the Formadd() function(s) are deprecated.

 

curl_mime_addpart
curl_mime_data
curl_mime_data_cb
curl_mime_encoder
curl_mime_filedata
curl_mime_filename
curl_mime_free
curl_mime_headers
curl_mime_init
curl_mime_name
curl_mime_subparts
curl_mime_type

https://curl.se/libcurl/c/allfuncs.html 

 

I may try to add these functions myself, but i have only time for this in a couple of weeks.

[Beege]

Thanks for the info @lbsl

I did start writing those when I was playing around with the smtp stuff (for attachment options) but never got around to doing a real test with them. Here's what I had if you wanted to do any testing.

expandcollapse popup

Func Curl_Mime_Init($Handle)
    ;curl_mime *curl_mime_init(CURL *easy_handle);
    Return DllCall($g_hlibcurl, (@AutoItX64 ? "ptr" : "ptr:cdecl"), "curl_mime_init", "ptr", $Handle)[0]
EndFunc   ;==>Curl_Mime_Init

Func Curl_Mime_Addpart($MimeHandle)
    ;curl_mimepart *curl_mime_addpart(curl_mime *mime);
    Return DllCall($g_hlibcurl, (@AutoItX64 ? "ptr" : "ptr:cdecl"), "curl_mime_addpart", "ptr", $MimeHandle)[0]
EndFunc   ;==>curl_mime_addpart

Func Curl_Mime_Subparts($MimePart, $MimeHandle)
    ;CURLcode curl_mime_subparts(curl_mimepart *part, curl_mime *subparts);
    Return DllCall($g_hlibcurl, (@AutoItX64 ? "int" : "int:cdecl"), "curl_mime_subparts", "ptr", $MimePart, 'ptr', $MimeHandle)[0]
EndFunc   ;==>curl_mime_subparts

Func Curl_Mime_Name($MimePart, $sName)
    ;CURLcode curl_mime_name(curl_mimepart *part, const char *name);
    Return DllCall($g_hlibcurl, (@AutoItX64 ? "int" : "int:cdecl"), "curl_mime_name", "ptr", $MimePart, 'str', $sName)[0]
EndFunc   ;==>curl_mime_name

Func Curl_Mime_Filedata($MimePart, $sFilename)
    ;CURLcode curl_mime_filedata(curl_mimepart *part, const char *filename);
    Return DllCall($g_hlibcurl, (@AutoItX64 ? "int" : "int:cdecl"), "curl_mime_filedata", "ptr", $MimePart, 'str', $sFilename)[0]
EndFunc   ;==>curl_mime_filedata

Func Curl_Mime_Filename($MimePart, $sFilename)
    ;CURLcode curl_mime_filename(curl_mimepart *part, const char *filename);
    Return DllCall($g_hlibcurl, (@AutoItX64 ? "int" : "int:cdecl"), "curl_mime_filename", "ptr", $MimePart, 'str', $sFilename)[0]
EndFunc   ;==>curl_mime_filename

Func Curl_Mime_Data($MimePart, $vData, $iSize = -1)
    ;CURLcode curl_mime_data(curl_mimepart *part, const char *data, size_t datasize);
    Local $Type = IsString($vData) ? 'str' : 'ptr'
    If IsDllStruct($vData) Then
        $Type = 'struct*'
        $iSize = DllStructGetSize($vData)
    EndIf
    Return DllCall($g_hlibcurl, (@AutoItX64 ? "int" : "int:cdecl"), "curl_mime_data", "ptr", $MimePart, $Type, $vData, 'uint_ptr', $iSize)[0]
EndFunc   ;==>curl_mime_data

Func Curl_Mime_Type($MimePart, $sMimetype)
    ;CURLcode curl_mime_type(curl_mimepart *part, const char *mimetype);
    Return DllCall($g_hlibcurl, (@AutoItX64 ? "int" : "int:cdecl"), "curl_mime_type", "ptr", $MimePart, 'str', $sMimetype)[0]
EndFunc   ;==>curl_mime_type

Func Curl_Mime_Encoder($MimePart, $sEnc)
    ;CURLcode curl_mime_encoder(curl_mimepart *part, const char *encoding);
    Return DllCall($g_hlibcurl, (@AutoItX64 ? "int" : "int:cdecl"), "curl_mime_encoder", "ptr", $MimePart, 'str', $sEnc)[0]
EndFunc   ;==>curl_mime_encoder

Func Curl_Mime_Headers($MimePart, $pSLIST, $iTakeOwnership = 1)
    ;CURLcode curl_mime_headers(curl_mimepart *part, struct curl_slist *headers, int take_ownership);
    Return DllCall($g_hlibcurl, (@AutoItX64 ? "int" : "int:cdecl"), "curl_mime_headers", "ptr", $MimePart, 'ptr', $pSLIST, 'int', $iTakeOwnership)[0]
EndFunc   ;==>curl_mime_headers

 

Edited September 1, 2022 by Beege

[lbsl]

On 9/2/2022 at 12:27 AM, Beege said:

Thanks for the info @lbsl

I did start writing those when I was playing around with the smtp stuff (for attachment options) but never got around to doing a real test with them. Here's what I had if you wanted to do any testing.

expandcollapse popup

Func Curl_Mime_Init($Handle)
    ;curl_mime *curl_mime_init(CURL *easy_handle);
    Return DllCall($g_hlibcurl, (@AutoItX64 ? "ptr" : "ptr:cdecl"), "curl_mime_init", "ptr", $Handle)[0]
EndFunc   ;==>Curl_Mime_Init

Func Curl_Mime_Addpart($MimeHandle)
    ;curl_mimepart *curl_mime_addpart(curl_mime *mime);
    Return DllCall($g_hlibcurl, (@AutoItX64 ? "ptr" : "ptr:cdecl"), "curl_mime_addpart", "ptr", $MimeHandle)[0]
EndFunc   ;==>curl_mime_addpart

Func Curl_Mime_Subparts($MimePart, $MimeHandle)
    ;CURLcode curl_mime_subparts(curl_mimepart *part, curl_mime *subparts);
    Return DllCall($g_hlibcurl, (@AutoItX64 ? "int" : "int:cdecl"), "curl_mime_subparts", "ptr", $MimePart, 'ptr', $MimeHandle)[0]
EndFunc   ;==>curl_mime_subparts

Func Curl_Mime_Name($MimePart, $sName)
    ;CURLcode curl_mime_name(curl_mimepart *part, const char *name);
    Return DllCall($g_hlibcurl, (@AutoItX64 ? "int" : "int:cdecl"), "curl_mime_name", "ptr", $MimePart, 'str', $sName)[0]
EndFunc   ;==>curl_mime_name

Func Curl_Mime_Filedata($MimePart, $sFilename)
    ;CURLcode curl_mime_filedata(curl_mimepart *part, const char *filename);
    Return DllCall($g_hlibcurl, (@AutoItX64 ? "int" : "int:cdecl"), "curl_mime_filedata", "ptr", $MimePart, 'str', $sFilename)[0]
EndFunc   ;==>curl_mime_filedata

Func Curl_Mime_Filename($MimePart, $sFilename)
    ;CURLcode curl_mime_filename(curl_mimepart *part, const char *filename);
    Return DllCall($g_hlibcurl, (@AutoItX64 ? "int" : "int:cdecl"), "curl_mime_filename", "ptr", $MimePart, 'str', $sFilename)[0]
EndFunc   ;==>curl_mime_filename

Func Curl_Mime_Data($MimePart, $vData, $iSize = -1)
    ;CURLcode curl_mime_data(curl_mimepart *part, const char *data, size_t datasize);
    Local $Type = IsString($vData) ? 'str' : 'ptr'
    If IsDllStruct($vData) Then
        $Type = 'struct*'
        $iSize = DllStructGetSize($vData)
    EndIf
    Return DllCall($g_hlibcurl, (@AutoItX64 ? "int" : "int:cdecl"), "curl_mime_data", "ptr", $MimePart, $Type, $vData, 'uint_ptr', $iSize)[0]
EndFunc   ;==>curl_mime_data

Func Curl_Mime_Type($MimePart, $sMimetype)
    ;CURLcode curl_mime_type(curl_mimepart *part, const char *mimetype);
    Return DllCall($g_hlibcurl, (@AutoItX64 ? "int" : "int:cdecl"), "curl_mime_type", "ptr", $MimePart, 'str', $sMimetype)[0]
EndFunc   ;==>curl_mime_type

Func Curl_Mime_Encoder($MimePart, $sEnc)
    ;CURLcode curl_mime_encoder(curl_mimepart *part, const char *encoding);
    Return DllCall($g_hlibcurl, (@AutoItX64 ? "int" : "int:cdecl"), "curl_mime_encoder", "ptr", $MimePart, 'str', $sEnc)[0]
EndFunc   ;==>curl_mime_encoder

Func Curl_Mime_Headers($MimePart, $pSLIST, $iTakeOwnership = 1)
    ;CURLcode curl_mime_headers(curl_mimepart *part, struct curl_slist *headers, int take_ownership);
    Return DllCall($g_hlibcurl, (@AutoItX64 ? "int" : "int:cdecl"), "curl_mime_headers", "ptr", $MimePart, 'ptr', $pSLIST, 'int', $iTakeOwnership)[0]
EndFunc   ;==>curl_mime_headers

 

I have added your functions into your Curl library and successfully cooked up some examples for Telegram:

There is enough in that Telegram UDF to steal for your own examples.

 

[Beege]

Nice work @lbsl!!

[argumentum]

request: CURLOPT_UPLOAD example. ( the one I see is for email and I'd like to post to PHP )

Edited November 20, 2022 by argumentum
clarify request

[lbsl]

On 11/20/2022 at 6:54 AM, argumentum said:

request: CURLOPT_UPLOAD example. ( the one I see is for email and I'd like to post to PHP )

That would mean combining these two examples:
https://curl.se/libcurl/c/CURLOPT_UPLOAD.html

The callback to track progress:

https://curl.se/libcurl/c/CURLOPT_READFUNCTION.html

 

There is a public help function Curl_FileReadCallback(), might need to reference to that function when using the CURLOPT_READFUNCTION
And there is also a Curl_DataReadCallback() function, used in the very CURL_OPT_UPLOAD example.

Edited November 29, 2022 by lbsl
Additional info

[lowbattery]

I'm trying to send some -d data (json actually) but I can't figure it out. I don't know which global Enum_ function to use. Can someone point me in the right direction?

Also, while I'm using the Curl rewrite UDF, I did see this post here which referred to using Ward's version. But I still can't make heads or tail of how to use it.

 

 

 

Edited May 13, 2023 by lowbattery

[jugador]

Curl now support websocket protocol.
https://curl.se/docs/websocket.html

@Beege if time permits add websocket function to your UDF.

curl_ws_recv
curl_ws_send
curl_ws_meta

websocket example: https://curl.se/libcurl/c/websocket.html

[jugador]

working example:  websocket handshake

#include <Constants.au3>
#include "Curl.au3"

__websocket_EgA()
Func __websocket_EgA()
    ConsoleWrite("ws.postman-echo.com...... " & @LF)

    Local $hRespHeaders = 2
    Local $Curl = Curl_Easy_Init()
    If Not $Curl Then Return

    Curl_Easy_Setopt($Curl, $CURLOPT_URL, "wss://ws.postman-echo.com/raw")
    Curl_Easy_Setopt($Curl, $CURLOPT_CONNECT_ONLY, 2)
    Curl_Easy_Setopt($Curl, $CURLOPT_HEADERFUNCTION, Curl_DataWriteCallback())
    Curl_Easy_Setopt($Curl, $CURLOPT_HEADERDATA, $hRespHeaders)
    Curl_Easy_Setopt($Curl, $CURLOPT_CAINFO, @ScriptDir & '\curl-ca-bundle.crt')

    Local $Code = Curl_Easy_Perform($Curl)
    If $Code <> $CURLE_OK Then Return ConsoleWrite(Curl_Easy_StrError($Code) & @LF)
    Local $Response_Headers = BinaryToString(Curl_Data_Get($hRespHeaders), 4)
    Curl_Easy_Cleanup($Curl)
    Curl_Data_Cleanup($Curl)

    ConsoleWrite(@CRLF & "Response Headers:" & @CRLF & $Response_Headers & @CRLF)
EndFunc

 

now to send and receive data need @Beege help.

[jugador]

note: need minimum sleep(250) before curl_ws_recv() to work or get $CURLE_AGAIN error.

[n3wbie]

First of all thanks for this wonderful udf.

I'm not sure if I need to post it here. 

But I was using curl from a very long time.Just got a latency lag because of serial processing using curl (inside loop) so wished to shift to curl multi. I could successfully fire 10's of request simultaneously. My question is how do I know which response is to which request.

Now I already saw that we can get the url for it but in my case url is same, post params are different.

Is it possible that we could map requests to response.or have index of response to requests

Also I tried searching on internet but I couldn't succeed. I know this might be a very lame question for many . But I wish if someone can draw my attention to any article or give some input so that I could proceed to use that.

 

Example would be

1. www.example.com/sum.php

Post params :var1=1&var2=5

This would result in getting 6 as answer on api.

But if i fire 100 of api's with different parameters it would be difficult right now for me to generate an understanding logic to determine which response corresponds to which request.

I hope I have explained well the scenario. Still if someone finds it unreasonable and understandable I would eloborate the same.

Thanks in advance.

[mLipok]

I understand.

Have no idea how to help you.

Sorry.

[TheXman]

11 hours ago, n3wbie said:

I could successfully fire 10's of request simultaneously. My question is how do I know which response is to which request.
...
But I wish if someone can draw my attention to any article or give some input so that I could proceed to use that.

The way I do it is to have curl store each response in a file whose file name uniquely identifies the request and then process the files accordingly. 

 

Longer explanation below:

Spoiler

I created a script that does something similar.  It runs a process that usually requires between 50-120 distinct HTTPS API requests in order to fulfill the task.  The number of requests are known prior to starting the process and are stored in and read from a file.  I used to process the requests serially.  But since I have plenty of Internet bandwidth and wanted to reduce the run time, I decided to change the processing of the requests from serial to a specified number of parallel requests.  I usually set the batch size to 3-5 parallel requests at a time and process the batches of parallel requests until all of the requests in the file have been handled.  My script uses the curl command line utility, not one of the curl UDFs.  However, the same can easily be done using the libcurl API's which the UDF's are built around.  I just just find it much simpler to use the command line utility and it requires far less code.

In my case, I not only needed to match responses with their respective requests, I also needed to process the responses in the order in which they were requested. As I'm sure you know, when processing requests in parallel, the order of the responses are not necessarily in the same order in which they were requested.  The way I chose to reliably process the responses in the order in which they were requested, was to have curl store each response in a file that had a uniquely identifiable file name that provided info about the request and the order in which it needed to be processed.

Sample log file:

Spoiler

2022-07-13 10:20:26    Script Started
2022-07-13 10:22:37    
2022-07-13 10:22:37    Getting links from file...
2022-07-13 10:22:37      Number of links = 80
2022-07-13 10:22:37    
2022-07-13 10:22:37    Downloading Files in parallel...
2022-07-13 10:22:37      Pausing for 1 second(s)...
2022-07-13 10:22:38      Downloading (001/080) C:\Projects\Personal\AutoIt\Downloader\Files\req_001.dat
2022-07-13 10:22:38                            https://some.site.com/v4/FBeSu3ppTZYP5)U=.dat
2022-07-13 10:22:38      Downloading (002/080) C:\Projects\Personal\AutoIt\Downloader\Files\req_002.dat
2022-07-13 10:22:38                            https://some.site.com/v4/0XeZdS6EnhwdLT0=.dat
2022-07-13 10:22:38      Downloading (003/080) C:\Projects\Personal\AutoIt\Downloader\Files\req_003.dat
2022-07-13 10:22:38                            https://some.site.com/v4/YQ9XxY2fUv5U70I=.dat
2022-07-13 10:22:48      Pausing for 1 second(s)...
2022-07-13 10:22:49      Downloading (004/080) C:\Projects\Personal\AutoIt\Downloader\Files\req_004.dat
2022-07-13 10:22:49                            https://some.site.com/v4/9wXhsQ)8bFu4ZAE=.dat
2022-07-13 10:22:49      Downloading (005/080) C:\Projects\Personal\AutoIt\Downloader\Files\req_005.dat
2022-07-13 10:22:49                            https://some.site.com/v4/P2YG2bvc0kDygXg=.dat
2022-07-13 10:22:49      Downloading (006/080) C:\Projects\Personal\AutoIt\Downloader\Files\req_006.dat
2022-07-13 10:22:49                            https://some.site.com/v4/Q7UY0iLPYHp5Hj0=.dat
2022-07-13 10:22:58      Pausing for 2 second(s)...
2022-07-13 10:23:00      Downloading (007/080) C:\Projects\Personal\AutoIt\Downloader\Files\req_007.dat
2022-07-13 10:23:00                            https://some.site.com/v4/x8GuPMCrvnsd5UU=.dat
2022-07-13 10:23:00      Downloading (008/080) C:\Projects\Personal\AutoIt\Downloader\Files\req_008.dat
2022-07-13 10:23:00                            https://some.site.com/v4/zXln8MqR1Xk(m10=.dat
2022-07-13 10:23:00      Downloading (009/080) C:\Projects\Personal\AutoIt\Downloader\Files\req_009.dat
2022-07-13 10:23:00                            https://some.site.com/v4/zzQWho2nSaXsq3s=.dat
2022-07-13 10:23:09      Pausing for 3 second(s)...
2022-07-13 10:23:12      Downloading (010/080) C:\Projects\Personal\AutoIt\Downloader\Files\req_010.dat
2022-07-13 10:23:12                            https://some.site.com/v4/PyLiDhHh43Y99TU=.dat
2022-07-13 10:23:12      Downloading (011/080) C:\Projects\Personal\AutoIt\Downloader\Files\req_011.dat
2022-07-13 10:23:12                            https://some.site.com/v4/ZLW6uhUZd31tgAA=.dat
2022-07-13 10:23:12      Downloading (012/080) C:\Projects\Personal\AutoIt\Downloader\Files\req_012.dat
2022-07-13 10:23:12                            https://some.site.com/v4/ST0FEaoKWTO61uU=.dat
. . .
2022-07-13 10:26:42      Pausing for 2 second(s)...
2022-07-13 10:26:44      Downloading (079/080) C:\Projects\Personal\AutoIt\Downloader\Files\req_079.dat
2022-07-13 10:26:44                            https://some.site.com/v4/vpcYVbfVRKcza6g=.dat
2022-07-13 10:26:44      Downloading (080/080) C:\Projects\Personal\AutoIt\Downloader\Files\req_080.dat
2022-07-13 10:26:44                            https://some.site.com/v4/wAjo)0dKNOxkHVY=.dat
2022-07-13 10:26:49      Downloading Complete
2022-07-13 10:26:49    
2022-07-13 10:26:49    Creating target file...
2022-07-13 10:27:02      Target file successfully created
2022-07-13 10:27:02      Deleting temp files
2022-07-13 10:27:02    
2022-07-13 10:27:02    Processing Complete
2022-07-13 10:27:04    
2022-07-13 10:27:04    Script Ended - Execution time: 6 Mins 37.562 Secs - Exit Code: 0
2022-07-13 10:27:04   

 

 

Edited August 27, 2023 by TheXman

[n3wbie]

15 hours ago, TheXman said:

The way I do it is to have curl store each response in a file whose file name uniquely identifies the request and then process the files accordingly. 

 

First of all thanks for the idea as it seems promising for what is to be achieved.

1.Can you share snippet of doing it via curl command line utility

2. Can you share a way doing the same via Lib Curl. saving different files names (Just that part of line)

3. How do you batch it out in group of 3-5 at time. Does Curl Provided it or you loop it in such a way that max 3-5 are grouped together.

(Yesterday i processed 175 request at same time. got results in net 40 seconds) where 30 seconds are actually utilized by server for processing request(Having 100 Mbps speed~ 12.5 MB/S) Almost 124 Results were properly retrieved but others failed. And I Couldn't trace out which of them failed

Another question is callback functions with write functions in different HTML would it work?

Curl_Easy_Setopt($Curl, $CURLOPT_WRITEFUNCTION, Curl_DataWriteCallback())
    Curl_Easy_Setopt($Curl, $CURLOPT_WRITEDATA, $Html[$i])

Where $html be different ?

Edited August 28, 2023 by n3wbie

[TheXman]

You're welcome.

Implicit in your questions is a lack of prerequisite research & understanding of curl and how it works.  Since this is an AutoIt forum and not a curl forum, I won't spend much time trying to help you learn & understand curl itself.  However, if you have specific questions, I'll most likely try to answer them.  Also, this topic is for this curl UDF.  Therefore, I won't go into too much detail about using the curl command line.  However, I would suggest that you get familiar with using the command line version of curl before trying to use the libcurl UDF's.  Having a firm understanding to the command line parameters and how they work will go a long way in terms of helping you understand how to use the libcurl UDF's.  If you want to ask more questions about the command line version, I would suggest creating a general help & support topic with your questions.

Answers to your enumerated questions above:

1. Maybe, right after you show a snippet of the code in which you at least try to implement it yourself.  As a hint, I have provided a sample curl command line below that sends 3 requests, in parallel, and outputs their individual responses to separate files.
    
2. See answer #1.
    
3. Does curl provide a way to set the maximum number of parallel requests to process at any given time?  Yes, the command line parameter is "--parallel-max".  How do I do it my script?  In my script, I have a CONST that tells the script the maximum number of requests that I wish to send in a single curl command.  As I mentioned in my previous post, I have a file that contains all of the requests.  I read in those requests and batch them together according to the CONST that specifies the max requests and execute the commands.  In my particular case, if any given request fails, the whole process needs to stop.  The parameter I use for that is "--fail-early".  If you want to learn more about curl parameters/options, there are several good resources on the web including the official curl site.  I have included a few of those reference links below.

Sample curl command line that my script generates and executes:

The following sample curl command makes 3 requests, in parallel, and outputs each response to a separate file.  If you open a command prompt and successfully execute it, you should see the 3 response files.

curl.exe -s -Z --parallel-max 3 --fail-early -L -o "~resp-file1.html" "https://example.com" --next -L -o "~resp-file2.html" "https://example.com" --next -L -o "~resp-file3.html" "https://example.com"

 

13 hours ago, n3wbie said:

Yesterday i processed 175 request at same time.

Hopefully all of those requests were not against the same server.  If so, sending 175 concurrent requests from a single IP address is a very good way to get your IP address temporarily or permanently banned on that web server and/or a very harsh warning from your ISP. 

13 hours ago, n3wbie said:

Another question is callback functions with write functions in different HTML would it work?

I have no idea whether that'll work because that tiny little snippet does not provide enough context about the overall implementation.  If $html is an array of curl handles, then maybe.  But as I said, that snippet is woefully inadequate to say.

 

A few curl reference links:

• Documentation Overview (official site)
• The manual on the official site
• Examples based on specific tasks (official site)
• Another good general curl information site
  • Info about parallel requests on this site

Edited August 28, 2023 by TheXman

[n3wbie]

3 hours ago, TheXman said:

have no idea whether that'll work because that tiny little snippet does not provide enough context about the overall implementation.  If $html is an array of curl handles, then maybe.  But as I said, that snippet is woefully inadequate to say.

I will get back to you on this with piece of my own working code.

For time being just updating new findings.

Use of $CURLOPT_PRIVATE this can be of some use.

I was unaware of curl command line parameters which eventually led me to other findings that could potentially help me grow my knowledge. Thanks @TheXman

Edited August 28, 2023 by n3wbie

[Mamrot]

First of all, thank you for the great UDF.
I don't know if it's my mistake, lack of knowledge or a bug.
I'm trying to send simple JSON data using the PUT method, when I use PostMan desktop app to do so, everything works fine.
When I try this from code I get a problem.

expandcollapse popup

#include <Date.au3>
#include <WinAPIDiag.au3>
#include "Curl.au3"

Test()

Func Test()

    ;Create an array of request headers
    $aHeaders = Curl_Slist_Append(0, "X-API-KEY: zzzzzzzE3bnpJdEFtYitHUTY=")
    $aHeaders = Curl_Slist_Append($aHeaders, "Content-Type: application/json")

    ;Set up and execute cURL request
    $hCurl = Curl_Easy_Init()
    If Not $hCurl Then Return MsgBox($MB_ICONERROR, "Error", "Curl_Easy_Init() failed.")

    Curl_Easy_Setopt($hCurl, $CURLOPT_URL, "https://postman-echo.com/put")
;~  Curl_Easy_Setopt($hCurl, $CURLOPT_URL, "https://jsonplaceholder.typicode.com/puts")
    Curl_Easy_Setopt($hCurl, $CURLOPT_HTTPHEADER, $aHeaders)
    Curl_Easy_Setopt($hCurl, $CURLOPT_USERAGENT, "AutoIt/cURL")
    Curl_Easy_Setopt($hCurl, $CURLOPT_VERBOSE, 1)
    Curl_Easy_Setopt($hCurl, $CURLOPT_SSL_VERIFYPEER, 0)
    Curl_Easy_Setopt($hCurl, $CURLOPT_READFUNCTION, Curl_DataReadCallback())
    Curl_Easy_Setopt($hCurl, $CURLOPT_CUSTOMREQUEST, "PUT")
    Curl_Easy_Setopt($hCurl, $CURLOPT_UPLOAD, 1)
    Curl_Easy_Setopt($hCurl, $CURLOPT_POSTFIELDS, '{"params":{"settings":{"settingModificationType":"edit"},"products":[{"productId":51,"productRetailPrice":48,"productVat":23}]}}')

    ;Get response code and response
    $iRespCode = Curl_Easy_Perform($hCurl)
    If $iRespCode <> $CURLE_OK Then Return ConsoleWrite("Status Message: " & Curl_Easy_StrError($iRespCode) & @LF)
    $sResponse = BinaryToString(Curl_Data_Get($hCurl))

    ;Clean up curl environment
    Curl_Easy_Cleanup($hCurl)
    Curl_Data_Cleanup($hCurl)

    ;Display response
    ConsoleWrite($sResponse & @CRLF)
EndFunc   ;==>Test

PostMan-Echo returns :

{
  "args": {},
  "data": "����",
  "files": {},
  "form": {},
  "headers": {
    "x-forwarded-proto": "https",
    "x-forwarded-port": "443",
    "host": "postman-echo.com",
    "x-amzn-trace-id": "Root=1-11111111111-0b5caa1f68b222370524152d",
    "content-length": "4",
    "user-agent": "AutoIt/cURL",
    "accept": "*/*",
    "x-api-key": "zzzzzzE3bnpJdEFtYitHUTY=",
    "content-type": "application/json",
    "accept-encoding": "gzip, deflate, br"
  },
  "json": null,
  "url": "https://postman-echo.com/put"

Why does "data": "����" contain such characters instead of my JSON data?

Any help is greatly appreciated.

 

Regards, M.

Edited February 12 by Jos
changed apikey