Struggling with Binary Data Using Beege's CURL UDF

[lowbattery]

I'm trying to upload a JPG image to my WordPress site. I have the command line curl working. It successfully uploads the jpg image.

$str = 'curl.exe -s --request POST --url https://www.mysite.com/wp-json/wp/v2/media --header "cache-control: no-cache" --header "content-disposition: attachment; filename=file.jpg" --header "authorization: Basic '&base64($WPAppUser&':'&$WPAppPW)&' " --header "content-type: image/jpg" --data-binary "@hardware_store.jpg" --location'

$iPidCurl = Run($str, '', @SW_HIDE, $STDERR_MERGED)

But when I try to replicate it in Beege's UDF, I get a "400 - Bad Request" response from my server. Here is what I have for the UDF code:

$imageLoc = @ScriptDir & "\hardware_store.jpg"

Local $hFileOpen = FileOpen($imageLoc, 16)
Local $sFileRead = FileRead($hFileOpen)
$sFileSize = BinaryLen($sFileRead)
FileClose($hFileOpen)

Local $Curl = Curl_Easy_Init()
If Not $Curl Then Return

Local $arrHeaders = Curl_Slist_Append(0, "Cache-Control: no-cache")
$arrHeaders = Curl_Slist_Append($arrHeaders, "Content-Type: image/jpg")
$arrHeaders = Curl_Slist_Append($arrHeaders, "Content-Disposition: attachment; filename=test.jpg")

Curl_Easy_Setopt($Curl, $CURLOPT_URL, "https://www.mysite.com/wp-json/wp/v2/media")
Curl_Easy_Setopt($Curl, $CURLOPT_HTTPHEADER, $arrHeaders)
Curl_Easy_Setopt($Curl, $CURLOPT_ACCEPT_ENCODING, '')
Curl_Easy_Setopt($Curl, $CURLOPT_USERAGENT, "AutoIt/Curl")
Curl_Easy_Setopt($Curl, $CURLOPT_WRITEFUNCTION, Curl_DataWriteCallback())
Curl_Easy_Setopt($Curl, $CURLOPT_WRITEDATA, $Curl) 
Curl_Easy_Setopt($Curl, $CURLOPT_MAXREDIRS, 10)
Curl_Easy_Setopt($Curl, $CURLOPT_TIMEOUT, 30)
Curl_Easy_Setopt($Curl, $CURLOPT_USERPWD, $WPAppUser&':'&$WPAppPW)
Curl_Easy_Setopt($Curl, $CURLOPT_FOLLOWLOCATION, 1)
Curl_Easy_Setopt($Curl, $CURLOPT_POSTFIELDS, $sFileRead)
Curl_Easy_Setopt($Curl, $CURLOPT_POSTFIELDSIZE, $sFileSize)

;peer verification
curl_easy_setopt($Curl, $CURLOPT_CAINFO, @ScriptDir & '\curl-ca-bundle.crt') ;
curl_easy_setopt($Curl, $CURLOPT_SSL_VERIFYPEER, 1)

Local $Code = Curl_Easy_Perform($Curl)
If $Code <> $CURLE_OK Then Return ConsoleWrite(Curl_Easy_StrError($Code) & @LF)
ConsoleWrite(BinaryToString(Curl_Data_Get($Curl)))

I've tried just about everything, but I can't figure out what's causing it not to work.

Edited August 17, 2023 by lowbattery

[KaFu]

Are you sure that the command line approach works? For both I get this response:

{"code":"rest_cannot_create","message":"Sorry, you are not allowed to create posts as this user.","data":{"status":401}}

I checked the response for the command line in the cmd window:

$str = 'curl.exe -s --request POST --url ' & $sURL & ' --header "cache-control: no-cache" --header "content-disposition: attachment; filename=test.jpg" --header "authorization: Basic ' & _Base64Encode($WPAppUser & ':' & $WPAppPW) & '" --header "content-type: image/jpg" --data-binary "@test.jpg" --location'

$iPidCurl = Run(@ComSpec & " /k " & $str)

Do you know the destination path? Is it "/wp-content/uploads/test.jpg" in this case?

[lowbattery]

Hi KaFu.

I'm 100% positive because when I run it, and refresh my media library, I see the image uploaded. Your message is likely because you haven't created an application password for a particular user on your WP site. Go to the user's account, scroll all the way to the bottom, and you can make an application password. I attached a screenshot to show you what it looks like.

Now the $WPAppUser variable is your user's username, but the $WPAppPW is the user's application password you just made (not their account password).

As for the URL, it's simply "https://www.mysite.com/wp-json/wp/v2/media" (replacing mysite with your domain). You shouldn't add any trailing destination as the filename is handled by the headers. Just note that the test.jpg file needs to be in your script directory, otherwise you need to @somedir/test.jpg to get to it.

As for the UDF, it's something to do with either the file read or the $CURLOPT_POSTFIELDSIZE.

When I don't specify CURLOPT_POSTFIELDSIZE I get a "{"code":"rest_upload_no_data","message":"No data supplied.","data":{"status":400}}Done." response. I also got it to upload a file into the media library (I forget how) but it was just an empty file. Nothing inside of it and so it wasn't a valid image. When I add CURLOPT_POSTFIELDSIZE, using the binary length (which I believe is the requirement for binary files), I get a HTML response with a H1 message saying 400 - Bad Request.

I'm stumped.

 

Edited August 17, 2023 by lowbattery

[mLipok]

Just as reference I see you are using @Beege UDF:

or am I wrong ?

Edited August 17, 2023 by mLipok

[lowbattery]

13 minutes ago, mLipok said:

did you notice new  @Beege UDF:

 

Oops. I have to apologize. I have so many UDFs I got it confused. Yes, that rewrite by @Beege is the one I'm using. I'm sorry about that mix-up. I re-titled the form post to reflect it.

Edited August 17, 2023 by lowbattery

[KaFu]

Setting both to verbose I see that the command line uses "HTTP 1.1" and the dll "HTTP 2".

But switching back to 1.1. in the dll fails for me, timeout of script.

Curl_Easy_Setopt($Curl, $CURLOPT_HTTP_VERSION, $CURL_HTTP_VERSION_1_1)

Edited August 17, 2023 by KaFu

[lowbattery]

1 hour ago, KaFu said:

Setting both to verbose I see that the command line uses "HTTP 1.1" and the dll "HTTP 2".

But switching back to 1.1. in the dll fails for me, timeout of script.

Curl_Easy_Setopt($Curl, $CURLOPT_HTTP_VERSION, $CURL_HTTP_VERSION_1_1)

I am seeing the same thing. Not sure what to do. Hmm.

I have tried this: Curl_Easy_Setopt($Curl, $CURLOPT_POSTFIELDS, '@Curlx64/hardware_store.jpg') and with $CURL_HTTP_VERSION_1_1 I get a file uploaded, it's the correct size (likely due to the $sFileSize), but it's not a valid JPG. So it seems that the file isn't being uploaded.

I tried that because https://stackoverflow.com/questions/10322646/how-does-one-post-a-binary-file-stored-locally-using-php-curl suggested something like that might be possible. But in PHP they insert the file link as an array. Not sure how array('file' => '@foo.ext') translates to AutoIt though...

Edited August 17, 2023 by lowbattery

[lowbattery]

Just a follow-up. I can use the UDF to make a new post. The code below does work. It even works if we add the Curl_Easy_Setopt($Curl, $CURLOPT_HTTP_VERSION, $CURL_HTTP_VERSION_1_1) line. It's just the image code above that I'm still stuck on. 

expandcollapse popup

Local $jsonData
_JSON_addChangeDelete($jsonData, "title", 'test article')
_JSON_addChangeDelete($jsonData, "content", 'test content')
_JSON_addChangeDelete($jsonData, "status", 'draft')
$jData = _JSON_GenerateCompact($jsonData)

$WPAppUser = "wordpress user name"
$WPAppPW = "wordpress app password for the user name"

Local $Curl = Curl_Easy_Init()
If Not $Curl Then Return

Local $Html = $Curl ; any number as identify
Local $Header = $Curl + 1 ; any number as identify

Local $arrHeaders = Curl_Slist_Append(0, "Accept: application/json")
$arrHeaders = Curl_Slist_Append($arrHeaders, "Content-Type: application/json")

Curl_Easy_Setopt($Curl, $CURLOPT_URL, "https://www.mysite.com/wp-json/wp/v2/posts")
Curl_Easy_Setopt($Curl, $CURLOPT_ACCEPT_ENCODING, '')
Curl_Easy_Setopt($Curl, $CURLOPT_WRITEFUNCTION, Curl_DataWriteCallback())
Curl_Easy_Setopt($Curl, $CURLOPT_WRITEDATA, $Html)
Curl_Easy_Setopt($Curl, $CURLOPT_MAXREDIRS, 10)
Curl_Easy_Setopt($Curl, $CURLOPT_TIMEOUT, 30)
Curl_Easy_Setopt($Curl, $CURLOPT_USERPWD, $WPAppUser&':'&$WPAppPW)
Curl_Easy_Setopt($Curl, $CURLOPT_FOLLOWLOCATION, 1)
Curl_Easy_Setopt($Curl, $CURLOPT_HTTPHEADER, $arrHeaders)
Curl_Easy_Setopt($Curl, $CURLOPT_POST, '1L')
Curl_Easy_Setopt($Curl, $CURLOPT_COPYPOSTFIELDS, $jData)

;peer verification
curl_easy_setopt($Curl, $CURLOPT_CAINFO, @ScriptDir & '\curl-ca-bundle.crt') ;
curl_easy_setopt($Curl, $CURLOPT_SSL_VERIFYPEER, 1)

Local $Code = Curl_Easy_Perform($Curl)
If $Code = $CURLE_OK Then
    ConsoleWrite("Content Type: " & Curl_Easy_GetInfo($Curl, $CURLINFO_CONTENT_TYPE) & @LF)
    ;ConsoleWrite("Download Size: " & Curl_Easy_GetInfo($Curl, $CURLINFO_SIZE_DOWNLOAD) & @LF)
    ConsoleWrite('Header: ' & @CRLF & BinaryToString(Curl_Data_Get($Header)) & @LF)
    MsgBox(0, 'Html', BinaryToString(Curl_Data_Get($Html)))
Else
    ConsoleWrite(Curl_Easy_StrError($Code) & @LF)
EndIf

Curl_Slist_Free_All($arrHeaders)
Curl_Easy_Cleanup($Curl)
Curl_Data_Cleanup($Header)
Curl_Data_Cleanup($Html)

 

Edited August 17, 2023 by lowbattery

[TheXman]

17 hours ago, lowbattery said:

I've tried just about everything, but I can't figure out what's causing it not to work.

I can definitely tell you why your script is returning with a 400 (Bad Request) status code.  I can also provide you an example script, using Beege's cURL UDF, that correctly mimics the cURL CLI command that's working.  But first, if you have a working solution using the cURL command line utility, why use the UDF?

Edited August 17, 2023 by TheXman

[lowbattery]

3 hours ago, TheXman said:

I can definitely tell you why your script is returning with a 400 (Bad Request) status code.  I can also provide you an example script, using Beege's cURL UDF, that correctly mimics the cURL CLI command that's working.  But first, if you have a working solution using the cURL command line utility, why use the UDF?

I use the CLI a lot but since some windows installs don’t come with curl or have a differently compiled version, I provide it to end users. However, some of their systems view it as a virus. Other systems flag it as a virus when I make requests through SSL (which I think might be avoided with libcurl).

Another reason for wanting to switch to the UDF is that the CLI through shell execution has length limits for some things (as far as I know).

So worst case, I could use the UDF for most things and the CLI for images, but having everything done one way would be a benefit.

[TheXman]

Below, you will see how the Curl_Easy_Setopt function is defined in the UDF.  If the value that is passed has a string data type, then the value is passed as a str (char *), all other data type values are passed as a ptr.  In you original post, you are setting the POSTFIELDS value to the binary data in the variable ($sFileRead) when it is expecting a pointer to the binary data.

Func Curl_Easy_Setopt($Handle, $Option, $Value)
    Return DllCall($g_hlibcurl, (@AutoItX64 ? "int" : "int:cdecl"), "curl_easy_setopt", "ptr", $Handle, "int", $Option, IsString($Value) ? "str" : "ptr", $Value)[0]
EndFunc   ;==>Curl_Easy_Setopt

I don't have access to a WordPress site in order to test the example script below, but I think it is either correct or very close to being correct.  Please give it a try and see if it works for you.  If it doesn't, please show any responses and I'm sure I can help you get it working. 

The only real difference between the example script below and your original script is that it reads the image file data into a binary buffer and then passes a pointer to that buffer in the POSTFIELDS value.  There are a couple other tweaks but they are not significant.  Of course you will need to change the path to the UDF, the endpoint URL, username, and app password.

expandcollapse popup

#AutoIt3Wrapper_AU3Check_Parameters=-w 3 -w 4 -w- 5 -w 6 -d

#include <Constants.au3>
#include <cURL-Beege\curl.au3> ;<== Modify path as needed


WordPress_Upload_Post_Example(@ScriptDir & "\hardware_store.jpg")

Func WordPress_Upload_Post_Example($sImgFilePath)
    Local $tImgData  = ""
    Local $sResponse = ""
    Local $iRespCode = 0, _
          $iImgSize  = FileGetSize($sImgFilePath)
    Local $hCurl     = -1
    Local $aHeaders[0]


    ;Validate image file size
    If $iImgSize = 0 Then Return MsgBox($MB_ICONERROR, "Error", "Input file is empty or does not exist.")

    ;Create a byte buffer for the image data and load the image file into it
    $tImgData        = DllStructCreate(StringFormat("byte bytes[%i];", $iImgSize))
    $tImgData.bytes  = FileRead($sImgFilePath)

    ;Create an array of request headers
    $aHeaders = Curl_Slist_Append(0        , "Cache-Control: no-cache")
    $aHeaders = Curl_Slist_Append($aHeaders, "Content-Type: image/jpg")
    $aHeaders = Curl_Slist_Append($aHeaders, "Content-Disposition: attachment; filename=test.jpg")

    ;Set up and execute cURL request
    $hCurl = Curl_Easy_Init()
    If Not $hCurl Then Return MsgBox($MB_ICONERROR, "Error", "Curl_Easy_Init() failed.")

    Curl_Easy_Setopt($hCurl, $CURLOPT_URL            , "https://www.mysite.com/wp-json/wp/v2/media")
    Curl_Easy_Setopt($hCurl, $CURLOPT_USERPWD        , $WPAppUser & ':' & $WPAppPW)
    Curl_Easy_Setopt($hCurl, $CURLOPT_HTTPHEADER     , $aHeaders)
    Curl_Easy_Setopt($hCurl, $CURLOPT_USERAGENT      , "AutoIt/cURL")
    Curl_Easy_setopt($hCurl, $CURLOPT_SSL_VERIFYPEER , 1)
    Curl_Easy_setopt($hCurl, $CURLOPT_CAINFO         , @ScriptDir & '\curl-ca-bundle.crt') ;
    Curl_Easy_Setopt($hCurl, $CURLOPT_ACCEPT_ENCODING, "")
    Curl_Easy_Setopt($hCurl, $CURLOPT_WRITEFUNCTION  , Curl_DataWriteCallback())
    Curl_Easy_Setopt($hCurl, $CURLOPT_WRITEDATA      , $hCurl)
    Curl_Easy_Setopt($hCurl, $CURLOPT_FOLLOWLOCATION , 1)
    Curl_Easy_Setopt($hCurl, $CURLOPT_POSTFIELDS     , DllStructGetPtr($tImgData))
    Curl_Easy_Setopt($hCurl, $CURLOPT_POSTFIELDSIZE  , DllStructGetSize($tImgData))

    ;Get response code and response
    $iRespCode = Curl_Easy_Perform($hCurl)
    If $iRespCode <> $CURLE_OK Then Return ConsoleWrite("Status Message: " & Curl_Easy_StrError($iRespCode) & @LF)
    $sResponse = BinaryToString(Curl_Data_Get($hCurl))

    ;Clean up curl environment
    Curl_Easy_Cleanup($hCurl)
    Curl_Data_Cleanup($hCurl)

    ;Display response
    ConsoleWrite($sResponse & @CRLF)
EndFunc

For the record, I'm not quite sure how/if $CURLOPT_SSL_VERIFYPEER and $CURLOPT_FOLLOWLOCATION are being set correctly since since their values are numeric data types.  I didn't take a deep dive to see.  But at first glance, it would seem to be interpreting those values as pointers.  It may turn out that since the API is expecting a LONG, it is only reading the first 32 bits of what is being passed and interpreting it as a LONG (which would work I guess). In a 32-bit environment a LONG and a Ptr are the same size.  Of course in a 64-bit environment a Ptr is 64 bits.  So I guess the high order 32 bits are just getting truncated (or ignored). If it were my UDF, in the DllCall, I would set the data types of the option parameters to their expected data types. 

Actual cURL API declarations of the 3 options
  
CURLcode curl_easy_setopt(CURL *handle, CURLOPT_POSTFIELDS, char *postdata);
CURLcode curl_easy_setopt(CURL *handle, CURLOPT_SSL_VERIFYPEER, long verify);
CURLcode curl_easy_setopt(CURL *handle, CURLOPT_FOLLOWLOCATION, long enable);

 

Edited August 18, 2023 by TheXman

[lowbattery]

WOW! It works! Thank you so much TheXMan!

The DllStruct functions are totally foreign to me and I would have never solved that on my own! 

$tImgData        = DllStructCreate(StringFormat("byte bytes[%i];", $iImgSize))
$tImgData.bytes  = FileRead($sImgFilePath)

That's like ninja level stuff. Is there somewhere I can learn more about the techniques you're using here? I can't make the mental leap from the DLLStructureCreate docs to what you created here, and the $tImgdata.bytes buffer.

[TheXman]

You're welcome! 

4 minutes ago, lowbattery said:

Is there somewhere I can learn more about the techniques you're using here?

If you are truly interested in learning lower-level coding, I'm sure you can find plenty of sources.    Most non-programmers don't have the desire to really take a deep dive into such topics.  Most people that do know that sort of stuff very well, are (or have been) getting paid to know it and/or do it - in other words, professional coders, instructors, and the like.   I guess learning and becoming proficient in C would definitely be the most direct route to gaining an understanding of how to consume C API's in other languages like AutoIt. 

Edited August 18, 2023 by TheXman