**SOLVED - Curl Json Post send base64 encoded instead of utf8 text

[Roy_]

Ciao,

This example reproduces the problem I encounter when sending a post request to an echo server.
Instead of being sent as text utf8, the data is transmitted base64 encoded.

Example:
- json String: '{"firstName":"Jonathan","lastName":"Freeman","loginCount":4,"active": "yes","text":"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Morbi vel volutpat nunc. Maecenas id tempus mi. Morbi ipsum enim, ultricies ac augue sit amet, ullamcorper finibus ex. Vestibulum vel posuere nibh, nec faucibus eros. Nam malesuada non lacus a suscipit. Nulla rhoncus tempus mi quis placerat. Curabitur commodo tincidunt justo quis sollicitudin."}'

- server response: "data:application/octet-stream;base64,AAAAALgE6QNYAIAAAgAAANQCAAAAAAAAWF8AAAAAAAD0VQAAAAAAAD9APQAAAAAAAwAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAAAABBAAAAAAABRUAAAAmIZwcAbRt8/LMa9UAAAAAAAAAAGFzIGlkIHRlbXB1cyBtaS4gTW9yYmkgaXBzdW0gZW5pbSwgdWx0cmljaWVzIGFjIGF1Z3VlIHNpdCBhbWV0LCB1bGxhbWNvcnBlciBmaW5pYnVzIGV4LiBWZXN0aWJ1bHVtIHZlbCBwb3N1ZXJlIG5pYmgsIG5lYyBmYXVjaWJ1cyBlcm9zLiBOYW0gbWFsZXN1YWRhIG5vbiBsYWN1cyBhIHN1c2NpcGl0LiBOdWxsYSByaG9uY3VzIHRlbXB1cyBtaSBxdWlzIHBsYWNlcmF0LiBDdXJhYml0dXIgY29tbW9kbyB0aW5jaWR1bnQganVzdG8gcXVpcyBzb2xsaWNpdHVkaW4uIn0="

 

How can I go about transmitting data in text format utf8?
Thanks in advance for the help.

In the zip file:
- CurlJsonPost.au3: this script
- Curl.au3: UDF by Ward (thank you!)
- data.json: json srting for command line test

 

Note: using Curl.exe with json string saved in a file (utf8 encoded) named data.json, works perfectly
To try with the command line tool:
- save data.json in curl\bin directory
- open cmd.exe and cd to curl\bin directory
- Enter the following command:
curl -H "Content-Type: application/json" --data @data.json https://httpbin.org/post

 

expandcollapse popup

#Include "Curl.au3"

Global $_cURL_OutputBuffer

Local $sJson  = '{"firstName":"Jonathan","lastName":"Freeman","loginCount":4,"active": "yes","text":"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Morbi vel volutpat nunc. Maecenas id tempus mi. Morbi ipsum enim, ultricies ac augue sit amet, ullamcorper finibus ex. Vestibulum vel posuere nibh, nec faucibus eros. Nam malesuada non lacus a suscipit. Nulla rhoncus tempus mi quis placerat. Curabitur commodo tincidunt justo quis sollicitudin."}'
Local $iRetCode, $sServerResponse

ConsoleWrite("=== Json post test ===" & @LF)
ConsoleWrite(StringFormat("Retcode: %s - %s", $iRetCode, Curl_Easy_strerror($iRetCode)) & @LF)
ConsoleWrite("Data returned from server" & @LF & @LF)
$sServerResponse = JsonPost_Test($sJson, $iRetCode)
ConsoleWrite($sServerResponse & @LF)


Func JsonPost_Test($sJson, ByRef $iRetCode)

    ; Init Easy Curl Interface e set url (echo service)
    Local $oCurl = Curl_Easy_Init()
    curl_easy_setopt($oCurl, $CURLOPT_URL, "https://httpbin.org/post")

    ; Set content type header
    Local $headers = curl_slist_append(0, "Content-Type: application/json")
    curl_easy_setopt($oCurl, $CURLOPT_HTTPHEADER, $headers)

    ; Post fields & size
    curl_easy_setopt($oCurl, $CURLOPT_POSTFIELDS, $sJson)
    curl_easy_setopt($oCurl, $CURLOPT_POSTFIELDSIZE, StringLen($sJson))

    ; Set callbac function to get server response back (see global var $_cURL_OutputBuffer)
    $hWriteFunc = DllCallbackRegister("WriteFunc_CallBack", "uint:cdecl", "ptr;uint;uint;ptr")
    curl_easy_setopt($oCurl, $CURLOPT_WRITEFUNCTION, DllCallbackGetPtr($hWriteFunc))

    ; Ignore ssl certificates check
    curl_easy_setopt($oCurl, $CURLOPT_SSL_VERIFYPEER, 0)
    curl_easy_setopt($oCurl, $CURLOPT_SSL_VERIFYHOST, 0)

    ; Execute the post request
    $iRetCode = curl_easy_perform($oCurl)

    ; Set return trasfer & clear output buffer global var
    Local $sReturnTransfer = $_cURL_OutputBuffer
    $_cURL_OutputBuffer = ""

    Return SetError(0, 0, $sReturnTransfer)
EndFunc

Func WriteFunc_CallBack($ptr,$nSize,$nMemb,$pStream)
    Local $vData = DllStructCreate ("byte[" & $nSize*$nMemb & "]",$ptr)
    $_cURL_OutputBuffer &= BinaryToString(DllStructGetData($vData,1))
    Return $nSize*$nMemb
EndFunc

 

curlJsonPost.zip

Edited February 5, 2020 by Roy_

[Roy_]

I have just found the solution and I want to share it with you

The secret lies in passing the json data as a pointer instead of as a string.

So changing these lines ...

; Post fields & size
    curl_easy_setopt($oCurl, $CURLOPT_POSTFIELDS, $sJson)

with these:

; Post fields & size
    Local $stString = DllStructCreate("char v[" & StringLen($sJson) & "]")
    DllStructSetData($stString, 1, $sJson)
    curl_easy_setopt($oCurl, $CURLOPT_POSTFIELDS, DllStructGetPtr($stString))

and it works perfectly, returning this response:

=== Json post test ===
Retcode:  - No error
Data returned from server

{
  "args": {}, 
  "data": "{\"firstName\":\"Jonathan\",\"lastName\":\"Freeman\",\"loginCount\":4,\"active\": \"yes\",\"text\":\"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Morbi vel volutpat nunc. Maecenas id tempus mi. Morbi ipsum enim, ultricies ac augue sit amet, ullamcorper finibus ex. Vestibulum vel posuere nibh, nec faucibus eros. Nam malesuada non lacus a suscipit. Nulla rhoncus tempus mi quis placerat. Curabitur commodo tincidunt justo quis sollicitudin.\"}", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Content-Length": "437", 
    "Content-Type": "application/json", 
    "Host": "httpbin.org", 
    "X-Amzn-Trace-Id": "Root=1-5e3a9a12-f59424ee2b283d6ca7874c48"
  }, 
  "json": {
    "active": "yes", 
    "firstName": "Jonathan", 
    "lastName": "Freeman", 
    "loginCount": 4, 
    "text": "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Morbi vel volutpat nunc. Maecenas id tempus mi. Morbi ipsum enim, ultricies ac augue sit amet, ullamcorper finibus ex. Vestibulum vel posuere nibh, nec faucibus eros. Nam malesuada non lacus a suscipit. Nulla rhoncus tempus mi quis placerat. Curabitur commodo tincidunt justo quis sollicitudin."
  }, 
  "origin": "87.8.171.14", 
  "url": "https://httpbin.org/post"
}

Attached correct version.

Roy

CurlJsonPostOk.au3

[Roy_]

... and finally if you have to send utf-8 characters like àèéìòù and so on:

1) specify this header

; Set content type header & specify charset
    Local $headers = curl_slist_append(0, "Content-Type: application/json;charset=UTF-8")
    curl_easy_setopt($oCurl, $CURLOPT_HTTPHEADER, $headers)

2) convert your json to utf-8 charset like this

; Post fields & size
    Local $sUtf8 = _WinAPI_WideCharToMultiByte($sJson, 65001)
    curl_easy_setopt($oCurl, $CURLOPT_POSTFIELDS, $sUtf8)

that's all.

Enjoy curl