cURL response not accessible

[Mr_Microphone]

Alright, I may be an idiot.

Three years ago, I wrote a program that pushed component information to a secure site via their API. I went back to add some attributes and (here's the idiot part) ended up losing the  source code and my modified code does not quite work. I have the compiled version that works minus the new attributes, so I know that their system has not changed. I stripped the larger program down from 3,000 lines to the part that is broken, but I am stumped. This was one of my first scripts, so it heavily leverages examples and isn't as pretty as I'd like it to be.

Be gentle. 

The program / script creates a new records as expected, but for some reason, I cannot access information in the response, which I need for a later step.

I use Charles, a web debugging proxy tool so I can see the request and the response and both are as expected. Also, when I write to log file, the JSON reply is exactly what I expect and need, but when I try to do anything with the http body, it seems to be blank. 

Here is the script minus  the URL and token:

expandcollapse popup

#include <Array.au3>
#include <Curl.au3>
#include <MsgBoxConstants.au3>

#include <json.au3>  ; this was added as an alternate way to read the data

Global $WM_serial_number = "WM20745001"
Global $wm_component_status_id = "10"
Global $wm_manufacturer ="Multi-Tech"
Global $wm_model = "MTR-LAT1-B07"
Global $cellular_carrier_id = "3"
Global $iccid_esn = "89010303300012345678"
Global $ip_address = "192.168.2.11"
Global $NewIDNumber

    Local $Curl = Curl_Easy_Init()
    Local $Html = $Curl ; any number as identify
    Local $Header = $Curl + 1 ; any number as identify
    Local $HtmlFile = "cURL_Request.html"
    Local $File = FileOpen($HtmlFile, 2 + 16)
    Local $Slist = Curl_Slist_Append(0, "content-type: multipart/form-data; boundary=---011000010111000001101001")

    $Slist = Curl_Slist_Append($Slist, "authorization: Token token=" & $Token)

    Curl_Easy_Setopt($Curl, $CURLOPT_PROXY, "127.0.0.1") ; needed to use Charles web debugging proxy
    Curl_Easy_Setopt($Curl, $CURLOPT_PROXYPORT, 8888) ; needed to use Charles
    Curl_Easy_Setopt($Curl, $CURLOPT_HTTPHEADER, $Slist) ;
    Curl_Easy_Setopt($Curl, $CURLOPT_URL, $Server & "wireless_module" & "s")
    Curl_Easy_Setopt($Curl, $CURLOPT_SSL_VERIFYPEER, 0)
    Curl_Easy_Setopt($Curl, $CURLOPT_TIMEOUT, 30)
    Curl_Easy_Setopt($Curl, $CURLOPT_WRITEDATA, $Html)
    Curl_Easy_Setopt($Curl, $CURLOPT_WRITEFUNCTION, Curl_FileWriteCallback())
    Curl_Easy_Setopt($Curl, $CURLOPT_WRITEDATA, $File)

    Local $HttpPost = ""
    Local $LastItem = ""

        Curl_FormAdd($HttpPost, $LastItem, $CURLFORM_COPYNAME, "wireless_module" & "[serial_number]", $CURLFORM_COPYCONTENTS, $WM_serial_number, $CURLFORM_END)
        Curl_FormAdd($HttpPost, $LastItem, $CURLFORM_COPYNAME, "wireless_module" & "[component_status_id]", $CURLFORM_COPYCONTENTS, $wm_component_status_id, $CURLFORM_END)
        Curl_FormAdd($HttpPost, $LastItem, $CURLFORM_COPYNAME, "wireless_module" & "[manufacturer]", $CURLFORM_COPYCONTENTS, $wm_manufacturer, $CURLFORM_END)
        Curl_FormAdd($HttpPost, $LastItem, $CURLFORM_COPYNAME, "wireless_module" & "[model]", $CURLFORM_COPYCONTENTS, $wm_model, $CURLFORM_END)
        Curl_FormAdd($HttpPost, $LastItem, $CURLFORM_COPYNAME, "wireless_module" & "[cellular_carrier_id]", $CURLFORM_COPYCONTENTS, $cellular_carrier_id, $CURLFORM_END)
        Curl_FormAdd($HttpPost, $LastItem, $CURLFORM_COPYNAME, "wireless_module" & "[iccid_esn]", $CURLFORM_COPYCONTENTS, $iccid_esn, $CURLFORM_END)
        Curl_FormAdd($HttpPost, $LastItem, $CURLFORM_COPYNAME, "wireless_module" & "[ip_address]", $CURLFORM_COPYCONTENTS, $ip_address, $CURLFORM_END)

        ; submit

        Curl_Easy_Setopt($Curl, $CURLOPT_HTTPPOST, $HttpPost)
        Local $Code = Curl_Easy_Perform($Curl)
        If $Code = $CURLE_OK Then

        ConsoleWrite("Content Type: " & Curl_Easy_GetInfo($Curl, $CURLINFO_CONTENT_TYPE) & @LF)
        ConsoleWrite("Download Size: " & Curl_Easy_GetInfo($Curl, $CURLINFO_SIZE_DOWNLOAD) & @LF)

        MsgBox(0, 'Html', BinaryToString(Curl_Data_Get($Html))) ; this is something I threw in for debugging, expecting to see SOMETHING. Returns nothing
        MsgBox(0, 'Header', BinaryToString(Curl_Data_Get($Header))) ; this is something I threw in for debugging, expecting to see SOMETHING. Returns nothing

        Local $response = Curl_Easy_GetInfo($Curl, $CURLINFO_RESPONSE_CODE)

            If $response = "409" Then $response = "Failed due to a conflict."
            If $response = "200" Then $response = "Was NOT created."
            If $response = "201" Then $response = "Was created."


            ; read the ID that was assigned and store it
        $NewIDNumber = StringRight(StringLeft(BinaryToString(Curl_Data_Get($Html)), 10), 4) ; this DID work, but now it doesn't. An old compiled version still works

;~         Global $JsonObject = json_decode($Html); another debugging attempt. Did not use json functions previously and the program worked without it.
;~         Global $NewIDNumber = json_get($JsonObject, '.id')


        ConsoleWrite(@CRLF &'! id:' & $NewIDNumber & @CRLF & @CRLF)    ; debugging feedback
        MsgBox(0, $response, $wm_serial_number & " new ID = " & $NewIDNumber); debugging feedback

        If $Code <> $CURLE_OK Then ConsoleWrite(Curl_Easy_StrError($Code) & @LF)

            Local $Data = BinaryToString(Curl_Data_Get($Curl))

            Curl_Easy_Cleanup($Curl)
            Curl_Data_Cleanup($Curl)
            Curl_Data_Cleanup($Header)
            Curl_Data_Cleanup($Html)
            Curl_FormFree($HttpPost)
            Curl_slist_free_all($Slist)

            curl_easy_reset($Curl)

            FileClose($File)

            ConsoleWrite(@LF)
        EndIf 

This is the captured request (minus the host and token)

expandcollapse popup

POST /api/v2/wireless_modules HTTP/1.1
Host: api.
Accept: */*
authorization: Token token=
Content-Length: 942
Expect: 100-continue
content-type: multipart/form-data; boundary=---011000010111000001101001; boundary=------------------------9adb0d87c7ea5061

--------------------------9adb0d87c7ea5061
Content-Disposition: form-data; name="wireless_module[serial_number]"

WM20745001
--------------------------9adb0d87c7ea5061
Content-Disposition: form-data; name="wireless_module[component_status_id]"

10
--------------------------9adb0d87c7ea5061
Content-Disposition: form-data; name="wireless_module[manufacturer]"

Multi-Tech
--------------------------9adb0d87c7ea5061
Content-Disposition: form-data; name="wireless_module[model]"

MTR-LAT1-B07
--------------------------9adb0d87c7ea5061
Content-Disposition: form-data; name="wireless_module[cellular_carrier_id]"

3
--------------------------9adb0d87c7ea5061
Content-Disposition: form-data; name="wireless_module[iccid_esn]"

89010303300012345678
--------------------------9adb0d87c7ea5061
Content-Disposition: form-data; name="wireless_module[ip_address]"

192.168.2.11
--------------------------9adb0d87c7ea5061--

and the captured response

HTTP/1.1 201 Created
Date: Sun, 04 Apr 2021 00:12:18 GMT
Server: Apache
Cache-Control: max-age=0, private, must-revalidate
Access-Control-Allow-Origin: not-allowed
Vary: Accept-Encoding
Access-Control-Max-Age: 1728000
X-XSS-Protection: 1; mode=block
X-Request-Id: 71cfcf36-6020-48a6-a822-d2b393a27b69
Access-Control-Allow-Credentials: true
Access-Control-Allow-Methods: PUT, OPTIONS, GET, POST
ETag: W/"25d97fe8a9387cb4b9029a9e62b0bfa2"
X-Frame-Options: SAMEORIGIN, SAMEORIGIN
X-Runtime: 0.344005
X-Content-Type-Options: nosniff
Access-Control-Request-Method: *
X-Powered-By: Phusion Passenger 5.2.1
Strict-Transport-Security: max-age=63072000; includeSubDomains; preload
Location: /wireless_modules/3195
Status: 201 Created
Connection: close
Transfer-Encoding: chunked
Content-Type: application/json; charset=utf-8
X-Charles-Received-Continue: HTTP/1.1 100 Continue

{"id":3195,"model":"MTR-LAT1-B07","serial_number":"WM20745001","manufacturer":"Multi-Tech","mfg_date":null,"iccid_esn":"89010303300012345678","ip_address":"192.168.2.11","purchase_order":null,"supplier":null,"cellular_carrier_id":3,"component_status_id":10,"component_status":{"id":10,"name":"Hold","description":"Available- Held for specific use"},"custom_attributes":[{"name":"Deactivated","type":"Boolean","value":false},{"name":"Port 3001","type":"Boolean","value":false}],"comments":[]}

 

Also attached is the log file. I need to read the id value. Clearly, it is arriving back to cURL, since it is being written out to the log, but I cannot seem to get to it within the code. 

It is established that I may be an idiot, but this idiot has wasted days in non-billable hours trying to figure out what should be a simple glitch.

Help???

 

cURL_Request.html

Edited April 4, 2021 by Mr_Microphone

[argumentum]

1 hour ago, Mr_Microphone said:

wasted days in non-billable hours trying to figure out what should be a simple glitch.

I would use RunWaitEx() and just run Curl.exe. That way I have better control on what is going on. Not very elegant but simpler to troubleshoot

[TheXman]

Try changing this:

MsgBox(0, 'Html', BinaryToString(Curl_Data_Get($Html)))

to this:

MsgBox(0, 'Html', BinaryToString(Curl_Data_Get($Curl)))

and see if you get any output.

 

Curl_Data_Get() should be using the handle returned by Curl_Easy_Init() in order to get the response data, right?  I see you that you set $Html to the same as $Curl initially, but I don't know why though.  I have to admit that I have never used the curl.au3 UDF lib.  I'm just going by examples that I found.

Like @argumentum, I just use curl.exe directly, if I use it at all. 

Edited April 4, 2021 by TheXman

[TheXman]

*Edit:  Please disregard this question. 

If you are writing the response data to the handle named $File, then why aren't you opening and reading the response data from the $File handle?

Edited April 4, 2021 by TheXman

[Mr_Microphone]

11 minutes ago, TheXman said:

*Edit:  Just disregard this question. 

If you are writing the response data to the handle named $File, then why aren't you opening and reading the response data from the $File handle?

I know you said ignore it, but in case someone else wonders, the file was just debugging feature. Usually, they will upload many units in a session, so I don’t output files. The files would have sequential number IDs, but that got stripped out in my sample code. My fall-back will be to read the file and suck out the data, but it SHOULD work because it did before if I can only find what I changed by accident. 

[Mr_Microphone]

36 minutes ago, TheXman said:

Try changing this:

MsgBox(0, 'Html', BinaryToString(Curl_Data_Get($Html)))

to this:

MsgBox(0, 'Html', BinaryToString(Curl_Data_Get($Curl)))

and see if you get any output.

 

Curl_Data_Get() should be using the handle returned by Curl_Easy_Init() in order to get the response data, right?  I see you that you set $Html to the same as $Curl initially, but I don't know why though.  I have to admit that I have never used the curl.au3 UDF lib.  I'm just going by examples that I found.

Like @argumentum, I just use curl.exe directly, if I use it at all. 

I’ll try that in the morning. Thanks. I assumed that the cURL UDF was splitting the results into $header and $html (for the body) but you know what they say about assuming...

“don’t.” 

[TheXman]

<snip> 

See post below.

 

Edited April 4, 2021 by TheXman

[TheXman]

I went and read about the libcurl C APIs and looked at the functions in the UDF lib to see exactly what they're doing.  Then I created a small test example to confirm that what I've written below works.

 

Okay, assuming that everything else in you original script was correct, there is one line missing.  The following line should be added:

Curl_Easy_Setopt($Curl, $CURLOPT_WRITEFUNCTION, Curl_DataWriteCallback())  ;<== This line should be added right before the following line
Curl_Easy_Setopt($Curl, $CURLOPT_WRITEDATA, $Html)

The 2 lines above will write the response to $Html and you can read it with Curl_Data_Get($Html). 

 

Or, if you want the response written to the file, then delete or comment out the 2 lines above and leave the 2 lines below:

Curl_Easy_Setopt($Curl, $CURLOPT_WRITEFUNCTION, Curl_FileWriteCallback())
Curl_Easy_Setopt($Curl, $CURLOPT_WRITEDATA, $File)

 

Looking at your original script, it looks like you were using the first pair of lines because you said your original parsing logic was using $Html, and it used to work.

 

I'm quite sure that you can't have $CURLOPT_WRITEDATA write to a file and a variable in a single request.  That's because the $CURLOPT_WRITEFUNCTION needs to be set to either Curl_FileWriteCallback() or Curl_DataWriteCallback().  When I tested it using both pairs of definitions, it just used the last one that was defined.

 

And lastly. if you want the header info written to a variable, you can add the following two lines and retrieve it using Curl_Data_Get($Header).  If you want it written to a file, then change the callback function and set the HEADERDATA value to a file handle.

Curl_Easy_Setopt($Curl, $CURLOPT_HEADERFUNCTION, Curl_DataWriteCallback())
Curl_Easy_Setopt($Curl, $CURLOPT_HEADERDATA, $Header)

 

Edited April 4, 2021 by TheXman
Fied a typo

[pseakins]

4 hours ago, Mr_Microphone said:

ended up losing the  source code

Not wanting to brag, (actually I do), but I still have code I wrote in the seventies. It was punched onto cassette tape from my M68 dev board. I have no way of reading the tapes, but I still have the tapes.

Actually, I imagine it wouldn't be too hard to knock up a reader with an Arduino dev board if push came to shove.

[TheXman]

Just in case you wanted to see the test script that I created and it's output, I have posted it below.

The API URL is a site used for testing HTTP JSON API requests (GET, POST, PUT, PATCH, & DELETE) and their JSON responses.

expandcollapse popup

#AutoIt3Wrapper_UseX64 = N ;Required because Ward's UDF uses the 32-bit DLL.

#include <Constants.au3>
#include <curl.au3>


curl_post_example()

Func curl_post_example()
    Local $hCurl        = Null, _
          $hReqHeaders  = Null, _
          $hRespBody    = 1, _
          $hRespHeaders = 2

    Local $iRetCode    = 0, _
          $iRespCode   = 0

    Local $tByteBuffer = ""

    ;Create binary buffer and store http post data in it
    $tByteBuffer = DllStructCreate("byte buffer[1000];")
    If @error Then Exit MsgBox($MB_ICONERROR + $MB_TOPMOST, "ERROR", "Failed to create string buffer - @error = " & @error)
    $tByteBuffer.buffer = '{"fld1": "This is a test string", "fld2": true}'

    ;Initialize EasyCurl POST request and set options
    $hCurl = Curl_Easy_Init()

    $hReqHeaders = Curl_Slist_Append($hReqHeaders, "Content-Type: application/json")

    Curl_Easy_Setopt($hCurl, $CURLOPT_URL, "https://jsonplaceholder.typicode.com/posts")
    Curl_Easy_Setopt($hCurl, $CURLOPT_HTTPHEADER, $hReqHeaders)
    Curl_Easy_Setopt($hCurl, $CURLOPT_SSL_VERIFYPEER, 0)
    Curl_Easy_Setopt($hCurl, $CURLOPT_FOLLOWLOCATION, 1)
    Curl_Easy_Setopt($hCurl, $CURLOPT_WRITEFUNCTION, Curl_DataWriteCallback())
    Curl_Easy_Setopt($hCurl, $CURLOPT_WRITEDATA, $hRespBody)
    Curl_Easy_Setopt($hCurl, $CURLOPT_HEADERFUNCTION, Curl_DataWriteCallback())
    Curl_Easy_Setopt($hCurl, $CURLOPT_HEADERDATA, $hRespHeaders)
    Curl_Easy_Setopt($hCurl, $CURLOPT_POSTFIELDS, DllStructGetPtr($tByteBuffer))

    ;Submit request
    $iRetCode = Curl_Easy_Perform($hCurl)
    If $iRetCode <> $CurlE_OK Then Exit MsgBox($MB_ICONERROR + $MB_TOPMOST, "ERROR", "Request failed." & @CRLF & $iRetCode & " " & Curl_Easy_StrError($iRetCode))

    ;Display response info
    ConsoleWrite("Curl RetCode:   " & $iRespCode & " " & Curl_Easy_StrError($iRetCode) & @CRLF)
    ConsoleWrite("HTTP Resp Code: " & Curl_Easy_GetInfo($hCurl, $CurlINFO_RESPONSE_CODE) & @CRLF)
    ConsoleWrite("Content Type:   " & Curl_Easy_GetInfo($hCurl, $CurlINFO_CONTENT_TYPE)  & @CRLF)
    ConsoleWrite("Content Size:   " & Curl_Easy_GetInfo($hCurl, $CurlINFO_SIZE_DOWNLOAD) & @CRLF)
    ConsoleWrite(@CRLF & "Response Headers:" & @CRLF & BinaryToString(Curl_Data_Get($hRespHeaders), $SB_UTF8) & @CRLF)
    ConsoleWrite(@CRLF & "Response Body:"    & @CRLF & BinaryToString(Curl_Data_Get($hRespBody)   , $SB_UTF8) & @CRLF)

    ;Clean up
    Curl_Data_Cleanup($hRespBody)
    Curl_Data_Cleanup($hRespHeaders)
    Curl_Slist_Free_All($hReqHeaders)
    Curl_Easy_Cleanup($hCurl)
EndFunc

Console output:

expandcollapse popup

Curl RetCode:   0 No error
HTTP Resp Code: 201
Content Type:   application/json; charset=utf-8
Content Size:   66

Response Headers:
HTTP/1.1 201 Created
Date: Sun, 04 Apr 2021 19:16:34 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 66
Connection: keep-alive
Set-Cookie: __cfduid=dafce47f575693fda672d549581698e381617563794; expires=Tue, 04-May-21 19:16:34 GMT; path=/; domain=.typicode.com; HttpOnly; SameSite=Lax
X-Powered-By: Express
X-Ratelimit-Limit: 1000
X-Ratelimit-Remaining: 999
X-Ratelimit-Reset: 1617563839
Vary: Origin, X-HTTP-Method-Override, Accept-Encoding
Access-Control-Allow-Credentials: true
Cache-Control: no-cache
Pragma: no-cache
Expires: -1
Access-Control-Expose-Headers: Location
Location: http://jsonplaceholder.typicode.com/posts/101
X-Content-Type-Options: nosniff
Etag: W/"42-Ii9+oWyCYDJs2O27gjmkyyjiao8"
Via: 1.1 vegur
CF-Cache-Status: DYNAMIC
cf-request-id: 093fe9d5cb0000118d168d6000000001
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Report-To: {"max_age":604800,"group":"cf-nel","endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report?s=p6Ib9avjC%2BRZ95QeequJhUgrMYyRfdP8E9ozBaSKav3n8LqNyLiYuxucHEV1tpUp%2BIpcifjCY0ejYJOfvpZn3OM3vHGsRRmhvpofY0BDNHzLHCHIwk18oGDFwub9"}]}
NEL: {"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 63acdf3619fb118d-MIA
alt-svc: h3-27=":443"; ma=86400, h3-28=":443"; ma=86400, h3-29=":443"; ma=86400



Response Body:
{
  "fld1": "This is a test string",
  "fld2": true,
  "id": 101
}

 

Edited April 4, 2021 by TheXman

[Mr_Microphone]

15 hours ago, TheXman said:

I'm quite sure that you can have $CURLOPT_WRITEDATA write to a file and a variable in a single request.  That's because the $CURLOPT_WRITEFUNCTION needs to be set to either Curl_FileWriteCallback() or Curl_DataWriteCallback().  When I tested it using both pairs of definitions, it just used the last one that was defined.

This was EXACTLY the problem.

I had 

Curl_Easy_Setopt($Curl, $CURLOPT_WRITEFUNCTION, Curl_DataWriteCallback())
        Curl_Easy_Setopt($Curl, $CURLOPT_WRITEDATA, $Html)

originally, but when I was cutting and pasting for the post, I took it out by mistake.

in my "real" program, the later lines are

If $CreateResponseFiles = "true" Then Curl_Easy_Setopt($Curl, $CURLOPT_WRITEFUNCTION, Curl_FileWriteCallback())
        If $CreateResponseFiles = "true" Then Curl_Easy_Setopt($Curl, $CURLOPT_WRITEDATA, $File)

When I enabled the response file (via an ini) it "broke" the data write because, as you said, it only does one or the other.

By simply turning the response file option off, which is what I want in production, it works fine. Thanks you SO much. I am going to study how this is supposed to work now that I know where to look, but for now I am back in business!

[Mr_Microphone]

18 hours ago, argumentum said:

I would use RunWaitEx() and just run Curl.exe. That way I have better control on what is going on. Not very elegant but simpler to troubleshoot

That won't work for this particular project because the program needs to be stand-alone and portable, but I am glad the learn about the existence of Curl.exe and will definitely look at it for future projects. Thanks!

[Mr_Microphone]

22 minutes ago, TheXman said:

Just in case you wanted to see the test script that I created and it's output, I have posted it below.

The API URL is a site used for testing HTTP JSON API requests (GET, POST, PUT, PATCH, & DELETE) and their JSON responses.

That URL is going to be a HUGE help. Now that I am out of trouble, I will enjoy sifting through your example. Thanks again.

[TheXman]

 

You're welcome, I'm glad that I was able to help. 

 

30 minutes ago, Mr_Microphone said:

That won't work for this particular project because the program needs to be stand-alone and portable

If/When you are looking for other stand-alone, portable, alternatives for doing HTTP requests, there are several other solutions available like: WinHTTP.WinHTTPRequest.5.1 (COM), the WinHTTP.au3 UDF library of functions, msxml2.serverxmlhttp.6.0 (COM), and XMLHttpRequest (COM) -- all of which are intrinsic to the Windows operating system.  Unless it's necessary, I prefer to use the COM objects.  I find that they are much easier to implement and maintain, and they don't require as low of a level of understanding of all of the internals of making HTTP requests,  There are numerous examples throughout the forum of all of the ones that I've mentioned.

You may want to look into one of those alternatives because as you may or may not know, Ward is no longer active in the forum and pulled a lot of his work.  So you probably will not see a 64-bit version of the cURL UDF library unless someone decides to take it on as a project.  And if you run across a bug in the version of the DLL that he used, getting some sort of maintenance done in relation to that UDF will, again, be up to someone else.  But as long as all is good, I guess if ain't broke, don't fix it.