blob object & Protocol Buffers

[jugador]

blob object ==>  
https://developer.mozilla.org/en-US/docs/Web/API/Blob

protobuf (Protocol Buffers) ==>
https://developers.google.com/protocol-buffers
https://developers.google.com/protocol-buffers/docs/overview

Now problem is when connecting to site via websocket some of the site sends binary data as blob object / protobuf.

I thought I could able to decode the data using base64 code available on forum.
I also try https://www.base64decode.org for decoding the data but no luck.

On google search find code / example to transfer this blob object & protobuf to readable string but all are written in python or javascript.
So how do I decode the blob object & protobuf data to readable string in Autoit?

If asked I can provide link of the site I am trying to connect.
Thanks.

Edited August 26, 2021 by jugador

[Danp2]

Yes, I would suggest providing some additional details such as --

• URL for site you are accessing
• Example of data being returned
• Links for "find code / example to transfer this blob object & protobuf to readable string but all are written in python or javascript"

[jugador]

for Protocol Buffers type data =>
check stackoverflow link it contain  site Name + returned data sample + python github link
https://stackoverflow.com/questions/59321908/decode-websocket-received-data

or to view sample data under ws tab of network monitor

Protocol Buffers sample data

blob object sample data

and to extract data from a Blob ==> https://stackoverflow.com/a/23024504
https://developer.mozilla.org/en-US/docs/Web/API/FileReader

[mLipok]

Is this mean that using webdriver it should be possible to catch downloaded pdf directly as a BinaryFormat instead as a file ?

Edited August 26, 2021 by mLipok

[Danp2]

Looks like a standard base64 encoded string to me. "No luck" doesn't really help explain why it didn't work for you. I'm guessing that you forgot to call BinaryToString --

expandcollapse popup

$sData = "CgRBTVpOFaQY3EQY4Kn0/99bKgNOTVMwCDgBRYjKzDxIyvN9ZQBQ4T7YAQQ="
$bData = _Base64Decode($sData)

ConsoleWrite($bData & @crlf)
ConsoleWrite(BinaryToString($bData) & @crlf)

Func _Base64Decode($input_string)

    Local $struct = DllStructCreate("int")

    $a_Call = DllCall("Crypt32.dll", "int", "CryptStringToBinary", _
            "str", $input_string, _
            "int", 0, _
            "int", 1, _
            "ptr", 0, _
            "ptr", DllStructGetPtr($struct, 1), _
            "ptr", 0, _
            "ptr", 0)

    If @error Or Not $a_Call[0] Then
        Return SetError(1, 0, "") ; error calculating the length of the buffer needed
    EndIf

    Local $a = DllStructCreate("byte[" & DllStructGetData($struct, 1) & "]")

    $a_Call = DllCall("Crypt32.dll", "int", "CryptStringToBinary", _
            "str", $input_string, _
            "int", 0, _
            "int", 1, _
            "ptr", DllStructGetPtr($a), _
            "ptr", DllStructGetPtr($struct, 1), _
            "ptr", 0, _
            "ptr", 0)

    If @error Or Not $a_Call[0] Then
        Return SetError(2, 0, ""); error decoding
    EndIf

    Return DllStructGetData($a, 1)

EndFunc   ;==>_Base64Decode

I haven't looked at the blob portion yet.

[jugador]

@Danp2

"No luck" define using base64 decode failed to get readable string.
Now when you run this code you get readable data / string

$sData = "SGVsbG8gV29ybGQ="
$bData = _Base64Decode($sData)

ConsoleWrite($bData & @crlf)
ConsoleWrite(BinaryToString($bData) & @crlf)

so on running your code do do you get readable output?

If you scroll down stackoverflow link(provided above) you will see on proper decode you will get string like this....

"{"id":"PCG","price":10.97,"time":"1576616325000","exchange":"NYQ","quoteType":"EQUITY","marketHours":"REGULAR_MARKET","changePercent":13.443642,"dayVolume":"31495549","change":1.3000002,"priceHint":"2"}"

https://stackoverflow.com/a/59382147

Quote

Protobuf is open technology which allows to create marshalling code for multiple languages automatically when meta description of message is known.

the given stackoverflow link contain
> .proto (protobuf meta description) file
> protobuf compiler process
> python github link

Please check the given link on this thread to know how Protobuf work

[Danp2]

@jugador I apparently misread your original post as I thought that you said the web based decode gave you the desired output. It would have been clearer if you had posted the desired output in your OP.

Maybe someone else will jump in here with some ideas for you. 🙂

[Danp2]

Here's an initial attempt at decoding a protobuf. Anyone is welcome to make improvements / corrections --

expandcollapse popup

$sData = "CgNQQ0cVH4UvQRiQvr/a4lsqA05ZUTAIOAFFKBlXQUj61YQeZWhmpj/YAQQ="

;~ Results from https://protogen.marcgravell.com/decode
;~ Field #1: 0A String Length = 3, Hex = 03, UTF8 = "PCG"
;~ Field #2: 15 Fixed32 Value = 1093633311, Hex = 1F-85-2F-41
;~ Field #3: 18 Varint Value = 3153232650000, Hex = 90-BE-BF-DA-E2-5B
;~ Field #5: 2A String Length = 3, Hex = 03, UTF8 = "NYQ"
;~ Field #6: 30 Varint Value = 8, Hex = 08
;~ Field #7: 38 Varint Value = 1, Hex = 01
;~ Field #8: 45 Fixed32 Value = 1096227112, Hex = 28-19-57-41
;~ Field #9: 48 Varint Value = 62991098, Hex = FA-D5-84-1E
;~ Field #12: 65 Fixed32 Value = 1067869800, Hex = 68-66-A6-3F
;~ Field #27: D8-01 Varint Value = 4, Hex = 04

$bData = _Base64Decode($sData)
ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $bData = ' & $bData & @CRLF) ;### Debug Console

$iPos = 1
$iLen = BinaryLen($bData)

While $iPos <= $iLen
    $bTag = BinaryMid($bData, $iPos, 1)

    $bField = BitShift($bTag, 3)
    $iWire  = BitAND($bTag, 7)

;~  ConsoleWrite("+> $iPos = " & $iPos & " - $bTag = " & $bTag & " - $bField = " & $bField & " - $iWire = " & $iWire  & @crlf)

    If BitAND($bField, 16) Then
        $bTag = BinaryMid($bData, $iPos + 1, 1) & BitAND($bField, 16)
        $iPos += 1
    EndIf

    Switch $iWire
        Case 0 ; VarInt
            $iEnd = $iPos + 1

            While $iEnd <= $iLen
                $bCompare = BinaryMid($bData, $iEnd, 1)
                If Not BitAND($bCompare, 128) Then ExitLoop

                $iEnd += 1
            WEnd

            $sResult = BinaryToString(BinaryMid($bData, $iPos + 1, $iEnd - $iPos))
            ConsoleWrite("(" & $iPos & ") " & $bField & " - " & Binary($sResult) & @crlf)
            $iPos = $iEnd

        Case 1 ; Fixed64

        Case 2 ; Length-delimited
            $bLen = BinaryMid($bData, $iPos + 1, 1)
            $sResult = BinaryToString(BinaryMid($bData, $iPos + 2, $bLen))

            ConsoleWrite("(" & $iPos & ") " & $bField & " - " & $sResult & @crlf)
            $iPos += Number($bLen) + 1

        Case 3 ; start group

        Case 4 ; end group

        Case 5 ; Fixed32
            $sResult = BinaryToString(BinaryMid($bData, $iPos + 1, 4))
            ConsoleWrite("(" & $iPos & ") " & $bField & " - " & Binary($sResult) & @crlf)
            $iPos += 4

    EndSwitch

    $iPos += 1
WEnd

Func _Base64Decode($input_string)

    Local $struct = DllStructCreate("int")

    $a_Call = DllCall("Crypt32.dll", "int", "CryptStringToBinary", _
            "str", $input_string, _
            "int", 0, _
            "int", 1, _
            "ptr", 0, _
            "ptr", DllStructGetPtr($struct, 1), _
            "ptr", 0, _
            "ptr", 0)

    If @error Or Not $a_Call[0] Then
        Return SetError(1, 0, "") ; error calculating the length of the buffer needed
    EndIf

    Local $a = DllStructCreate("byte[" & DllStructGetData($struct, 1) & "]")

    $a_Call = DllCall("Crypt32.dll", "int", "CryptStringToBinary", _
            "str", $input_string, _
            "int", 0, _
            "int", 1, _
            "ptr", DllStructGetPtr($a), _
            "ptr", DllStructGetPtr($struct, 1), _
            "ptr", 0, _
            "ptr", 0)

    If @error Or Not $a_Call[0] Then
        Return SetError(2, 0, ""); error decoding
    EndIf

    Return DllStructGetData($a, 1)

EndFunc   ;==>_Base64Decode

 

[jugador]

Thanks @Danp2

Sorry for the trouble and it’s just because I don’t know python.
So thinking off opening a new thread to ask how to install python 😅 & run this github code.

;~ CgNQQ0cVH4UvQRiQvr/a4lsqA05ZUTAIOAFFKBlXQUj61YQeZWhmpj/YAQQ=
;~ Field #1: "PCG"      
;~ Field #2: 1093633311     ==> 10.97
;~ Field #3: 3153232650000  ==> 1576616325000
;~ Field #5: "NYQ"
;~ Field #8: 1096227112     ==> 13.4436416625977
;~ Field #9: 62991098       ==> 31495549
;~ Field #12: 1067869800    ==> 1.30000019073486


ConsoleWrite("-> price = " & IntToFlt(1093633311) & @crlf)              ;~ price = 10.9700002670288
ConsoleWrite("-> changePercent = " & IntToFlt(1096227112) & @crlf)      ;~ changePercent = 13.4436416625977
ConsoleWrite("-> change = " & IntToFlt(1067869800) & @crlf)             ;~ change = 1.30000019073486

ConsoleWrite("-> time = " & (3153232650000/2) & @crlf)                  ;~ time = 1576616325000
ConsoleWrite("-> dayVolume = " & (62991098/2) & @crlf)                  ;~ dayVolume = 31495549

Func IntToFlt($iVal)
; https://www.autoitscript.com/forum/topic/53842-integer-to-float-conversion/?do=findComment&comment=407905
Local $result
$fs = DllStructCreate("float")
$is = DllStructCreate("int64",DllStructGetPtr($fs))

DllStructSetData($is,1,$iVal)

$result =  DllStructGetData($fs,1)
$fs = 0
$is = 0
return $result
endfunc

 

Edited August 31, 2021 by jugador

[Danp2]

FWIW, I don't know Python either. I just found discussions like this one and then attempted to code it in AutoIt.

[jugador]

able to convert Hex to Fixed32 Value correctly....

;~ Field #2: 15 Fixed32 Value = 1093633311, Hex = 1F-85-2F-41
ConsoleWrite(__Hex2Dec('0x1F852F41', 3) & @CRLF)    ;~ 1093633311

; #FUNCTION# =============================================================================
; Name...........: __Hex2Dec
; =========================================================================================
Func __Hex2Dec($x_Hex, $x_StartPoint = 1)
    Local $o_Temp
    For $i = $x_StartPoint To StringLen($x_Hex) Step 2
        $o_Temp = StringMid($x_Hex, $i, 2) & $o_Temp
    Next
    Return Dec($o_Temp, 0)
EndFunc ;==>__Hex2Dec

but failed to convert Hex to Varint Value correctly.

;~ Field #3: 18 Varint Value = 3153232650000, Hex = 90-BE-BF-DA-E2-5B

@Danp2

help needed to convert     0x90BEBFDAE25B ==> 3153232650000
or is there way to get the value within the above code posted by you.

Edited September 2, 2021 by jugador

[Nine]

I think you got an error somewhere :

ConsoleWrite(Hex(3153232650000) & @CRLF)

shows it is 0x02DE2B4FDF10.  In no way related to your hex...

Here another way to convert binary string to dec :

$bVal = Binary("0x1F852F41")
$tStruct = DllStructCreate("int data")
$tStruct.data = $bVal
ConsoleWrite($tStruct.data & @CRLF)

 

Edited September 2, 2021 by Nine

[jugador]

@Nine  if you look at @Danp2 code  you will see this at the beginning

;~ Field #1: 0A String Length = 3, Hex = 03, UTF8 = "PCG"
;~ Field #2: 15 Fixed32 Value = 1093633311, Hex = 1F-85-2F-41
;~ Field #3: 18 Varint Value = 3153232650000, Hex = 90-BE-BF-DA-E2-5B
;~ Field #5: 2A String Length = 3, Hex = 03, UTF8 = "NYQ"
;~ Field #6: 30 Varint Value = 8, Hex = 08
;~ Field #7: 38 Varint Value = 1, Hex = 01
;~ Field #8: 45 Fixed32 Value = 1096227112, Hex = 28-19-57-41
;~ Field #9: 48 Varint Value = 62991098, Hex = FA-D5-84-1E
;~ Field #12: 65 Fixed32 Value = 1067869800, Hex = 68-66-A6-3F
;~ Field #27: D8-01 Varint Value = 4, Hex = 04

 

you can crosscheck the base64(date) using this site https://protogen.marcgravell.com/decode

CgNQQ0cVH4UvQRiQvr/a4lsqA05ZUTAIOAFFKBlXQUj61YQeZWhmpj/YAQQ=

 

for protobuf varints check this link https://developers.google.com/protocol-buffers/docs/encoding#varints

 

check this Python code on https://www.tutorialspoint.com/execute_python3_online.php
you will get same result as per protogen.marcgravell.com

# ;~ Field #9: 48 Varint Value = 62991098, Hex = 0xFAD5841E
import struct

def pack_varint(val):
    total = b''
    if val < 0:
        val = (1<<32)+val
    while val>=0x80:
        bits = val&0x7F
        val >>= 7
        total += struct.pack('B', (0x80|bits))
    bits = val&0x7F
    total += struct.pack('B', bits)
    return total

print ('protobuf varints ==> ' , pack_varint(62991098));
print('Hex ==> ' , hex(62991098))

 

python code for Hex to Varints but it's giving error....

def unpack_varint(buff):
    total = 0
    shift = 0
    val = 0x80
    while val&0x80:
        val = struct.unpack('B', buff.read(1))[0]
        total |= ((val&0x7F)<<shift)
        shift += 7
    if total&(1<<31):
        total = total - (1<<32)
    return total

 

Edited September 2, 2021 by jugador

[Nine]

I see.  I was not aware of those VarInt things.   So here how to convert VarInt Hex to Decimal (up to 64 bits) :

#AutoIt3Wrapper_UseX64=y
#include "x64_Ops.au3"

Local $bData = Binary("0x90BEBFDAE25B")
;Local $bData = Binary("0xFAD5841E")
Local $iResult = 0, $iShift = 0
For $i = 1 To BinaryLen($bData)
  $iResult += _BitSHIFT64(BitAND(BinaryMid($bData, $i, 1), 0x7F), $iShift)
  $iShift -= 7
Next

ConsoleWrite($iResult & @CRLF)

You will need to use my x64 UDF to make it work though (see my signature).

Edited September 3, 2021 by Nine

[jugador]

Thanks @Nine
hope not to disturb you guys on protobuf issue anymore

@Nine one more thing to ask
do you know what blob object is and how to decode it.

can you please have a look at the blob object sample data

under  ws(websocket) Tab of network monitor (<= Chrome / Firefox)

will get json string (guessing) on decoding the blob object.

Edited September 3, 2021 by jugador

[Nine]

44 minutes ago, jugador said:

do you know what blob object is and how to decode it

No idea...

[Nine]

Integrated my snippets inside @Danp2 example to create a more generalized script :

expandcollapse popup

#AutoIt3Wrapper_UseX64=y
#include "x64_Ops.au3"
#include <Constants.au3>

Local Const $aType = ["VarInt", "Fixed64", "String", "Start Group", "End Group", "Fixed32"]

$sData = "CgNQQ0cVH4UvQRiQvr/a4lsqA05ZUTAIOAFFKBlXQUj61YQeZWhmpj/YAQQ="

;~ Results from https://protogen.marcgravell.com/decode
;~ Field #1: 0A String Length = 3, Hex = 03, UTF8 = "PCG"
;~ Field #2: 15 Fixed32 Value = 1093633311, Hex = 1F-85-2F-41
;~ Field #3: 18 Varint Value = 3153232650000, Hex = 90-BE-BF-DA-E2-5B
;~ Field #5: 2A String Length = 3, Hex = 03, UTF8 = "NYQ"
;~ Field #6: 30 Varint Value = 8, Hex = 08
;~ Field #7: 38 Varint Value = 1, Hex = 01
;~ Field #8: 45 Fixed32 Value = 1096227112, Hex = 28-19-57-41
;~ Field #9: 48 Varint Value = 62991098, Hex = FA-D5-84-1E
;~ Field #12: 65 Fixed32 Value = 1067869800, Hex = 68-66-A6-3F
;~ Field #27: D8-01 Varint Value = 4, Hex = 04

$bData = _Base64Decode($sData)
ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $bData = ' & $bData & @CRLF) ;### Debug Console

Local $iPos = 1, $iLen = BinaryLen($bData), $iTag, $sResult, $iResult

While $iPos <= $iLen
  $sResult = VarInt($bData, $iPos, $iTag)
  $iField = BitShift($iTag, 3)
  $iWire = BitAND($iTag, 7)
  ConsoleWrite("Field #" & $iField & ": " & $sResult & " " & $aType[$iWire])

  Switch $iWire
    Case 0     ; VarInt
      $sResult = VarInt($bData, $iPos, $iResult)
      ConsoleWrite(" Value = " & $iResult & " Hex = " & $sResult & @CRLF)
    Case 1     ; Fixed64
      $sResult = BinaryMid($bData, $iPos, 8)
      $iResult = Fixed($sResult)
      ConsoleWrite(" Value = " & $iResult & " Hex = " & Hex($sResult) & @CRLF)
      $iPos += 8
    Case 2     ; Length-delimited
      $bLen = BinaryMid($bData, $iPos, 1)
      $sResult = BinaryToString(BinaryMid($bData, $iPos + 1, $bLen), $SB_UTF8)
      ConsoleWrite(" (" & Int($bLen) & ") = " & $sResult & @CRLF)
      $iPos += Number($bLen) + 1
    Case 3     ; start group
      ConsoleWrite(@CRLF)
    Case 4     ; end group
      ConsoleWrite(@CRLF)
    Case 5     ; Fixed32
      $sResult = BinaryMid($bData, $iPos, 4)
      $iResult = Fixed($sResult)
      ConsoleWrite(" Value = " & $iResult & " Hex = " & Hex($sResult) & @CRLF)
      $iPos += 4
  EndSwitch
WEnd

Func _Base64Decode($input_string)
  Local $struct = DllStructCreate("int")
  $a_Call = DllCall("Crypt32.dll", "int", "CryptStringToBinary", _
      "str", $input_string, _
      "int", 0, _
      "int", 1, _
      "ptr", 0, _
      "ptr", DllStructGetPtr($struct, 1), _
      "ptr", 0, _
      "ptr", 0)
  If @error Or Not $a_Call[0] Then Return SetError(1, 0, "")   ; error calculating the length of the buffer needed

  Local $a = DllStructCreate("byte[" & DllStructGetData($struct, 1) & "]")
  $a_Call = DllCall("Crypt32.dll", "int", "CryptStringToBinary", _
      "str", $input_string, _
      "int", 0, _
      "int", 1, _
      "ptr", DllStructGetPtr($a), _
      "ptr", DllStructGetPtr($struct, 1), _
      "ptr", 0, _
      "ptr", 0)
  If @error Or Not $a_Call[0] Then Return SetError(2, 0, "")  ; error decoding
  Return DllStructGetData($a, 1)
EndFunc   ;==>_Base64Decode

Func VarInt(ByRef Const $bStream, ByRef $iPos, ByRef $iResult)
  Local $iVal, $iStart = $iPos, $iShift
  $iResult = 0
  While True
    $iVal = BinaryMid($bStream, $iPos, 1)
    $iResult += _BitSHIFT64(BitAND($iVal, 0x7F), $iShift)
    $iShift -= 7
    $iPos += 1
    If Not BitAND($iVal, 0x80) Then ExitLoop
  WEnd
  Return Hex(BinaryMid($bStream, $iStart, $iPos - $iStart))
EndFunc   ;==>VarInt

Func Fixed($bVal)
  Local $iLen = BinaryLen($bVal)
  Local $tStruct = DllStructCreate(($bVal = 4 ? "int" : "int64") & " data")
  $tStruct.data = $bVal
  Return $tStruct.data
EndFunc

 

Edited September 3, 2021 by Nine

[jugador]

@Nine & @Danp2 Idea is to decode Protobuf data to json ==> https://stackoverflow.com/a/59382147

CgRBTVpOFaQY3EQY4Kn0/99bKgNOTVMwCDgBRYjKzDxIyvN9ZQBQ4T7YAQQ=
"{"id":"PCG","price":10.97,"time":"1576616325000","exchange":"NYQ","quoteType":"EQUITY","marketHours":"REGULAR_MARKET","changePercent":13.443642,"dayVolume":"31495549","change":1.3000002,"priceHint":"2"}"

ProtobufToJson

prerequisite => @Nine x64 bitwise operations udf
link to Proto schema for this code

ProtobufToJson.au3

 

@Danp2 can you test this code with your autoit-websocket udf

Local $aSites[1][3] = [['streamer.finance.yahoo.com', '', '{"subscribe":["AMZN"]}']]
ConsoleWrite("Received message from the server: '" & __Protobuf_ToJson(BinaryToString($bRecv)) & "'" & @CRLF)

 

Edited September 4, 2021 by jugador

[Danp2]

5 hours ago, jugador said:

can you test this code with your autoit-websocket udf

For some reason the existing example is no longer running correctly for me. Have you tried to use the UDF?

[jugador]

@Danp2  forgot to mention.
Saturday & Sunday market close so price will not update as its live stream.
If you monitor AMZN data under ws Tab you will see no incoming message.

when market open you will see price changing .

you can try on Monday during market hour.

Edited September 4, 2021 by jugador