How can I directly access / manipulate bytes in binary data?

[AnyOne2]

I extracted some binary data using the function BinaryMid.
Now I want to modify some of those byte before interpreting the whole binary sequence as UTF-16LE string.

How to do this?

Note: searching for UDFs I found one called Binary.UDF (by Ward), but I seems not to be available anymore; the function "_BinaryPoke" would have been very helpful I think...

Edited January 16, 2017 by AnyOne2

[anthonyjr2]

I found a copy of the UDF on Github:

https://github.com/fediaFedia/Omnimo/blob/master/AutoIT/Includes/Binary.au3

[AnyOne2]

Thanks for the quick response!
I'll have a look if this really fits my needs

[AnyOne2]

OK, here's my conclusion after some tries with this Binary.au3: the manipulations of Binaries using DLLStructs are incredible slow.
The compiled AutoIt script needs about 100 seconds, using PHP the same algorithm finishes after just 2 seconds.

So why using AutoIt? Because I need to be able to compile the algorithm and distribute it as self-contained executable...

Any hints?

[AndyG]

9 hours ago, AnyOne2 said:

the manipulations of Binaries using DLLStructs are incredible slow.
The compiled AutoIt script needs about 100 seconds, using PHP the same algorithm finishes after just 2 seconds.

Manipulating of Binaries using DLLStructs is incredibly FAST (if you know what to do!)

Without a Script to verify your statement, any other discussion is useless...please show a little example script. 

[AnyOne2]

19 hours ago, AndyG said:

Manipulating of Binaries using DLLStructs is incredibly FAST (if you know what to do!)

You're right, this was probably a hasty conclusion, I'm sorry.

So here's a simplified version of my code to show you what I'm doing in my script.
I would really appreciate it if someone could tell me why it runs so slow.

expandcollapse popup

; open input file
$hFileIn = FileOpen($sFilePath, $FO_READ + $FO_BINARY)
$dBinary = FileRead($hFileIn)
FileClose($hFileIn)
$iBinLen = BinaryLen($dBinary)

; open output file
$sFileOut = $sFilePath & '.txt'
$hFileOut = FileOpen($sFileOut, $FO_OVERWRITE)

$iOffset = 4

Do
  ; extracting length
  $iLenStr = Int(BinaryMid($dBinary, $iOffset+8, 1)) + (256 * Int(BinaryMid($dBinary, $iOffset+9, 1)))

  ; get raw text
  $dRawStr = BinaryMid($dBinary, $iOffset, $iLenStr)
  $iOffset += $iLenStr + 10

  ; adapt and convert text
  $iVal = Mod(iLenStr, 32) ; added/updated
  adapt($dRawStr, $iVal);
    $sText = BinaryToString($dRawStr, 2) ; 2=UTF16LE

  ; write output
  FileWriteLine($hFileOut, $sText)

    $iCnt += 1
    If $iCnt >= 500 Then
        ConsoleWrite('*')
        $iCnt = 0
  EndIf

Until $iOffset >= $iBinLen

FileClose($hFileOut)

;------- INTERNAL FUNCTION ------------------------

Func adapt(ByRef $dBinary, $iVal)

    ; create C struct from binary variable
    Local $dBuffer = DllStructCreate("byte[" & (BinaryLen($dBinary)) & "]")
    DllStructSetData($dBuffer, 1, $dBinary)

    ; loop through structure and add val to all bytes
    For $i=1 To BinaryLen($dBinary) Step 1
        $iNewValue = DllStructGetData($dBuffer, 1, $i) + $iVal
        DllStructSetData($dBuffer, 1, $iNewValue, $i);
    Next

    ; write result back to binary variable
    $dBinary = DllStructGetData($dBuffer, 1)

EndFunc

 

Edited January 14, 2017 by AnyOne2

[AndyG]

Hi,

do you have an example file to convert and a value for $iVal?

I think you have been caught by the trap called XY-Problem. 

The goal is to add a (always the same?) value to some bytes within a file?!

The job should be done within some milliseconds...

[Melba23]

AndyG,

Love the link!

M23

[AnyOne2]

You're right again, I forgot to include the following line (I now updated my initial post):

$iVal = Mod($iStrLen,32)

it shows that the value of $iVal changes for each loop iteration (between 0 and 31).

Another info: the input files have sizes between 5 and 10 MB, each loop iteration handles about 250 bytes (to average).
My PHP script takes 2260 ms for a file of about 6.5 MB (Win10, Intel i7-4900MQ 2.8 GHz, 16GB RAM, 500 GB SSD).

Thank you for bearing with me!

Edited January 14, 2017 by AnyOne2

[AnyOne2]

I solved my speed problems, the script now finishes within 6-8 seconds, that's acceptable for me.

What did I do?
Instead of reading the file as binary, I read it as ANSI string and used StringToASCIIArray to get an array of integers in the range of 0-255.
Like this I could do all my manipulations on the array and I made the byte-swap (to read UTF16LE) myself before calling StringFromASCIIArray.

The conversion to an array takes about 2 seconds, 4-6 seconds are needed to process the file (7MB).
I'm happy

Thanks for your replies anyway!

Edited January 15, 2017 by AnyOne2