Binary UDF

[Ward]

"Binary" is a built-in variable type in AutoIt. But functions that deal with binary variable are inadequate. Someone says string functions can also deal with binary variable, but actually, it may cause some problem. For example, using StringInStr to search "0x11" in "0x0110" will return a false result.

 

This UDF solves the problem. It provide a lot of binary related functions. Furthermore, most functions in the UDF are optimized by machine code, so the speed is incredible.

 

Here is the function list:

 

_BinaryAnd($Binary1, $Binary2)
_BinaryOR($Binary1, $Binary2)
_BinaryXOR($Binary1, $Binary2)
_BinaryNot($Binary)
_BinaryShift($Binary, $Shift)
_BinaryRotate($Binary, $Shift)
_BinaryReverse($Binary)
_BinaryInBin($Binary, $Search, $Occur = 1, $Start = 1)
_BinaryReplace($Binary, $Search, $Replace, $Occur = 0)
_BinarySplit($Binary, $Search)
_BinaryRight($Binary, $Count)
_BinaryLeft($Binary, $Count)
_BinaryTrimLeft($Binary, $Count)
_BinaryTrimRight($Binary, $Count)

The usage of these functions are just like the built-in StringXXXX or BitXXXX functions.

 

Attentions:

BinarySplit has a little bit different, it don't support flag parameter in StringSplit.

BinaryInBin and BinaryReplace also don't need casesense parameter.

 

 

_BinaryPeek($Binary, $Start, $Type = "byte")
_BinaryPoke($Binary, $Start, $Value, $Type = "byte")
_BinaryToInt32($Binary)
_BinaryFromInt32($Value)
_BinaryToInt64($Binary)
_BinaryFromInt64($Value)
_BinaryRandom($Length, $Start = 0, $To = 255)

These functions are some helper function.

 

BinaryPeek and BinaryPoke work like stardand BASIC keyword peek and poke. They can handle types defined in DllStructCreate and DllCall (include str and wstr). The $Start are also like other AutoIt functions, count from 1, not from 0.

 

BinaryRandom generate a random binary data for test or something. It has built-in Mersenne Twister PRNG.

 

BinaryToXXX and BinaryFromXXX convert variable type between binary and int32/int64.

 

 

_BitShift64($Value, $Shift)
_BitRotate64($Value, $Shift)
_BitNOT64($Value)
_BitOR64($Value1, $Value2, [$Value3, ...])
_BitAND64($Value1, $Value2, [$Value3, ...])
_BitXOR64($Value1, $Value2, [$Value3, ...])
_Hex64($Value, $Length = 16)
_Dec64($HexString)

These functions are just the same as built-in functin in the same name, but these can handle 64 bit signed integer.

(Built-in function only for 32 bit integer)

 

Binary UDF.zip

 

 

2011/09/15 Update Note:

• Add following functions:

   

  _BinaryToInt16($Binary)
  _BinaryFromInt16($Value)
  _BinaryToDouble($Binary)
  _BinaryFromDouble($Value)
  _BinaryToMemory($Binary, $Ptr)
  _BinaryFromMemory($Ptr, $Size)
  _BinaryToDLLStruct($Binary)
  _BinaryFromDLLStruct(ByRef $DLLStruct)
  _BinaryFillDLLStruct($Binary, ByRef $DLLStruct)

• _BinaryRandom can specify the random seed now. $Seed = 0 mean no seed is specified.

   

  _BinaryRandom($Length, $Start = 0, $To = 255, $Seed = 0)

• Use static variable to store the binary code to avoid repeat memory allocation.

Binary UDF.zip

Edited November 23, 2015 by Ward

[wraithdu]

Excellent! The 64-bit binary functions will really be useful.

[ynbIpb]

Hi Ward!

Thank you for this UDF

I have a problem: I need to replace a few bytes of binary data. But replace the data is longer than the original data. In this situation it would fit function _BinaryReplace, but it changes, not where I want. I know the exact start Offset from which data and their length.

example:

Input data: 00 00 11 00 11 00 11 00 00 00 ; I need to replace the red

Output data: 00 00 11 00 FF FF 00 11 00 00 00 ; like this

or

Input data: 00 00 11 00 11 00 11 00 00 00

Output data: 00 00 11 00 FF 11 00 00 00

I do not know exactly what it is first or last, but I know the offset and size

It is possible to refine the function _BinaryReplace, so she was not looking for a string and tell it the exact offset

sorry for bad english

[Ward]

Hi Ward!

Thank you for this UDF

I have a problem: I need to replace a few bytes of binary data. But replace the data is longer than the original data. In this situation it would fit function _BinaryReplace, but it changes, not where I want. I know the exact start Offset from which data and their length.

Since you know the offset to start replace, maybe you can use BinaryMid() to get part of binary data before _BinaryReplace().

[MvGulik]

Interesting ... Bummer I don't have 64bit capabilities yet.

---

Collection some stuff while playing around with it ...

• Could use some '#forceref's (5x) to make it not trip on main scripts that use '#AutoIt3Wrapper_AU3Check_Parameters=-q -d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6 -w 7'
• Some way to control random with some seed feature. (if possible)
• _BinaryRandom() ... speed is, well

  [1] |St| A | 1.74 mSec.   | 256 * Random()
  [2] |St| B | 145.95 mSec. | 65536 * Random()
  [3] |St| A | 105.52 mSec. | 256 * _BinaryRandom(8)
  [4] |St| B | 26.31 Sec.   | 65536 * _BinaryRandom(8)

  Probably because function startup penalty. ... (taking a other approche to using it) ...

  [4] |St| B | 31.68 mSec.  | _BinaryRandom(8*65536)

  That's more like it. But Random() not leaving much slack to play around with.
• ...

Edited August 28, 2011 by iEvKI3gv9Wrkd41u

[Ward]

@iEvKI3gv9Wrkd41u

Thanks for your testing.

_BinaryRandom (or other functions use machine code) convert HEX to binary and call it every time.

A way to avoid it is use static variable, for example:

Func _BinaryRandom2($Length, $Start = 0, $To = 255)
    If $Length = 0 Then Return Binary("")
    Static $CodeBufferPtr

    If Not $CodeBufferPtr Then
        Local $Code
        If @AutoItX64 Then
            $Code = Binary("0x4157415641554154555756534881ECF80900004538C876034587C185D20F847F0100004C8D5C2410450FB6C9410FB6C04183C10183EA0148C78424D0090000000000004129C1498D8338060000488D5C1101498D730431D2498DBB3406000048894424084D8D93C0090000EB4789C2C1EA0B31C289D0C1E0072580562C9D31D031D289C5C1E50F81E50000C6EF31C589E8C1E81231E841F7F1418D141088114883C1014839D90F84F6000000488B9424D00900004885D20F84850000004C8BBC24E00900004C8BA424D80900004D8D6F044C89E0448B308B2A4C89A424D00900004983C4044C89AC24E00900004C89A424D80900004489F081E5000000804183E60125FFFFFF7F41F7DE09E84181E6DFB00899D1E84431F04133074D39D5890274164D39D40F8542FFFFFF4C899C24D8090000E935FFFFFF4C899C24E0090000EBE00F3131D04989F4BD010000008944241089C2C1EA1E31C269C26589076C01E883C501418904244983C40481FD7002000075DE4C899C24D00900004889B424D80900004889F04889BC24E00900004C8B6C24084C89DA4989FF4989F4E932FFFFFF4881C4F80900005B5E5F5D415C415D415E415FC3")
        Else
            $Code = Binary("0x5557565381ECE40900000FB68424000A0000884424130FB68424040A000038442413760A0FB654241392885424138B9424FC09000085D20F84740100000FB65424130FB6C031DB83C0018DB424D8090000C78424D8090000000000008D4C241829D031D28944240C893424EB5289C2C1EA0B31C289D0C1E0072580562C9D31D031D289C6C1E60F81E60000C6EF31C689F0C1E81231F0F774240C8B8424F80900000254241388141883C301399C24FC0900000F86F90000008B9424D809000085D20F84850000008BAC24E00900008BBC24DC0900008D750489F8897424048B008B3289BC24D809000083C70489BC24DC0900008944240825FFFFFF7F81E60000008009F08B742408D1E883E601F7DE81E6DFB0089931F08B342433450089028B54240439F2899424E009000074153B3C240F8536FFFFFF898C24DC090000E92AFFFFFF898C24E0090000EBE20F3131D0BE010000008944241889C2C1EA1E31C269C26589076C01F08904B183C60181FE7002000075E38D44241C8D94244C06000089C78DB42450060000899424E00900008DAC244C06000089CA898C24D8090000898424DC09000089742404E92DFFFFFF81C4E40900005B5E5F5DC21000")
        EndIf
        $CodeBufferPtr = __BinaryCodeBufferAlloc($Code)
    EndIf

    Local $Buffer = DllStructCreate("byte[" & $Length & "]")
    Local $Ret = DllCall($__Binary_User32Dll, "uint", "CallWindowProc", "ptr", $CodeBufferPtr, _
                                                                        "ptr", DllStructGetPtr($Buffer), _
                                                                        "uint", $Length, _
                                                                        "byte", $Start, _
                                                                        "byte", $To)
    Return DllStructGetData($Buffer, 1)
EndFunc

This function is faster than the old one.

But the memory space can't be free (it seems not important).

Is convert all functions to this format necessary?

[wraithdu]

I think the Static method is good practice to avoid repeated memory allocations. You could always build in a memory clearing feature to the function if it is important to free that buffer at some point.

[MvGulik]

This function is faster than the old one.

Definitely puts a nice dent in the time needed.

[5] |St| B    | 7.51 Sec.    | 65536 * _BinaryRandom_Static(8)

But the memory space can't be free (it seems not important).

Yep. Speed versus Memory use. Don't think where talking that much memory here to. (The Asm code only by the look of it.)

Is convert all functions to this format necessary?

I personally would use Static ... Without it the general function call(s) seem way to slow for any serious(high call rate) use. Even with Static use the function startup penalty's are not to be ignored. (for those functions that are available in AutoIt itself -> 32-bit one's)

e: forum troubles ...

---

Alternative (older) way/topic for some 64-bit support: by jchd.

Edited August 30, 2011 by iEvKI3gv9Wrkd41u

[Ward]

@iEvKI3gv9Wrkd41u

What you suggested is added now.

[czardas]

Interesting thread. I occasionally use bitwise functions, but I mostly use string functions with binary strings. Why? Perhaps, as you say,. because the available bitwise functions seem inflexible and extremely limited. The motivation for creating my modal UDF was aimed at modular arithmatic, binary being the first target base. I don't know all the methods and I still believe that there must be other solutions (perhaps using Struct which I haven't tried yet). My recent three dimensional bit spin topic uses strings of ones and zeros, because the bitwise functions don't hack it.

[MvGulik]

@iEvKI3gv9Wrkd41u

What you suggested is added now.

Cool.

[MvGulik]

Just to see if _BinaryRandom() is compatible in some way with AutoIt's Random().

#include "Binary.au3"
Global Const $INT32u = Int('0x0100000000') ;; (2^32), int64. (4294967296)

_Test5(8, 2, 1000) ;; Count, Seed, LoopIt.

Func _Test5(Const $iCount = 1, Const $iSeed=0, Const $iLoopIt = 1) ;; $iSeed=0 -> random seed.
    Local Const $iBytes = 1, $iMax = Int(2^(8*$iBytes))-1 ;0, 255, 65535, ...
    Local $R = ''

    For $i = 1 to $iLoopIt

    ;; AutoIt random() output.
    SRandom($iSeed)
    Local $bFirstBytes = Binary('')
    For $1 = 1 To $iCount
        $R = Floor(Random() * ($INT32u + 1))
        $bFirstBytes &= BinaryMid($R,1,1)
    Next

    ;; _BinaryRandom() output.
    $R =  _BinaryRandom($iCount, 0, $iMax, $iSeed)

    If $R <> $bFirstBytes Then
        MsgBox(0,'MsgBox','Oops: $R <> $bFirtsBytes ?')
        Exit 911
    EndIf

    Next
    MsgBox(0,'MsgBox',' :) ')
EndFunc

Although I do wonder a bit what happened to the other bytes.

E: Oops, forgot some things.

+ _BinaryRandom() probably using one Byte where Random it using 4 bytes.

Edited September 16, 2011 by iEvKI3gv9Wrkd41u

[wraithdu]

Just thought I'd toss out there that the UDF works great with the new DllCallAddress() function in 3.3.7.15 beta (ignore the Au3Check warnings about wrong number of parameters, that will be fixed in the next beta). Anyone feel like running a speed test versus CallWindowProc? This would also allow for an eventual rewrite with more sane parameters.

Edited September 16, 2011 by wraithdu

[Ward]

Finally we have standard support for DllCallAddress. Not need my MemoryFuncCall anymore. . I can develop my machine code library more easily.

I thank DllCallAddress should much faster than MemoryFuncCall, but have no much difference to CAllWindowProc. (just guess, not yes test.)

[step887]

Hi Ward,

Sorry to post on this old topic. however, I looking for assistance, I am looking parse large binary data, most of it is 0x00, which I can replace with binaryreplace. So that works amazing.

but I am looking to remove everything below 0x20(except 0xA and 0xD) and 0x80 and above. While I could use binaryreplace to go through all of them using loop, I was wondering if there was a way to use this function like _BinaryReplace($Binary, '>0x7F', '0x')

Thanks,

Stephen

[jchd]

Try this:

Local $ansi
For $i = 0 To 255
    $ansi &= Chr($i)
Next
Local $ascii7 = StringRegExpReplace($ansi, "[^\x0A\x0D\x20-\x7F]", "")
_ArrayDisplay(StringToASCIIArray($ascii7))

BTW do you really want to keep 0x7F (the "delete" control character)?

Edited August 13, 2014 by jchd

[step887]

Thanks jchd, I do not think I would have ever thought about using regex this way.

And your right I did not want 0x7F

Edited August 13, 2014 by step887

[AnyOne2]

Where can I find this UDF?

It seems the links are not valid anymore ...

[argumentum]

On 1/12/2017 at 9:39 AM, AnyOne2 said:

Where can I find this UDF?

gone. I have a version from 2011.09.15, not the last one from 2015 !, so, if it doesn't work as is, well, good luck.
Binary UDF[2011.09.15].zip

Edited December 21, 2017 by argumentum

[anthonyjr2]

The one you have (from 2011) is the most recent version, I'm not sure why Ward removed all of his UDFs.