Most efficient ways to handle binary?

[EddieBoy]

Most of the tools I make with Autoit handle data of the binary type, but I don't know efficient ways to process binary in Autoit.

EDITED to simplify:

Can anyone point me toward a tutorial on how to manipulate binary data efficiently in Autoit? Specifically, I need to read an entire binary file (such as a photo) into a variable, then have the ability to efficiently split and edit portions of the binary as needed. I get the impression that DllStructCreate is the area I should be looking at, but trying to comprehend the helpfile on this topic (which is 70% alien programmer-isms) is daunting since I'm not even sure that's the right track.

I'm a relative beginner, and just a hobbyist so sorry if I'm asking a stupid question.  

Edited December 1, 2017 by EddieBoy

[EddieBoy]

Edited my question (above) to be more concise.

For other boobs...I mean noobs like me who are trying to optimize their code, Here's a link to the AU3Profiler tool which I found extremely helpful:

 https://www.autoitscript.com/forum/topic/106811-au3profiler/



 

[RTFC]

On 12/1/2017 at 2:42 AM, EddieBoy said:

a tutorial on how to manipulate binary data

Dllstructs are explained at the end of this Tutorial.

On 12/1/2017 at 2:42 AM, EddieBoy said:

DllStructCreate is the area I should be looking at

Absolutely. Image files will have complicated binary header structures. You'll need to create a struct that duplicates that structure, then read the entire header part of your image file into it. Then you can access the content of individual items with DllstructGet/SetData.

[LarsJ]

EddieBoy, If you are talking about the raw image data you need a technique as demonstrated here to read data from a binary file to a DllStruct.

[EddieBoy]

Thank you both very much. I'll study those tutorials. My old coding methods where taking a couple seconds just to do something simple like flip an image. This will be a huge help!

 

EDIT: A quote from the tutorial PDF linked by RTFC:

Quote

Another good thing is that it allows direct memory access in AutoIt.

I've handled assembly language a fair bit and this is exactly what I was missing when using Autoit. It also highlights the usefulness of the  code Larsj provided. Thank you both so much- this opens whole new ways to process binary files.

Edited December 3, 2017 by EddieBoy

[RTFC]

11 hours ago, EddieBoy said:

handled assembly language a fair bit

Since you're obviously not a beginner, should you be on x64, you may wish to give my HighMem UDF a spin, if it's of any potential use to you.

[EddieBoy]

Thank you, I will check that out.

I am actually a relative beginner; I just went about learning in a strange backwards way. My experience with MIPS based assembly language came from doing fan translations of old console games. That required doing things like adding support for variable width English fonts at the assembly level. I only started to dig into high level programming later. Because high level programming is far more abstract than assembly, I find it generally harder to learn, but easier to use.

[EddieBoy]

Processing is much faster for binary images thanks to RTFC  LarsJ. I'm still a bit perplexed by limitations. For example, the function below converts a 16bit image in xBGR order into 24bit  RGB.  It takes about 2 seconds to process a small 300x300 image.  In assembly, it would take a fraction of a second.
 

 The biggest slowdown  seems to come from the DLLStructSetData() which is run 3 times per pixel to store the converted color values to their new structure.  This was necessary to extract 1 byte from the 4byte long values. I know all numbers are 32bit to a 32bit processor, but at the assembly level, you use a command like 'store byte' to quickly discard the upper 24bits.
 

Is there a faster way 'extract' and merge single bytes from multiple 32bit values?  BinaryMid() is painfully slow.

expandcollapse popup

Func _16to24(ByRef $BMPimage)
    ;BMPimage = structure with 16bit image data

    Local $pointer, $size, $NewBMPimage, $count = 1

    ;calculate number of 2byte blocks (aka pixel count)
    $size = DllStructGetSize($BMPimage) / 2

    ;get pointer to image location
    $pointer = DllStructGetPtr($BMPimage)

    ;define new structure sized for 2byte (1pixel) processing
    $temp = DllStructCreate('USHORT [' & $size & ']', $pointer)

    ;define new structure for 24bit image (3bytes per pixel)
    $NewBMPimage = DllStructCreate('ubyte image[' & $size * 3 & ']')

    ;convert 1 pixel per loop from 16bit tBGR > RGB 24bit
    For $i = 1 To $size

        ;Get 1 pixel
        $2byte = DllStructGetData($temp, 1, $i)

        ;extract 5bit colorchannels, convert to 8bit, then store in new structure
        DllStructSetData($NewBMPimage, 1, BitShift($2byte, 10) * 8.22, $count) ;blue
        DllStructSetData($NewBMPimage, 1, BitShift(BitShift($2byte, -22), 27) * 8.22, $count + 1) ;green
        DllStructSetData($NewBMPimage, 1, BitAND(0x001F, $2byte) * 8.22, $count + 2) ;red

        $count += 3
    Next

    ;clear unneeded structures
    $temp = 0
    $BMPimage = 0

    Return $NewBMPimage
EndFunc

 

Edited December 7, 2017 by EddieBoy

[RTFC]

Tried these? See also here.

[LarsJ]

EddieBoy, Then implement the For-loop in assembler. The zip-file below contains a complete set of resources and templates to use flat assembler code in AutoIt scripts. However, you must download FASM.DLL from the flat assembler forum yourself. FASM.DLL is dated 2017-10-20.

Contents in zip-file:

• fasm\ - Top folder
  • fasm\ - Resources
    • FASM.DLL - Save FASM.DLL here
    • *.au3 - 3 AutoIt include files
  • MsgBox\ - Templates based on MessageBoxW function
    • x64\MsgBox-x64.asm - 64 bit assembler source code
    • x64\MsgBox-x64.au3 - Compiles and runs the assembler code
    • x64\MsgBox-x64.bin - Contains the binary code (generated by MsgBox-x64.au3)
    • x64\MsgBox-x64.txt - Binary code as text string (generated by MsgBox-x64.au3)
    • x64\MsgBox-x64.log - Information about compilation (generated by MsgBox-x64.au3)
    • x86\ - 32 bit templates
    • MsgBox-1.au3 - Loads and executes fasm code from binary string
    • MsgBox-2.au3 - Loads and executes fasm code from binary file

This is the x64 and x86 source files. MsgBox-x64.asm:

; flat assembler code

; Documentation
        ; MessageBoxW( hWnd, pText, pCaption, iType )

        ; Parameters:
        ; rcx        : hWnd
        ; rdx        : pText
        ; r8         : pCaption
        ; r9         : iType
        ; [rsp + 40] : pMessageBoxW

; Init directive
        use64                                    ; 64 bit code

; Function code
        mov        rax,      qword [rsp + 40]    ; pMessageBoxW -> rax

        sub        rsp,      40                  ; Stack space and alignment
        call       rax                           ; Call MessageBoxW
        add        rsp,      40                  ; Restore stack pointer

; Exit code
        ret                                      ; Return

MsgBox-x86.asm:

; flat assembler code

; Documentation
        ; MessageBoxW( hWnd, pText, pCaption, iType )

        ; Parameters:
        ; [ebp + 08] : hWnd
        ; [ebp + 12] : pText
        ; [ebp + 16] : pCaption
        ; [ebp + 20] : iType
        ; [ebp + 24] : pMessageBoxW

; Init directive
        use32                                    ; 32 bit code

; Entry code
        push       ebp                           ; Store base pointer on stack
        mov        ebp,                esp       ; Use stack pointer as base pointer

; Function code
        push       dword [ebp + 20]              ; 4. parameter: iType
        push       dword [ebp + 16]              ; 3. parameter: pCaption
        push       dword [ebp + 12]              ; 2. parameter: pText
        push       dword [ebp + 08]              ; 1. parameter: hWnd
        call       dword [ebp + 24]              ; Call MessageBoxW

; Exit code
        pop        ebp                           ; Restore base pointer from stack
        ret        20                            ; Return and cleanup stack

You can try to run the AutoIt scripts in MsgBox folder.

Make a copy of the entire MsgBox folder and rename it for your own project. Delete bin-, txt- and log-files. Rename asm- and au3-files for your project. Start coding flat assembler.

fasm.7z

fasm-NoBins.7z (see post 12 and 13 below)

Edited February 26, 2018 by LarsJ
fasm-NoBins.7z

[EddieBoy]

Wow, being able to run assembly from within Autoit would be extremely useful for some types of processing. If I'm understanding the Binary.au3 by Ward, it seems to forward assembly-code snippets along with arguments to Windows DLLs for processing; similarly FLAT.DLL lets you build assembly code in memory and run it from there.  I'm speculating that  Binary.au3 involves a fair amount of overhead since it creates a new copy of the assembly-code each time it's run, whereas FLAT uses a pointer to the assembly-code location thus letting it be reused without generating a new copy. That may be completely wrong. I will have to do some studying to get up to speed, but this knowledge will be very helpful. Thank you both for your tips and patience

[Bilgus]

The bin files in the archive gave my virus scanner a fit I had to un7z with the command line to even get it on my machine

Flat assembler is for sure the way to go... way better than looking up opcodes!!

Thanks

[LarsJ]

Bilgus, I'm sorry about that. I've not had any false viruses with Avira. I've added a new version of the 7z-file without the binary files.

[Bilgus]

Rather than hard coding the fasm version it seems the latest version of the dll has a version function exposed

Local $iFasmVer = DllCall( $sFasmDll, "int", "fasm_GetVersion")[0]
    Local $sFasmVer = BitAnd($iFasmVer, 0x0000FFFF) & "." & BitShift(BitAnd($iFasmVer, 0xFFFF0000), 16)

As for Virus warnings I kind of expect it from small assembly snippets / small binaries

[LarsJ]

JoeBar, If you are interested in using compiled code with AutoIt code, then there is a brief summary at the bottom of this thread.