Fastest way to read and work with a binary file - (Locked)

[Invicore]

Hi, Im working on a script that I need to work with a Binary file Inside of loop and read its bytes Consecutive , (also be able to Go back and forth inside the file and etc)

But I dont know I need to use what method so its not bad for performance 

My first attempt was to use FileRead with FileHandle, But sadly This made the speed of the program slower as the program progressed and read further from file  (Program finish after like 30 min for a 10MB file)

expandcollapse popup

$File = FileOpen($Name, 0 + 16)
If $File = -1 Then
    MsgBox(0, "Error", "Can't open " & $Name & " file.")
    Exit
EndIf
$Magic = FileRead($File, 4)
If $Magic <> "0x43523257" Then ;Magic
    MsgBox(0, "Error", $Name & " is not a valid File.")
    Exit
EndIf
$NewFile = $Magic
$NewFile &= FileRead($File, 36)
$NewFile &= FileRead($File, 120)
FileSetPos($File, 40, 0)

Local $StringOffset = 0
Local $SkipSize = 0
For $Table = 0 To 400 ;Just example, its need to loop like 60K
    $Offset = _BinaryToInt32(FileRead($File, 4))
    $itemCount = _BinaryToInt32(FileRead($File, 4))
    $CRC32 = _BinaryToInt32(FileRead($File, 4))
    If $Table == 0 Then ;Strings
        $StringOffset = $Offset
        $SkipSize += $itemCount
    ElseIf $Table == 1 Then ;CNames
        $BeforeCName = FileGetPos($File)
        $SkipSize += ($itemCount * 8)
        FileSetPos($File, $Offset, 0)
        Global $Cname[$itemCount]
        For $CNames = 0 To $itemCount - 1
            $SOffset = _BinaryToInt32(FileRead($File, 4))
            $SHash = _BinaryToInt32(FileRead($File, 4))
            $pos = FileGetPos($File)
            FileSetPos($File, $StringOffset + $SOffset, 0)
            $S = ""
            $Str = ""
            Do
                $Str &= $S
                $S = FileRead($File, 1)
            Until $S = 0
            $Str = BinaryToString("0x" & StringRegExpReplace($Str, "0x", ""), 4)
            $Cname[$CNames] = $Str
            ;ConsoleWrite($Str & @CRLF)
            FileSetPos($File, $pos, 0)
        Next
        FileSetPos($File, $BeforeCName, 0)
    ElseIf $Table == 3 Then ;CR2WProperty
        $SkipSize += ($itemCount * 16)
    ElseIf $Table == 4 Then ;CR2WExports
        FileSetPos($File, $Offset, 0)
        $SkipSize += ($itemCount * 24)
        For $Exports = 1 To $itemCount
            $stringIndex_ClassName = _BinaryToInt16(FileRead($File, 2))
            $ObjectFlags = _BinaryToInt16(FileRead($File, 2))
            $parentID = _BinaryToInt32(FileRead($File, 4))
            $dataSize = _BinaryToInt32(FileRead($File, 4))
            $dataOffset = _BinaryToInt32(FileRead($File, 4))
            $template = _BinaryToInt32(FileRead($File, 4))
            $CRC32 = _BinaryToInt32(FileRead($File, 4))
        Next
    EndIf
Next

So I tried to first read whole File into a variable and then binary from it with BinaryMid

Here is a example

expandcollapse popup

;POS
Global $GNOW = 1
;=>End

$File = FileOpen($Name, 0 + 16)
$File_R = FileRead($File, -1)
If $File = -1 Then
    MsgBox(0, "Error", "Can't open " & $Name & " file.")
    Exit
EndIf
$Magic = _BinaryRead(4)
If $Magic <> "0x43523257" Then ;Magic
    MsgBox(0, "Error", $Name & " is not a valid File.")
    Exit
EndIf
$NewFile = $Magic
$NewFile &= _BinaryRead(36)
$NewFile &= _BinaryRead(120)
_BinarySetPos(41)
;FileSetPos($File, 40, 0)

Local $StringOffset = 0
Local $SkipSize = 0
For $Table = 0 To 4
    $Offset = _BinaryToInt32(_BinaryRead(4))
    $itemCount = _BinaryToInt32(_BinaryRead(4))
    $CRC32 = _BinaryToInt32(_BinaryRead(4))
    If $Table == 0 Then ;Strings
        $StringOffset = $Offset
        $SkipSize += $itemCount
    ElseIf $Table == 1 Then ;CNames
        $BeforeCName = _BinaryGetPos()
        $SkipSize += ($itemCount * 8)
        _BinarySetPos($Offset+1)
        Global $Cname[$itemCount]
        For $CNames = 0 To $itemCount - 1
            $SOffset = _BinaryToInt32(_BinaryRead(4))
            $SHash = _BinaryToInt32(_BinaryRead(4))
            $pos = _BinaryGetPos()
            _BinarySetPos($StringOffset + $SOffset + 1)
            $S = ""
            $Str = ""
            Do
                $Str &= $S
                $S = _BinaryRead(1)
            Until $S = 0
            $Str = BinaryToString("0x" & StringRegExpReplace($Str, "0x", ""), 4)
            $Cname[$CNames] = $Str
            ;ConsoleWrite($Str & @CRLF)
            _BinarySetPos($pos)
        Next
        _BinarySetPos($BeforeCName)
    ElseIf $Table == 3 Then ;CR2WProperty
        $SkipSize += ($itemCount * 16)
    ElseIf $Table == 4 Then ;CR2WExports
        _BinarySetPos($Offset+1)
        $SkipSize += ($itemCount * 24)
        For $Exports = 1 To $itemCount
            $stringIndex_ClassName = _BinaryToInt16(_BinaryRead(2))
            $ObjectFlags = _BinaryToInt16(_BinaryRead(2))
            $parentID = _BinaryToInt32(_BinaryRead(4))
            $dataSize = _BinaryToInt32(_BinaryRead(4))
            $dataOffset = _BinaryToInt32(_BinaryRead(4))
            $template = _BinaryToInt32(_BinaryRead(4))
            $CRC32 = _BinaryToInt32(_BinaryRead(4))
        Next
    EndIf
Next

Func _BinaryRead($iCount)
  $GNOW += $iCount
  Return BinaryMid($File_R, $GNOW - $iCount, $iCount)
EndFunc

But this method was even more slower...

so I want to know what can I do for reading Bin file as fast as possible?

sorry if its not a good question, Im new in autoit

Edited March 19, 2021 by Invicore

[JockoDundee]

So these programs actually work correctly, but are just slow?

Where’s all the code, like _BinarySetPos, and other functions?

Where’s the sample input file?

Why the -1 here:

$File_R = FileRead($File, -1)

 

[Invicore]

1 hour ago, JockoDundee said:

So these programs actually work correctly, but are just slow?

Where’s all the code, like _BinarySetPos, and other functions?

Where’s the sample input file?

Why the -1 here:

$File_R = FileRead($File, -1)

_BinaryToInt* defined in a UDF Name Binary

Set Pose and get pos are just two simple function

Func _BinarySetPos($iPos = 0)
  $GNOW = $iPos
EndFunc

Func _BinaryGetPos()
    Return $GNOW
EndFunc

and about -1 in FileRead, it read all The file into $File_R (I know it possible to just not define anything, but its possible too)

Edited March 19, 2021 by Invicore

[Jos]

33 minutes ago, Invicore said:

$Magic = _BinaryRead(4) 
If $Magic <> "0x43523257" Then 
   ;Magic
   MsgBox(0, "Error", $Name & " is not a valid File.")
   Exit 
EndIf

 

You say this is actually working? This part doesn't test for a hex code of 4 characters but in stead for a literal string, so doubt that if is ever true.

 

[Nine]

And why loop in first example is 

For $Table = 0 To 400

While in second example is 

For $Table = 0 To 4

And why do you assign repeatedly to the same variables 

For $Exports = 1 To $itemCount
            $stringIndex_ClassName = _BinaryToInt16(_BinaryRead(2))
            $ObjectFlags = _BinaryToInt16(_BinaryRead(2))
            $parentID = _BinaryToInt32(_BinaryRead(4))
            $dataSize = _BinaryToInt32(_BinaryRead(4))
            $dataOffset = _BinaryToInt32(_BinaryRead(4))
            $template = _BinaryToInt32(_BinaryRead(4))
            $CRC32 = _BinaryToInt32(_BinaryRead(4))
        Next

 

[Invicore]

Just now, Jos said:

You say this is actually working? This part doesn't test for a hex code of 4 characters but in stead for a literal string, so doubt that if is ever true.

 

Yes Its fully working, but very slow

[Invicore]

59 minutes ago, Nine said:

And why loop in first example is 

For $Table = 0 To 400

While in second example is 

For $Table = 0 To 4

And why do you assign repeatedly to the same variables 

For $Exports = 1 To $itemCount
            $stringIndex_ClassName = _BinaryToInt16(_BinaryRead(2))
            $ObjectFlags = _BinaryToInt16(_BinaryRead(2))
            $parentID = _BinaryToInt32(_BinaryRead(4))
            $dataSize = _BinaryToInt32(_BinaryRead(4))
            $dataOffset = _BinaryToInt32(_BinaryRead(4))
            $template = _BinaryToInt32(_BinaryRead(4))
            $CRC32 = _BinaryToInt32(_BinaryRead(4))
        Next

 

as I said its not complete code its a snip from two of my codes, and about assign repeatedly, its becuse I only want to skip these bytes for now, in the future all var save in array

Edited March 19, 2021 by Invicore

[Invicore]

_BinaryRead take like 4ms for read bytes (4 byte)

FileRead Start very fast but as the program progress and go further of file it will be slow down

[Nine]

But the loop is useless now.  And you complain about speed !

Really, you need to clarify what you want to achieve and provide a clear runable snippet of the code (without useless loops) that we can actually test.  As Jocko asked, provide a file example that we can use.  And upload the binary udf so we do not have to search for it.

[Jos]

22 minutes ago, Invicore said:

Yes Its fully working, but very slow

Don't think the shown code works! Simply POC:

Wrong test ....sorry.... your code is correct!

So, what about you stop with all this vague stuff and start explaining what you are really trying to do here with all this binary file hacking?

Jos

Edited March 19, 2021 by Jos

[Jos]

Updated previous post as the POC was wrong ...my mistake!

[Invicore]

So let me show my full code

I think it better way

it the one that load all file and use BinaryMid (its so slow, even slower then FileRead)

expandcollapse popup

#include <File.au3>
#include <Binary.au3>

;POS
Global $GNOW = 1
;=>End

Dim $NEWdata, $k = 0
If $CmdLine[0] <> 0 Then
    $TxtPath = $CmdLine[1]
Else
    $TxtPath = FileOpenDialog("Select the txt file", @ScriptDir, "txt files (*.txt)", 1)
EndIf
If @error = 1 Then Exit
_FileReadToArray($TxtPath, $NEWdata)
$Name = StringTrimRight(CompGetFileName($TxtPath), 4)
$File = FileOpen($Name, 0 + 16)
$File_R = FileRead($File, -1)
If $File = -1 Then
    MsgBox(0, "Error", "Can't open " & $Name & " file.")
    Exit
EndIf
$Magic = _BinaryRead(4)
If $Magic <> "0x43523257" Then ;Magic
    MsgBox(0, "Error", $Name & " is not a valid CR2W File.")
    Exit
EndIf
$NewFile = $Magic
$NewFile &= _BinaryRead(36)
$NewFile &= _BinaryRead(120)
_BinarySetPos(41)
;FileSetPos($File, 40, 0)

Local $StringOffset = 0
Local $SkipSize = 0
For $CR2WTable = 0 To 4
    $Offset = _BinaryToInt32(_BinaryRead(4))
    $itemCount = _BinaryToInt32(_BinaryRead(4))
    $CRC32 = _BinaryToInt32(_BinaryRead(4))
    If $CR2WTable == 0 Then ;Strings
        $StringOffset = $Offset
        $SkipSize += $itemCount
    ElseIf $CR2WTable == 1 Then ;CNames
        $BeforeCName = _BinaryGetPos()
        $SkipSize += ($itemCount * 8)
        _BinarySetPos($Offset+1)
        Global $Cname[$itemCount]
        For $CNames = 0 To $itemCount - 1
            $SOffset = _BinaryToInt32(_BinaryRead(4))
            $SHash = _BinaryToInt32(_BinaryRead(4))
            $pos = _BinaryGetPos()
            _BinarySetPos($StringOffset + $SOffset + 1)
            $S = ""
            $Str = ""
            Do
                $Str &= $S
                $S = _BinaryRead(1)
            Until $S = 0
            $Str = BinaryToString("0x" & StringRegExpReplace($Str, "0x", ""), 4)
            $Cname[$CNames] = $Str
            ;ConsoleWrite($Str & @CRLF)
            _BinarySetPos($pos)
        Next
        _BinarySetPos($BeforeCName)
    ElseIf $CR2WTable == 3 Then ;CR2WProperty
        $SkipSize += ($itemCount * 16)
    ElseIf $CR2WTable == 4 Then ;CR2WExports
        _BinarySetPos($Offset+1)
        $SkipSize += ($itemCount * 24)
        For $CR2WExports = 1 To $itemCount
            $stringIndex_ClassName = _BinaryToInt16(_BinaryRead(2))
            $ObjectFlags = _BinaryToInt16(_BinaryRead(2))
            $parentID = _BinaryToInt32(_BinaryRead(4))
            $dataSize = _BinaryToInt32(_BinaryRead(4))
            $dataOffset = _BinaryToInt32(_BinaryRead(4))
            $template = _BinaryToInt32(_BinaryRead(4))
            $CRC32 = _BinaryToInt32(_BinaryRead(4))
        Next
    EndIf
Next

_BinarySetPos($StringOffset+1)    ;
$NewFile &= _BinaryRead($SkipSize + 1) ;Skip header for now
$NewFile &= _BinaryRead(25)        ;Get Json Resource
$NewFile &= _BinaryRead(4)         ;Get Type Array
$SizeOfStringSec = FileGetPos($File)
$NewTextFile = _BinaryRead(4)      ;TODO: Change size
$CharCount = _BinaryRead(4)
$NewTextFile &= $CharCount
$CharCount = _Binarytoint32($CharCount)
$NewTextFile &= _BinaryRead(1)

ProgressOn("Importing", "Working, this will take so long so take a sit :|", "", 300, 300, 2 + 16)

$k = 1
Local $Global = 0
Local $hTimer = TimerInit()
For $i = 1 To $CharCount ;$Cname
    While $Global = 0
        $Index = _BinaryRead(2)
        $NewTextFile &= $Index
        If $Index = 0x000A Then
            $NewTextFile &= _BinaryRead(2) ;DataType
            $Unit32Size = _BinaryRead(4)
            $NewTextFile &= $Unit32Size
            $Unit32Size = _BinaryToInt32($Unit32Size)
            $NewTextFile &= _BinaryRead($Unit32Size - 4)
        Else
            If $Cname[$Index] <> "secondaryKey" Then
                $NewTextFile &= _BinaryRead(2) ;$DataType
                $NEWdata[$k] = StringReplace($NEWdata[$k], "<cf>", @CRLF)
                $NEWdata[$k] = StringReplace($NEWdata[$k], "<lf>", @LF)
                $NEWdata[$k] = StringReplace($NEWdata[$k], "<cr>", @CR)
                $NewText = StringToBinary(StringTrimLeft($NEWdata[$k], 2), 4)
                $truesize = BinaryLen($NewText) - 1
                If $truesize > 64 Then
                    $CreatedLen = _SizeG($truesize)
                    If $CreatedLen[0] Then
                        _XAdd_Update($truesize, $CreatedLen[1], $CreatedLen[2])
                    Else
                        $CreatedLen = _XSizeG($truesize)
                        _XAdd_Update($truesize, $CreatedLen[1], $CreatedLen[2])
                    EndIf
                Else
                    $CreatedLen = _SizeG($truesize)
                    _Add_Update($truesize, $CreatedLen)
                EndIf

                $NewTextFile &= $NewText
                ;ConsoleWrite($NEWdata[$k] & @CRLF & $NewTextFile & @CRLF)
                $k += 1

            Else ; Keys copy from file without change
                $NewTextFile &= _BinaryRead(2) ;$DataType
                $NewTextFile &= _BinaryRead(4) ;$Unit32Size
                $CStrlen = _BinaryRead(1)
                $NewTextFile &= $CStrlen           ;$CStrlen
                $CStrlen = _BinaryToInt8($CStrlen)
                If $CStrlen >= 192 Then
                    $ExtraStrlen = _BinaryRead(1)
                    $NewTextFile &= $ExtraStrlen
                    $ExtraStrlen = _BinaryToInt16($ExtraStrlen)
                    $ActualLen = ($CStrlen - 128) + 64 * ($ExtraStrlen - 1)
                Else
                    $ActualLen = $CStrlen - 128
                EndIf
                $NewTextFile &= _BinaryRead($ActualLen) ; String key
            EndIf

            $thereIsmore = _BinaryGetPos()
            $CheckStart = _BinaryRead(4)
            If $CheckStart = 0x0A000000 Then
                _BinarySetPos($thereIsmore)
                $NewTextFile &= _BinaryRead(3)
                $Global = 1
            ElseIf $CheckStart = 0x00000000 Then
                _BinarySetPos($thereIsmore)
                $NewTextFile &= _BinaryRead(4)
                $Global = 1
            Else
                _BinarySetPos($thereIsmore)
            EndIf
        EndIf
    WEnd
    $Global = 0
    ProgressSet(Int($i / ($CharCount / 100)))
    ;ConsoleWrite($i & @CRLF)
Next
$LenOfStrings = BinaryLen($NewTextFile)
$NewFile &= $NewTextFile
$FileSize = BinaryLen($NewFile)
$Newfile = _BinaryPoke($Newfile,$SizeOfStringSec+1,$LenOfStrings+2,"dword")
$Newfile = _BinaryPoke($Newfile,25,$FileSize,"dword")
$Newfile = _BinaryPoke($Newfile,29,$FileSize,"dword")
$hNewfile = FileOpen("NEW_" & $Name, 2 + 16)
FileWrite($hNewfile, $NewFile)
FileClose($hNewfile)
Local $fDiff = TimerDiff($hTimer)
ConsoleWrite($fDiff & @CRLF)

Func _SizeG($truesizef)
    Local $SizeGen[3]
    If $truesizef > 64 Then
        $extralen = Int($truesizef / 64)
        $strlen = 193 + Mod($truesizef, 64)
        If $strlen == 0 Then
            $strlen = 192
            $extralen += 1
        EndIf
        If $strlen >= 256 Then
            $strlen = 192
            $extralen += 1
        EndIf
        Local $isOk = 0
        Local $TestActualLen = ($strlen - 128) + 64 * ($extralen - 1)
        If $TestActualLen = $truesizef+1 Then
            $isOk = 1
        EndIf
        Local $SizeGen[3] = [$isOk, $strlen, $extralen]
        Return $SizeGen
    Else
        Return 129 + $truesizef
    EndIf
EndFunc

Func _XSizeG($truesizef)
    $Answer = 0
    Local $strlen = 192
    Local $extralen = 1
    While $Answer = 0 ; In this rare situation use brute force
        $ActualLen = ($strlen - 128) + 64 * ($extralen - 1)
        If $ActualLen = $truesize + 1 Then ;Get the answer?
            ConsoleWrite("CStrlen= " & $strlen & @CRLF & "ExtraStrlen= " & $extralen & @CRLF)
            Local $SizeGen[3] = [1, $strlen, $extralen]
            Return $SizeGen
            $Answer = 1
        EndIf
        If $extralen = 255 Then
            If $strlen = 255 Then
                $strlen = 255
                If $extralen = 255 Then ;Program reach limit
                    ConsoleWrite("Nothing" & @CRLF & "CStrlen= " & $strlen & @CRLF & "ExtraStrlen= " & $extralen & @CRLF)
                EndIf
            Else
                $strlen += 1
            EndIf
            $extralen = 1
        Else
            $extralen += 1
        EndIf
    WEnd
EndFunc

Func _Add_Update($truesize, $strlen)
    $NewTextFile &= _BinaryFromInt32($truesize + 1 + 1 + 4) ;unit32Size
    $NewTextFile &= _BinaryFromInt8($strlen)
    _BinaryRead(4)
    $ORCStrlen = _BinaryToInt8(_BinaryRead(1))
    If $ORCStrlen >= 192 Then
        $ORExtraStrlen = _BinaryToInt8(_BinaryRead(1))
        $ORActualLen = ($ORCStrlen - 128) + 64 * ($ORExtraStrlen - 1)
    Else
        $ORActualLen = $ORCStrlen - 128
    EndIf
    _BinaryRead($ORActualLen)
EndFunc

Func _XAdd_Update($truesize, $strlen, $extralen)
    $NewTextFile &= _BinaryFromInt32($truesize + 1 + 2 + 4) ;unit32Size
    $NewTextFile &= _BinaryFromInt8($strlen) & _BinaryFromInt8($extralen)
    _BinaryRead(4)
    $ORCStrlen = _BinaryToInt8(_BinaryRead(1))
    If $ORCStrlen >= 192 Then
        $ORExtraStrlen = _BinaryToInt8(_BinaryRead(1))
        $ORActualLen = ($ORCStrlen - 128) + 64 * ($ORExtraStrlen - 1)
    Else
        $ORActualLen = $ORCStrlen - 128
    EndIf
    _BinaryRead($ORActualLen)
EndFunc

Func _BinaryRead($iCount)
  $GNOW += $iCount
  Return BinaryMid($File_R, $GNOW - $iCount, $iCount)
EndFunc

Func _BinarySetPos($iPos = 0)
  $GNOW = $iPos
EndFunc

Func _BinaryGetPos()
    Return $GNOW
EndFunc

Func _RoundDown($nVar, $iCount)
    Return Round((Int($nVar * (10 ^ $iCount))) / (10 ^ $iCount), $iCount)
EndFunc   ;==>_RoundDown

Func CompGetFileName($Path)
    If StringLen($Path) < 4 Then Return -1
    $ret = StringSplit($Path, "\", 2)
    If IsArray($ret) Then
        Return $ret[UBound($ret) - 1]
    EndIf
    If @error Then Return -1
EndFunc   ;==>CompGetFileName

Here is file example that script work with 

Google drive

[Jos]

Already corrected my statement (wrong example), but how about you explain what it is we are trying to get working (faster) here as it all look pretty strange to me?

Edited March 19, 2021 by Jos

[Invicore]

Just now, Jos said:

Already corrected my wrong example, but how about you explain what it is we are trying to get working (faster) here as it all look pretty strange to me?

as i said in first post, I want to read from a binary file as fast as possible, becuse the time its take to read from a binary file and data make my script so slow

FileRead Start Fast but it will slow down

read file to a var and then use BinaryMid for read it slow from begin

so I wonder what can I do

[Jos]

mmm yes that much I already understood , but currently fail to understand why at this moment.
So I am interested in the goals/result of this script! 

[Invicore]

1 minute ago, Jos said:

mmm yes that much I already understood , but currently fail to understand why at this moment.
So I am interested in the goals/result of this script! 

I had create two script

1. for extract string from a file

2. Import exported strings back

first script work perfectly without any problem

second one also work (its still need work), but slow as hell (imagine it take like 30 min for import all strings)

I think its because the way I read binary file (I test it with TimerInit())

[Jos]

Again ... why? ( not what!) 
I understand that English in not your own language, so hope you understand the question, but I like to understand why you are processing this binary information.

[Invicore]

2 minutes ago, Jos said:

Again ... why? ( not what!) 
I understand that English in not your own language, so hope you understand the question, but I like to understand why you are processing this binary information.

Oh sorry, I get it now

I need to read binary file to get some information from it

script need those information to create new file with edited strings

In other word I get data from original file, and use them to create new file

-

btw why I cant edit my posts?

[Jos]

Just now, Invicore said:

btw why I cant edit my posts?

Soon ... when you are in another group, which I will put you in after this message.
 

 

1 minute ago, Invicore said:

Oh sorry, I get it now

Not really, only partially. It is still very vague. So what about you make all files available to run this script so we can test/understand/see if it can be made more efficient?
The script is now posted but there is still a binary.au3 include in there that is not standard!
We also would need an input file to be able to test and maybe some guidelines how to run it?

Jos

[Invicore]

5 minutes ago, Jos said:

Soon ... when you are in another group, which I will put you in after this message.
 

 

Not really, only partially. It is still very vague. So what about you make all files available to run this script so we can test/understand/see if it can be made more efficient?
The script is now posted but there is still a binary.au3 include in there that is not standard!
We also would need an input file to be able to test and maybe some guidelines how to run it?

Jos

I uploaded file example here

35 minutes ago, Invicore said:

Here is file example that script work with 

Google drive

and here Binary.au3

Binary.rar

Edited March 19, 2021 by Invicore