Read File to Byte Array - very very slow

[queensoft]

Goal (excuse the non-technical terms, I will post picture and script to help ) :

- read a file, binary format (you can use a JPG for testing) - FileOpen - $FO_BINARY (16) = Force binary mode

- split the file in hex numbers, each represented by 2 characters

- store everything in a simple 1D array

Picture:

- left side = Total Commander (either Compare files operation or View > Hex)

- right side = _ArrayDisplay..... it is identical, so the output is perfect

Problem:

- FileOpen & FileRead & BinaryLen = all work great

- now, splitting the file into hex numbers: I have tried many solutions, finally settled on DllStructCreate & DllStructSetData  - BUT, this is very slow, at least 6-7 seconds for a 2 Mb file (i7 7700K @ 4.5 GHz, 16 Gb RAM)

#include <File.au3>
#include <Array.au3>
#include <AutoItConstants.au3>

$s = TimerInit()

Local $hFileOpen = FileOpen('Image1.jpg', 16) ; 16+0=Read binary
Local $sFileRead = FileRead($hFileOpen)
FileClose($hFileOpen)

Local $File_struct = DllStructCreate("BYTE File[" & BinaryLen($sFileRead) & "]")
DllStructSetData($File_struct, "File", $sFileRead)

Local $iSize = BinaryLen($sFileRead)
Local $aByte[$iSize]
For $i = 0 To $iSize - 1
    if Mod($i+1, 10000) = 0 Then ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $i = ' & $i & ' of ' & $iSize - 1 & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console
    $aByte[$i] = hex(DllStructGetData($File_struct, "File",$i),2)
Next

ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : = ' & TimerDiff($s)/1000 & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console

_ArrayDisplay($aByte, '', "500")    ; show only the first 500 lines

How do I do the splitting faster??

I need to read [tens of] thousands of files.

Thank you.

[ioa747]

maybe it has to do with the file?

with  this file https://www.autoitscript.com/forum/uploads/profile/photo-30661.jpg

console out: @@ Debug(23) : = 0.0092673

 

Edit:
with  this file  https://upload.wikimedia.org/wikipedia/commons/3/3a/Cat03.jpg

console out: @@ Debug(23) : = 0.8744486

Spoiler

expandcollapse popup

@@ Debug(19) : $i = 9999 of 279602
>Error code: 0
@@ Debug(19) : $i = 19999 of 279602
>Error code: 0
@@ Debug(19) : $i = 29999 of 279602
>Error code: 0
@@ Debug(19) : $i = 39999 of 279602
>Error code: 0
@@ Debug(19) : $i = 49999 of 279602
>Error code: 0
@@ Debug(19) : $i = 59999 of 279602
>Error code: 0
@@ Debug(19) : $i = 69999 of 279602
>Error code: 0
@@ Debug(19) : $i = 79999 of 279602
>Error code: 0
@@ Debug(19) : $i = 89999 of 279602
>Error code: 0
@@ Debug(19) : $i = 99999 of 279602
>Error code: 0
@@ Debug(19) : $i = 109999 of 279602
>Error code: 0
@@ Debug(19) : $i = 119999 of 279602
>Error code: 0
@@ Debug(19) : $i = 129999 of 279602
>Error code: 0
@@ Debug(19) : $i = 139999 of 279602
>Error code: 0
@@ Debug(19) : $i = 149999 of 279602
>Error code: 0
@@ Debug(19) : $i = 159999 of 279602
>Error code: 0
@@ Debug(19) : $i = 169999 of 279602
>Error code: 0
@@ Debug(19) : $i = 179999 of 279602
>Error code: 0
@@ Debug(19) : $i = 189999 of 279602
>Error code: 0
@@ Debug(19) : $i = 199999 of 279602
>Error code: 0
@@ Debug(19) : $i = 209999 of 279602
>Error code: 0
@@ Debug(19) : $i = 219999 of 279602
>Error code: 0
@@ Debug(19) : $i = 229999 of 279602
>Error code: 0
@@ Debug(19) : $i = 239999 of 279602
>Error code: 0
@@ Debug(19) : $i = 249999 of 279602
>Error code: 0
@@ Debug(19) : $i = 259999 of 279602
>Error code: 0
@@ Debug(19) : $i = 269999 of 279602
>Error code: 0
@@ Debug(23) : = 0.8744486

 

 

Edited March 2 by ioa747

[Danyfirex]

You could use RegExp. 

StringRegExp($sFileRead,".{2}",3)

 

But for displaying(_ArrayDisplay) it would take a bunch of time for large array. (Probably a fast display  or virtual UDF would help.)

 

 

Saludos

[UEZ]

This should be faster:

;Coded by UEZ build 2024-03-02
#include <Array.au3>
#include <WinAPIFiles.au3>
#include <WinAPIHObj.au3>

Global $sFile = "Captured.bmp"
ConsoleWrite("File size = " & FileGetSize($sFile) & @CRLF)
Global $t = TimerInit()
Global $a1D = _WinAPI_File2Hex1DArray($sFile)
ConsoleWrite(TimerDiff($t) & " ms" & @CRLF)
;~ _ArrayDisplay($a1D)

Func _WinAPI_File2Hex1DArray($sFile)
    Local $iSize = FileGetSize($sFile)
    If @error Or Not $iSize Then Return SetError(1, 0, 0)
    Local $tBuffer = DllStructCreate("byte data[" & $iSize & "]"), $nBytes
    Local $hFile = _WinAPI_CreateFile($sFile, 2, 2)
    _WinAPI_ReadFile($hFile, $tBuffer, $iSize, $nBytes)
    _WinAPI_CloseHandle($hFile)
    Local $aResult = StringRegExp(Binary($tBuffer.data), ".{2}", 3, 3)
    $tBuffer = Null
    Return $aResult
EndFunc

 

Edited March 2 by UEZ

[SmOke_N]

I like UEZs approach.  Here's the same approach with AutoIt default funcs (speed is very similar though).

#include <Array.au3>
#include <FileConstants.au3>
#include <StringConstants.au3>

Global $sFile = "Captured.bmp"
ConsoleWrite("File size = " & FileGetSize($sFile) & @CRLF)
Global $t = TimerInit()
Global $a1D = _Aut_File2Hex1DArray($sFile)
ConsoleWrite(TimerDiff($t) & " ms" & @CRLF)
ConsoleWrite("BYTES: " & UBound($a1D) & " : 1ST BYTE: " & $a1D[0] & @CRLF)
;~ _ArrayDisplay($a1D)

Func _Aut_File2Hex1DArray($sFile)
    If Not FileExists($sFile) Then Return SetError(1, 0, 0)
    Local $hFile = FileOpen($sFile, $FO_BINARY)
    Local $nBytes = FileRead($hFile)
    FileClose($hFile)
    Return StringRegExp($nBytes, ".{2}", $STR_REGEXPARRAYGLOBALMATCH, 3)
EndFunc

 

Edited March 2 by SmOke_N
Forgot to offset the "0x" prefix

[UEZ]

Here another way which seems to be litte faster then my previous version above:

;Coded by UEZ build 2024-03-02
#include <Array.au3>
#include <WinAPIFiles.au3>
#include <WinAPIHObj.au3>

Const $CRYPT_STRING_HEX = 4, $CRYPT_STRING_NOCRLF = 0x40000000
Global $sFile = "LCD_Test.png"
ConsoleWrite("File size = " & FileGetSize($sFile) & @CRLF)
Global $t = TimerInit()
Global $a1D = _WinAPI_File2Hex1DArray2($sFile)
ConsoleWrite(TimerDiff($t) & " ms" & @CRLF)
;_ArrayDisplay($a1D)

Func _WinAPI_File2Hex1DArray2($sFile)
    Local $iSize = FileGetSize($sFile)
    If @error Or Not $iSize Then Return SetError(1, 0, 0)
    Local $tBuffer = DllStructCreate("byte data[" & $iSize & "]"), $nBytes
    Local $hFile = _WinAPI_CreateFile($sFile, 2, 2)
    _WinAPI_ReadFile($hFile, $tBuffer, $iSize, $nBytes)
    _WinAPI_CloseHandle($hFile)
    Local $tOut = DllStructCreate("char hex[" & DllCall("Crypt32.dll", "bool", "CryptBinaryToStringA", "struct*", $tBuffer, "dword", DllStructGetSize($tBuffer), "dword", $CRYPT_STRING_HEX + $CRYPT_STRING_NOCRLF, "struct*", Null, "dword*", Null)[5] & "]")
    DllCall("Crypt32.dll", "bool", "CryptBinaryToStringA", "struct*", $tBuffer, "dword", DllStructGetSize($tBuffer), "dword", $CRYPT_STRING_HEX + $CRYPT_STRING_NOCRLF, "struct*", $tOut, "dword*", DllStructGetSize($tOut))
    Local $aResult = StringSplit(BinaryToString($tOut.hex), " ", $STR_NOCOUNT)
    $tBuffer = Null
    Return $aResult
EndFunc

 

[SmOke_N]

@UEZ, very nice, that's like 10+% faster than the other versions.

[jchd]

5 hours ago, queensoft said:

- store everything in a simple 1D array

Remember that AutoIt arrays can't store more than 2^24 = 16777216 elements.

Now what's the real purpose of all this, instead of viewing the binary file using any hex editor?

[UEZ]

Additionally, displaying an array for 1mb+ files takes very long which makes no sense...

[mLipok]

You should read only part of file not entire at once.
Display only part then goes to next part when user choose to look at the next part.

[SmOke_N]

1 hour ago, jchd said:

Remember that AutoIt arrays can't store more than 2^24 = 16777216 elements.

Now what's the real purpose of all this, instead of viewing the binary file using any hex editor?

What is that, like a 15mb file? 😲

[queensoft]

15 hours ago, ioa747 said:

maybe it has to do with the file?

with  this file https://www.autoitscript.com/forum/uploads/profile/photo-30661.jpg

console out: @@ Debug(23) : = 0.0092673

 

Edit:
with  this file  https://upload.wikimedia.org/wikipedia/commons/3/3a/Cat03.jpg

console out: @@ Debug(23) : = 0.8744486

  Reveal hidden contents

expandcollapse popup

@@ Debug(19) : $i = 9999 of 279602
>Error code: 0
@@ Debug(19) : $i = 19999 of 279602
>Error code: 0
@@ Debug(19) : $i = 29999 of 279602
>Error code: 0
@@ Debug(19) : $i = 39999 of 279602
>Error code: 0
@@ Debug(19) : $i = 49999 of 279602
>Error code: 0
@@ Debug(19) : $i = 59999 of 279602
>Error code: 0
@@ Debug(19) : $i = 69999 of 279602
>Error code: 0
@@ Debug(19) : $i = 79999 of 279602
>Error code: 0
@@ Debug(19) : $i = 89999 of 279602
>Error code: 0
@@ Debug(19) : $i = 99999 of 279602
>Error code: 0
@@ Debug(19) : $i = 109999 of 279602
>Error code: 0
@@ Debug(19) : $i = 119999 of 279602
>Error code: 0
@@ Debug(19) : $i = 129999 of 279602
>Error code: 0
@@ Debug(19) : $i = 139999 of 279602
>Error code: 0
@@ Debug(19) : $i = 149999 of 279602
>Error code: 0
@@ Debug(19) : $i = 159999 of 279602
>Error code: 0
@@ Debug(19) : $i = 169999 of 279602
>Error code: 0
@@ Debug(19) : $i = 179999 of 279602
>Error code: 0
@@ Debug(19) : $i = 189999 of 279602
>Error code: 0
@@ Debug(19) : $i = 199999 of 279602
>Error code: 0
@@ Debug(19) : $i = 209999 of 279602
>Error code: 0
@@ Debug(19) : $i = 219999 of 279602
>Error code: 0
@@ Debug(19) : $i = 229999 of 279602
>Error code: 0
@@ Debug(19) : $i = 239999 of 279602
>Error code: 0
@@ Debug(19) : $i = 249999 of 279602
>Error code: 0
@@ Debug(19) : $i = 259999 of 279602
>Error code: 0
@@ Debug(19) : $i = 269999 of 279602
>Error code: 0
@@ Debug(23) : = 0.8744486

 

 

Those files are very small. My files are at least 2 Mb.

I tested with EXE, JPG and MP3, same slow processing.

I have to process thousands of files, the difference from 5-7 seconds to.... let's hope 2-3 seconds, is huge.

Edited March 3 by queensoft

[queensoft]

14 hours ago, Danyfirex said:

You could use RegExp. 

StringRegExp($sFileRead,".{2}",3)

 

But for displaying(_ArrayDisplay) it would take a bunch of time for large array. (Probably a fast display  or virtual UDF would help.)

 

 

Saludos

Already tried that - at least 2 different RegExp - even slower. No need for _ArrayDisplay, I only check the first 100-500 results.

[queensoft]

13 hours ago, UEZ said:

This should be faster:

;Coded by UEZ build 2024-03-02
#include <Array.au3>
#include <WinAPIFiles.au3>
#include <WinAPIHObj.au3>

Global $sFile = "Captured.bmp"
ConsoleWrite("File size = " & FileGetSize($sFile) & @CRLF)
Global $t = TimerInit()
Global $a1D = _WinAPI_File2Hex1DArray($sFile)
ConsoleWrite(TimerDiff($t) & " ms" & @CRLF)
;~ _ArrayDisplay($a1D)

Func _WinAPI_File2Hex1DArray($sFile)
    Local $iSize = FileGetSize($sFile)
    If @error Or Not $iSize Then Return SetError(1, 0, 0)
    Local $tBuffer = DllStructCreate("byte data[" & $iSize & "]"), $nBytes
    Local $hFile = _WinAPI_CreateFile($sFile, 2, 2)
    _WinAPI_ReadFile($hFile, $tBuffer, $iSize, $nBytes)
    _WinAPI_CloseHandle($hFile)
    Local $aResult = StringRegExp(Binary($tBuffer.data), ".{2}", 3, 3)
    $tBuffer = Null
    Return $aResult
EndFunc

 

Much much much better, slightly under 1 second!!!!

I was going to ask "how come this StringRegExp is so fast?" but then I remembered I absolutely do not understand StringRegExp, so I'm not gonna ask that.

But I did I test one of my previous attempts and now I get similar results, around 1.1 - 1.2 seconds:

StringRegExp($aData[1], ".{2}", 3)

It uses: CryptBinaryToString function (Windows)
    ; http://msdn.microsoft.com/en-us/library/windows/desktop/aa379887(v=vs.85).aspx

I think the _ArrayDisplay was throwing me off track, I later realised that is taking a huge time also.

[queensoft]

12 hours ago, SmOke_N said:

I like UEZs approach.  Here's the same approach with AutoIt default funcs (speed is very similar though).

#include <Array.au3>
#include <FileConstants.au3>
#include <StringConstants.au3>

Global $sFile = "Captured.bmp"
ConsoleWrite("File size = " & FileGetSize($sFile) & @CRLF)
Global $t = TimerInit()
Global $a1D = _Aut_File2Hex1DArray($sFile)
ConsoleWrite(TimerDiff($t) & " ms" & @CRLF)
ConsoleWrite("BYTES: " & UBound($a1D) & " : 1ST BYTE: " & $a1D[0] & @CRLF)
;~ _ArrayDisplay($a1D)

Func _Aut_File2Hex1DArray($sFile)
    If Not FileExists($sFile) Then Return SetError(1, 0, 0)
    Local $hFile = FileOpen($sFile, $FO_BINARY)
    Local $nBytes = FileRead($hFile)
    FileClose($hFile)
    Return StringRegExp($nBytes, ".{2}", $STR_REGEXPARRAYGLOBALMATCH, 3)
EndFunc

 

Indeed, similar results. Maybe faster because of no DLL calls?!

[queensoft]

10 hours ago, UEZ said:

Here another way which seems to be litte faster then my previous version above:

;Coded by UEZ build 2024-03-02
#include <Array.au3>
#include <WinAPIFiles.au3>
#include <WinAPIHObj.au3>

Const $CRYPT_STRING_HEX = 4, $CRYPT_STRING_NOCRLF = 0x40000000
Global $sFile = "LCD_Test.png"
ConsoleWrite("File size = " & FileGetSize($sFile) & @CRLF)
Global $t = TimerInit()
Global $a1D = _WinAPI_File2Hex1DArray2($sFile)
ConsoleWrite(TimerDiff($t) & " ms" & @CRLF)
;_ArrayDisplay($a1D)

Func _WinAPI_File2Hex1DArray2($sFile)
    Local $iSize = FileGetSize($sFile)
    If @error Or Not $iSize Then Return SetError(1, 0, 0)
    Local $tBuffer = DllStructCreate("byte data[" & $iSize & "]"), $nBytes
    Local $hFile = _WinAPI_CreateFile($sFile, 2, 2)
    _WinAPI_ReadFile($hFile, $tBuffer, $iSize, $nBytes)
    _WinAPI_CloseHandle($hFile)
    Local $tOut = DllStructCreate("char hex[" & DllCall("Crypt32.dll", "bool", "CryptBinaryToStringA", "struct*", $tBuffer, "dword", DllStructGetSize($tBuffer), "dword", $CRYPT_STRING_HEX + $CRYPT_STRING_NOCRLF, "struct*", Null, "dword*", Null)[5] & "]")
    DllCall("Crypt32.dll", "bool", "CryptBinaryToStringA", "struct*", $tBuffer, "dword", DllStructGetSize($tBuffer), "dword", $CRYPT_STRING_HEX + $CRYPT_STRING_NOCRLF, "struct*", $tOut, "dword*", DllStructGetSize($tOut))
    Local $aResult = StringSplit(BinaryToString($tOut.hex), " ", $STR_NOCOUNT)
    $tBuffer = Null
    Return $aResult
EndFunc

 

Holy Cow!

Indeed faster, about 15-25% faster, down to 0.75 seconds.

[queensoft]

10 hours ago, jchd said:

Remember that AutoIt arrays can't store more than 2^24 = 16777216 elements.

Now what's the real purpose of all this, instead of viewing the binary file using any hex editor?

I'm well under the 16 Mb file size.

The purpose is to compare / extract / save the differences between 2 .bin files, rinse, repeat with thousands of files.

[queensoft]

10 hours ago, UEZ said:

Additionally, displaying an array for 1mb+ files takes very long which makes no sense...

The _ArrayDisplay is just for checking the results.

[queensoft]

9 hours ago, mLipok said:

You should read only part of file not entire at once.
Display only part then goes to next part when user choose to look at the next part.

I need the entire file (files - 2, see reply above) at once.

[jchd]

1 hour ago, queensoft said:

No need for _ArrayDisplay, I only check the first 100-500 results.

1 hour ago, queensoft said:

I need the entire file (files - 2, see reply above) at once.

All this is nonsensical.

Tell us what's the REAL purpose. No, you aren't going to visually compare hex byte to hex byte [tens of] thousands 2Mb files.

Tell us what you're really after and maybe we can help.