Read File to Byte Array - very very slow

[queensoft]

That IS the real purpose.

Read 2 files, same size, very similar content.

Find differences between files, extract those differences (from file 2, position and value).

Put those differences in another file.

Just like Total Commander > Compare files, but instead of showing the differences, extract & save them.

[jchd]

Then why not use fc?

Quote

> C:\Users\jc\Documents\tmp> fc.exe /b img1.bmp img2.bmp
Comparaison des fichiers img1.bmp et IMG2.BMP
0009AB99: F0 C7
0009AB9A: EF CB
0009AB9B: 81 34
0009C795: 23 00
0009C796: 7C 80
0009C797: F5 FF
> C:\Users\jc\Documents\tmp>

It's then trivial to redirect ouput to a file.

Quote

> C:\Users\jc\Documents\tmp> fc.exe /b img1.bmp img2.bmp > imgdif.txt

Super fast, free, available since W95.

imgdif.txt img1.bmp img2.bmp

[queensoft]

I will surely try that, thanks.

I'm working right now on the compare part, this is also tricky / slow.....

[Gianni]

here a minimalist version... gets the same result as fc.exe but much much slower...

Local $sFile1 = "img1.bmp" 
Local $sFile2 = "img2.bmp"

Local $hFile = FileOpen($sFile1, 16)   ; 16 = binary mode
Local $sBinData = FileRead($hFile)
Local $iBytes1 = @extended
FileClose($hFile)
ConsoleWrite($sFile1 & ' = ' & $iBytes1 & @CRLF)

Local $tBytes1 = DllStructCreate("byte[" & $iBytes1 & "]")
DllStructSetData(($tBytes1), 1, $sBinData)

$hFile = FileOpen($sFile2, 16)
$sBinData = FileRead($hFile)
Local $iBytes2 = @extended
FileClose($hFile)
ConsoleWrite($sFile2 & ' = ' & $iBytes2 & @CRLF)

Local $tBytes2 = DllStructCreate("byte[" & $iBytes2 & "]")
DllStructSetData(($tBytes2), 1, $sBinData)

; If $iBytes1 <> $iBytes2 Then Exit ; 2 files have <> len

For $i = 1 To $iBytes1
    $BYTE1 = DllStructGetData($tBytes1, 1, $i)
    $BYTE2 = DllStructGetData($tBytes2, 1, $i)

    If $BYTE1 <> $BYTE2 Then
        ConsoleWrite(Hex($i - 1) & ": " & Hex($BYTE1, 2) & ' ' & Hex($BYTE2, 2) & @CRLF)
    EndIf
Next

 

[jchd]

Why not, but isn't it reinventing the wheel? Well, unless the OP has untold extra requirements.

[Gianni]

.. ? .. I don't get StdoutRead() .. 

P.S.

Script  Fixed,  (thanks to @nine ... see post below this)

Local $sFile1 = "img1.bmp"
Local $sFile2 = "img2.bmp"

; Local $sCmd = @ComSpec & " /c " & 'fc.exe /b ' & $sFile1 & ' ' & $sFile2
Local $sCmd = 'fc.exe /b ' & $sFile1 & ' ' & $sFile2

Local $sOut
Local $h = Run($sCmd, '.', '', 0x1 + 0x8) ;  $STDIN_CHILD (0x1) + $STDERR_MERGED (0x8)

Do
    ConsoleWrite('.') ; just for debug
    $sOut &= StdoutRead($h)
Until Not ProcessExists($h)

ConsoleWrite($sOut & @CRLF)

 

Edited March 3 by Gianni
Fixed script (thanks to @nine ... see post below this)

[Nine]

@Gianni

#include <Constants.au3>

Local $sFile1 = "img1.bmp"
Local $sFile2 = "img2.bmp"

Local $sCmd = @ComSpec & " /C fc /b " & $sFile1 & ' ' & $sFile2

Local $sOut
Local $iPID = Run($sCmd, "", @SW_HIDE, $STDIN_CHILD + $STDERR_MERGED)
ProcessWaitClose($iPID)
$sOut = StdoutRead($iPID)

ConsoleWrite($sOut & @CRLF)

 

Edited March 3 by Nine

[Gianni]

thanks @Nine  👍

.... you're right, it's better not to forget $STDIN_CHILD... 

[queensoft]

54 minutes ago, Nine said:

@Gianni

#include <Constants.au3>

Local $sFile1 = "img1.bmp"
Local $sFile2 = "img2.bmp"

Local $sCmd = @ComSpec & " /C fc /b " & $sFile1 & ' ' & $sFile2

Local $sOut
Local $iPID = Run($sCmd, "", @SW_HIDE, $STDIN_CHILD + $STDERR_MERGED)
ProcessWaitClose($iPID)
$sOut = StdoutRead($iPID)

ConsoleWrite($sOut & @CRLF)

 

When I'm stoopid, I'm stoopid!!!

I was still reading the 2 files into arrays before using fc.exe!!!! I didn't realized fc.exe is working with the original files directly.

But I will still need to read the files, because I have to process & save them.

In the end, I think a simple For loop - direct compare between the 2 arrays (files) - is fast enough for my needs. And it eliminates the need to capture & parse fc.exe output.

Edited March 3 by queensoft

[jchd]

7 hours ago, jchd said:

Well, unless the OP has untold extra requirements

4 hours ago, queensoft said:

But I will still need to read the files, because I have to process & save them.

No surprise here!

[queensoft]

It's coming along just fine, no worries!

[queensoft]

In the end, I used UEZ function + jchd FC.exe - everything works great

[UEZ]

Did you test?

It is based on your requirements. 

Edited March 9 by UEZ

[queensoft]

Dude, you da man!!!

Maybe I'll use it on version 2.

Right now, program is working just fine as is.

[Nine]

For the fun of it :

expandcollapse popup

Local $tFile1 = ReadBinaryFile('img1.bmp')
Local $tFile2 = ReadBinaryFile('img2.bmp')

Local $hInit = TimerInit()
Local $tResult = CompareData($tFile1, $tFile2), $nResult = @extended, $pResult = DllStructGetPtr($tResult)
ConsoleWrite("Time:     " & TimerDiff($hInit) & " ms" & @CRLF)

For $i = 1 To $nResult
  $tDiff = DllStructCreate("align 1;byte;byte;dword;", $pResult)
  ConsoleWrite(Hex(DllStructGetData($tDiff, 1), 2) & @TAB & Hex(DllStructGetData($tDiff, 2), 2) & @TAB & _
      Hex(DllStructGetData($tDiff, 3), 8) & @CRLF)
  $pResult += 6
Next

Func CompareData(Const ByRef $tFile1, Const ByRef $tFile2)
  If DllStructGetSize($tFile1) <> DllStructGetSize($tFile2) Then Return SetError(1)
  Local $tResult = DllStructCreate("byte data[1000]") ; make it large enough to receive results
  Local $sCode = '0x8B7424048B7C24088B4C240C8B54241031DB8A068A2738E0740B8802886201895A0283C206434647E2E889D0C21000'
  Local $dCode = Binary($sCode)
  Local $iCodeSize = BinaryLen($dCode)
  Local $tBuffer = DllStructCreate('byte Code[' & $iCodeSize & ']')
  DllStructSetData($tBuffer, 'Code', $dCode)
  Local $aCall = DllCallAddress('int', DllStructGetPtr($tBuffer), 'ptr', DllStructGetPtr($tFile1), 'ptr', DllStructGetPtr($tFile2), _
      'int', DllStructGetSize($tFile1), 'ptr', DllStructGetPtr($tResult))
  Local $iResult = ($aCall[0] - DllStructGetPtr($tResult)) / 6
  Return SetExtended($iResult, $tResult)
EndFunc   ;==>CompareData

Func ReadBinaryFile($sPath)
  Local $iSize = FileGetSize($sPath)
  Local $tFile = DllStructCreate("byte Data[" & $iSize & "]")
  Local $hFile = FileOpen($sPath, 16)       ; FO_READ + FO_BINARY
  $tFile.Data = FileRead($hFile)
  FileClose($hFile)
  Return $tFile
EndFunc   ;==>ReadBinaryFile

#cs

  mov    esi,DWORD PTR [esp+0x04]
  mov    edi,DWORD PTR [esp+0x08]
  mov    ecx,DWORD PTR [esp+0x0c]
  mov    edx,DWORD PTR [esp+0x10]
  xor    ebx,ebx
  l1:
  mov    al,BYTE PTR [esi]
  mov    ah,BYTE PTR [edi]
  cmp    al,ah
  je     l2
  mov    BYTE PTR [edx],al
  mov    BYTE PTR [edx+1],ah
  mov    DWORD PTR [edx+2],ebx
  add    edx,6
  l2:
  inc    ebx
  inc    esi
  inc    edi
  loop   l1
  mov    eax,edx
  ret    16

#ce

2 ms...

Edited March 9 by Nine

[queensoft]

There's a small problem:

Running:(3.3.16.1):C:\Program Files (x86)\AutoIt3\autoit3.exe "D:\..........................\_testF15.au3"    
+>Setting Hotkeys...--> Press Ctrl+Alt+Break to Restart or Ctrl+BREAK to Stop.
!>16:53:46 AutoIt3.exe ended.rc:-1073741819
+>16:53:46 AutoIt3Wrapper Finished.
>Exit code: 3221225477    Time: 2.101

Files are .bin, 2 Mb each (please do not share): https://drive.google.com/file/d/1ZquEnq_l-avs60WodgqSr9q2pdafrTeu/view?usp=sharing

Testing with another pair of .bine files, 4 Mb each = same error.

The .bin files are OK, I have already processed them using my script and there were checked by client.

Testing with .jpg and .au3 (180, 250 Kb) / .exe & .dll (0.9 Mb each) = OK.

Testing with 2 .exe files (2.0 & 2.3 Mb) = OK !!!!

Also, how do I get the result? I know it's in the $nResult / $pResult variables, but I need to research DllStructGetData, right?

Hmmm, nevermind, I have found the code to display the results, but it does not display anything.

Tested with various random files, it seems to work, it shows Time: ......., but no results.

[Nine]

Tested your 2 bin files.  Like I mentioned in the code you need to provide a large enough buffer to receive the results.  Increase it to 20000 and it will work.

As for the others, maybe it is the same problem.  Or the files are not exactly equal in size...

Edited March 9 by Nine

[queensoft]

Holy cow, this is fast !!!!!!! Working great now !!

[Nine]

Well since you like it, here a new version with buffer size check :

expandcollapse popup

Local $tFile1 = ReadBinaryFile('file1.bin')
Local $tFile2 = ReadBinaryFile('file2.bin')

Local $hInit = TimerInit()
Local $tResult = CompareData($tFile1, $tFile2, 10000)
If @error Then Exit MsgBox($MB_OK, "Error", @error = 1 ? "Size not equal" : "Increase buffer size")
Local $nResult = @extended, $pResult = DllStructGetPtr($tResult)
ConsoleWrite("Time:     " & TimerDiff($hInit) & " ms" & @CRLF)

For $i = 1 To $nResult
  $tDiff = DllStructCreate("align 1;byte;byte;dword;", $pResult)
  ConsoleWrite(Hex(DllStructGetData($tDiff, 1), 2) & @TAB & Hex(DllStructGetData($tDiff, 2), 2) & @TAB & _
      Hex(DllStructGetData($tDiff, 3), 8) & @CRLF)
  $pResult += 6
Next

Func CompareData(Const ByRef $tFile1, Const ByRef $tFile2, $iSize)
  If DllStructGetSize($tFile1) <> DllStructGetSize($tFile2) Then Return SetError(1)
  Local $tResult = DllStructCreate("byte data[" & $iSize & "]") ; make it large enough to receive results
  Local $sCode = '0x8B7424048B7C24088B4C240C8B54241031DB8A068A2738E07412836C24140678158802886201895A0283C206434647E2E189D0C2140031C083E801C21400'
  Local $dCode = Binary($sCode)
  Local $iCodeSize = BinaryLen($dCode)
  Local $tBuffer = DllStructCreate('byte Code[' & $iCodeSize & ']')
  DllStructSetData($tBuffer, 'Code', $dCode)
  Local $aCall = DllCallAddress('int', DllStructGetPtr($tBuffer), 'ptr', DllStructGetPtr($tFile1), 'ptr', DllStructGetPtr($tFile2), _
      'int', DllStructGetSize($tFile1), 'ptr', DllStructGetPtr($tResult), 'int', $iSize)
  If $aCall[0] = -1 Then Return SetError(2)
  Local $iResult = ($aCall[0] - DllStructGetPtr($tResult)) / 6
  Return SetExtended($iResult, $tResult)
EndFunc   ;==>CompareData

Func ReadBinaryFile($sPath)
  Local $iSize = FileGetSize($sPath)
  Local $tFile = DllStructCreate("byte Data[" & $iSize & "]")
  Local $hFile = FileOpen($sPath, 16)       ; FO_READ + FO_BINARY
  $tFile.Data = FileRead($hFile)
  FileClose($hFile)
  Return $tFile
EndFunc   ;==>ReadBinaryFile

#cs

  mov    esi,DWORD PTR [esp+0x04]
  mov    edi,DWORD PTR [esp+0x08]
  mov    ecx,DWORD PTR [esp+0x0c]
  mov    edx,DWORD PTR [esp+0x10]
  xor    ebx,ebx
  l1:
  mov    al,BYTE PTR [esi]
  mov    ah,BYTE PTR [edi]
  cmp    al,ah
  je     l2
  sub    DWORD PTR [esp+0x14],6
  js     l3
  mov    BYTE PTR [edx],al
  mov    BYTE PTR [edx+1],ah
  mov    DWORD PTR [edx+2],ebx
  add    edx,6
  l2:
  inc    ebx
  inc    esi
  inc    edi
  loop   l1
  mov    eax,edx
  ret    20
  l3:
  xor    eax,eax
  sub    eax,1
  ret    20

#ce

 

Edited March 10 by Nine