_WinAPI_FileCompareBinary v0.70 build 2024-03-22 beta - reinventing the wheel ^^

UEZ · March 4

As discussed here:

I created a DLL (x86 and x64) to do a file compare like the FC.exe does (only binary comparison!) but filling up the memory which can be read out by Autoit.

UDF _WinAPI_FileCompare.au3:

;Version v0.70 build 2024-03-23 beta
#include-once

; #FUNCTION# ====================================================================================================================
; Name ..........: _WinAPI_FileCompareBinaryString
; Description ...: Reads two files into the memory and compares them byte by byte. Result is saved in the memory.
; Syntax ........: _WinAPI_FileCompareBinaryString($sFile1, $sFile2, Byref $tMemArray)
; Parameters ....: $sFile1              - first file to read.
;                  $sFile2              - second file to read.
;                  $tMemArray           - a ptr dll struct which holds a ptr entry which is filled by the DLL.
; Return values .: Number of unequal bytes found in both files otherwise negative value on error.
; Author ........: UEZ
; Modified ......:
; Remarks .......: The memory pointer $tMemArray must be released using _MemGlobalFree() which holds the pointer to the result.
;                  The result contains hex string values (offset, file1, file2) from the differences between the two files.
;                  Large memory consumption for files > 80 MB as everything is kept in memory!
; Related .......:
; Link ..........: https://www.autoitscript.com/forum/index.php?showtopic=211619
; Example .......: No
; ===============================================================================================================================
Func _WinAPI_FileCompareBinaryString($sFile1, $sFile2, ByRef $tMemArray)
    If Not FileExists($sFile1) Then Return SetError(-10, 0, 0)
    If Not FileExists($sFile2) Then Return SetError(-20, 0, 0)
    If Not FileGetSize($sFile1) Then Return SetError(-30, 0, 0)
    If Not FileGetSize($sFile2) Then Return SetError(-40, 0, 0)
    If Not IsDllStruct($tMemArray) Then Return SetError(-50, 0, 0)
    Local $aReturn = DllCall(@AutoItX64 ? "_WinAPI_FileCompare_x64.dll" : "_WinAPI_FileCompare.dll", "int64", "_WinAPI_FileCompareBinaryString", "str", $sFile1, "str", $sFile2, "struct*", $tMemArray)
    If @error Then Return SetError(-60, 0, 0)
    If $aReturn[0] < 0 Then SetError($aReturn[0], 0, 0)
    Return $aReturn[0]
EndFunc   ;==>_WinAPI_FileCompareBinaryString

; #FUNCTION# ====================================================================================================================
; Name ..........: _WinAPI_FileCompareBinary
; Description ...: Reads two files into the memory and compares them byte by byte. Result is saved in the memory.
; Syntax ........: _WinAPI_FileCompareBinary($sFile1, $sFile2, Byref $tMemArray)
; Parameters ....: $sFile1              - first file to read.
;                  $sFile2              - second file to read.
;                  $tMemArray           - a ptr dll struct which holds a ptr entry which is filled by the DLL.
; Return values .: Number of unequal bytes found in both files otherwise negative value on error.
; Author ........: UEZ
; Modified ......:
; Remarks .......: The memory pointer $tMemArray must be released using _MemGlobalFree() which holds the pointer to the result.
;                  The result contains integer values (offset, file1, file2) from the differences between the two files.
;                  Large memory consumption as everything is kept in memory but 1/3 less than _WinAPI_FileCompareBinaryString.
; Related .......:
; Link ..........: https://www.autoitscript.com/forum/index.php?showtopic=211619
; Example .......: No
; ===============================================================================================================================
Func _WinAPI_FileCompareBinary($sFile1, $sFile2, ByRef $tMemArray)
    If Not FileExists($sFile1) Then Return SetError(-10, 0, 0)
    If Not FileExists($sFile2) Then Return SetError(-20, 0, 0)
    If Not FileGetSize($sFile1) Then Return SetError(-30, 0, 0)
    If Not FileGetSize($sFile2) Then Return SetError(-40, 0, 0)
    If Not IsDllStruct($tMemArray) Then Return SetError(-50, 0, 0)
    Local $aReturn = DllCall(@AutoItX64 ? "_WinAPI_FileCompare_x64.dll" : "_WinAPI_FileCompare.dll", "int64", "_WinAPI_FileCompareBinary", "str", $sFile1, "str", $sFile2, "struct*", $tMemArray)
    If @error Then Return SetError(-60, 0, 0)
    If $aReturn[0] < 0 Then SetError($aReturn[0], 0, 0)
    Return $aReturn[0]
EndFunc   ;==>_WinAPI_FileCompareBinary

; #FUNCTION# ====================================================================================================================
; Name ..........: _WinAPI_FileComparePrint
; Description ...: Reads two files into the memory and compares them byte by byte. Result will be printed to the console only.
; Syntax ........: _WinAPI_FileComparePrint($sFile1, $sFile2[, $bUnbuffered = False])
; Parameters ....: $sFile1              - first file to read.
;                  $sFile2              - second file to read.
;                  $bUnbuffered         - [optional] a boolean value. Default is False. If $bUnbuffered then file will be read
;                                         byte by byte and result will be displayed, otherwise both files will be read into the
;                                         memory and then compared.
; Return values .: 1 if successfull, otherwise 0.
; Author ........: UEZ
; Modified ......:
; Remarks .......: Use #AutoIt3Wrapper_Change2CUI=y when compiling to display result in console.
; Related .......:
; Link ..........: https://www.autoitscript.com/forum/index.php?showtopic=211619
; Example .......: No
; ===============================================================================================================================
Func _WinAPI_FileComparePrint($sFile1, $sFile2, $bUnbuffered = False)
    If Not FileExists($sFile1) Then Return SetError(-10, 0, 0)
    If Not FileExists($sFile2) Then Return SetError(-20, 0, 0)
    If Not FileGetSize($sFile1) Then Return SetError(-30, 0, 0)
    If Not FileGetSize($sFile2) Then Return SetError(-40, 0, 0)
    DllCall(@AutoItX64 ? "_WinAPI_FileCompare_x64.dll" : "_WinAPI_FileCompare.dll", "none", "_WinAPI_FileComparePrint", "str", $sFile1, "str", $sFile2, "bool", $bUnbuffered)
    If @error Then Return SetError(-60, 0, 0)
    Return 1
EndFunc   ;==>_WinAPI_FileComparePrint

Func _WinAPI_FileCompareAbout()
    DllCall(@AutoItX64 ? "_WinAPI_FileCompare_x64.dll" : "_WinAPI_FileCompare.dll", "none", "About")
EndFunc   ;==>_WinAPI_FileCompareAbout

Example1:

;Coded by UEZ build 2024-03-22 beta
#AutoIt3Wrapper_UseX64=y
#include <WinAPIFiles.au3>
#include <Memory.au3>

#include "_WinAPI_FileCompare.au3"

Global $tMemArray = DllStructCreate("ptr addr")
Global $fTimer = TimerInit()
Global $iReturn = _WinAPI_FileCompareBinaryString("img1.bmp", "img2.bmp", $tMemArray)
ConsoleWrite("Dll runtime: " & TimerDiff($fTimer) & " ms" & @CRLF)

If $iReturn Then
    If _WinAPI_IsBadReadPtr($tMemArray.addr, $iReturn) <> 1 Then
        ConsoleWrite(@CRLF)
        ConsoleWrite("Displaying result..." & @CRLF)
        ConsoleWrite("Offset" & @TAB & @TAB & "File1" & @TAB & "File2" & @CRLF)
        Global $i, $j, $t, $c = 0
        For $i = 0 To $iReturn
            For $j = 0 To 2
                $t = DllStructCreate("char string[15]", Ptr($tMemArray.addr + $i * 15 + ($j = 0 ? $j : 6 + $j * 3)))
                ConsoleWrite($t.string & ($j = 0 ? ":" : "") & @TAB)
            Next
            $c += 1
            ConsoleWrite(@CRLF)
        Next
        ConsoleWrite(@CRLF)
        ConsoleWrite("Found " & $c & " differences!" & @CRLF)
        ConsoleWrite(@CRLF)
        If $tMemArray.addr Then _MemGlobalFree($tMemArray.addr)
    Else
        ConsoleWrite("Access violation to memory address" & @CRLF)
    EndIf
Else
    ConsoleWrite("Files are equal!"  & @CRLF)
EndIf

Example2:

;Coded by UEZ build 2024-03-22 beta
#AutoIt3Wrapper_UseX64=y
#include <WinAPIFiles.au3>
#include <Memory.au3>

#include "_WinAPI_FileCompare.au3"

Global $tMemArray = DllStructCreate("ptr addr")
Global $fTimer = TimerInit()
Global $iReturn = _WinAPI_FileCompareBinary("img1.bmp", "img2.bmp", $tMemArray)
ConsoleWrite("Dll runtime: " & TimerDiff($fTimer) & " ms" & @CRLF)

If $iReturn Then
    If _WinAPI_IsBadReadPtr($tMemArray.addr, $iReturn) <> 1 Then
        ConsoleWrite(@CRLF)
        ConsoleWrite("Displaying result..." & @CRLF)
        ConsoleWrite("Offset" & @TAB & @TAB & "File1" & @TAB & "File2" & @CRLF)
        Global $i, $j, $t, $c = 0
        For $i = 0 To $iReturn
            For $j = 0 To 2
                Switch $j
                    Case 0
                        $t = DllStructCreate("ulong offset", Ptr($tMemArray.addr + $i * 8))
                        ConsoleWrite(Hex($t.offset, 8) & ":" & @TAB)
                    Case 1
                        $t = DllStructCreate("ubyte hex1", Ptr($tMemArray.addr + $i * 8 + 4))
                        ConsoleWrite(Hex($t.hex1, 2) & @TAB)
                    Case 2
                        $t = DllStructCreate("ubyte hex2", Ptr($tMemArray.addr + $i * 8 + 5))
                        ConsoleWrite(Hex($t.hex2, 2) & @TAB)
                EndSwitch
            Next
            $c += 1
            ConsoleWrite(@CRLF)
        Next
        ConsoleWrite(@CRLF)
        ConsoleWrite("Found " & $c & " differences!" & @CRLF)
        ConsoleWrite(@CRLF)
        If $tMemArray.addr Then _MemGlobalFree($tMemArray.addr)
    Else
        ConsoleWrite("Access violation to memory address" & @CRLF)
    EndIf
Else
    ConsoleWrite("Files are equal!"  & @CRLF)
EndIf

Example3:

;Coded by UEZ build 2024-03-22 beta
#AutoIt3Wrapper_UseX64=n
#AutoIt3Wrapper_Change2CUI=y

#include "_WinAPI_FileCompare.au3"

_WinAPI_FileComparePrint("img1.bmp", "img2.bmp")

I used the two images from here:

Examples output should look like this here (always hex values):

Dll runtime: 5.6315 ms

Displaying result...
Offset      File1   File2
0009AB99:   F0  C7  
0009AB9A:   EF  CB  
0009AB9B:   81  34  
0009C795:   23  00  
0009C796:   7C  80  
0009C797:   F5  FF  

Found 6 differences!

You can read out the result and put it to an array...

With this version you should be able to use multi-threading / multi-calls of the DLL as the memory is now allocated with each call.

For x86 it may fail when the memory cannot be allocated (2GB limit) and the dll will return 0.

I may add another feature to the DLL to get around the x86 limitation at the expense of speed by using chunked memory allocation if it makes sense...

All files can be found also in the 7-Zip archive.

_WinAPI_FileCompare DLL v0.70 build 2024-03-22.7z

Edited March 23 by UEZ
New version

yuser · March 6

Hi, thanks for the release. I believe one wouldn't need an external dll to compare bytes. The trick to comparing it faster is not letting autoit do it x). Simply loading whatever file you want in memory (not as string) and comparing bytes one by one using asm will be faster:

#include <WinAPI.au3>
#include <Memory.au3>
#include <Array.au3>

$iTimer = TimerInit()
$pBuff1 = ReadFile("big file.dat")
$pBuff2 = ReadFile("similarly big file.dat")
$aCmp = CompareFiles($pBuff1,$pBuff2,0)
$aBindex = CompareByteIndex($pBuff1,$pBuff2,$aCmp)
$iTimer = TimerDiff($iTimer)
_MemVirtualFree(DllStructGetPtr($pBuff1),0,$MEM_RELEASE)
_MemVirtualFree(DllStructGetPtr($pBuff2),0,$MEM_RELEASE)
_ArrayDisplay($aBindex,$iTimer & "ms")
Exit

Func CompareByteIndex($pBuffer1,$pBuffer2,$aCmp)
    Local $aReturn[$aCmp[0]+1][3]
    For $i = 1 To $aCmp[0]
        $iOffset = $aCmp[$i]
        $aReturn[$i][0] = Hex($iOffset,8)
        $aReturn[$i][1] = Hex(DllStructGetData($pBuffer1,1,$iOffset),2)
        $aReturn[$i][2] = Hex(DllStructGetData($pBuffer2,1,$iOffset),2)
    Next
    Return $aReturn
EndFunc

Func ReadFile($sPath)
    $hFile = _WinAPI_CreateFile($sPath,2,2)
    $iLen = _WinAPI_GetFileSizeEx($hFile)
    Local $pAlloc = _MemVirtualAlloc(Null,$iLen,$MEM_COMMIT,$PAGE_READWRITE) ; Creating my own pages to avoid conflict with autoit when i want to free them
    $pBuffer = DllStructCreate("Byte[" & $iLen & "]",$pAlloc)
    $iRead = 0
    _WinAPI_ReadFile($hFile,$pBuffer,$iLen,$iRead)
    _WinAPI_CloseHandle($hFile)
    Return $pBuffer
EndFunc

Func CompareFiles($pBuffer1,$pBuffer2,$iStart,$iEnd=-1)
    $iSizeBuffer1 = DllStructGetSize($pBuffer1)
    $iSizeBuffer2 = DllStructGetSize($pBuffer2)
    $iSizeMin = ($iSizeBuffer1 < $iSizeBuffer2) ? $iSizeBuffer1 : $iSizeBuffer2
    If $iEnd > $iSizeMin OR $iEnd = -1 Then $iEnd = $iSizeMin ; Cases for mismatching sizes
    $pAlloc = _MemVirtualAlloc(Null, 128, $MEM_COMMIT, $PAGE_EXECUTE_READWRITE)
    $pOut = DllStructCreate("uint_ptr[" & 0x4000 & "]") ; Modify if you wish to have more
    If @AutoItX64 Then
        $pFunc = DllStructCreate("Byte[128]",$pAlloc)
        DllStructSetData($pFunc, 1, "0x55488BEC4883EC10C745F001000000C745F400000000498BF848897DF8488BC14801F84C8BD24901FA488B7DF84C39CF0F8332000000418A1238100F841C0000004C8B5D30488B7DF048C1E7034901FB488B7DF848FFC749893BFF45F048FFC049FFC2FF45F8EBC14C8B5DF0488B7D304C891F488BE55DC3")
    Else
        $pFunc = DllStructCreate("Byte[90]",$pAlloc)
        DllStructSetData($pFunc, 1, "0x558BEC83EC0CC745F8010000008B7D10897DFC8B450801F88B4D0C01F98B7DFC3B7D140F83230000008A1138100F84120000008B7DFC478B75F8C1E602037518893EFF45F84041FF45FCEBD18B75F88B7D1889378BE55DC21400")
    EndIf
    DllCallAddress("int",DllStructGetPtr($pFunc),"UINT_PTR",DllStructGetPtr($pBuffer1),"UINT_PTR",DllStructGetPtr($pBuffer2),"int",$iStart,"int",$iEnd,"UINT_PTR",DllStructGetPtr($pOut))
    $iLen = DllStructGetData($pOut,1,1)
    Dim $aReturn[$iLen]
    $aReturn[0] = $iLen-1
    For $i = 1 To $aReturn[0]
        $aReturn[$i] = DllStructGetData($pOut,1,$i+1)
    Next
    Return $aReturn
EndFunc

x64 code is slightly different as i don't usually deal with it

I've tried comparing files of 1.5gb and as long as there's enough memory to load the file this should work. Autoit has large address aware on by default so even on x86 you should be able to handle quite a bit. Feel free to improve it, I have no real use for it so I can't come up with any x)

Cheers

UEZ · March 6

6 hours ago, yuser said:

I believe one wouldn't need an external dll to compare bytes.

Why not? Inline assembler is also possible as you have shown but too complicated and maintaining the code is not easy but much shorter. The DLL is also assembler code but outsourced to a DLL.

As fas as I know x86 code can assign only 2GB of memory.

I will compare large files with your version and my DLL. I assume that the result should not differ much from each other as your asm code doesn't use MMX/SSE. Reading the files to memory is the same way using CreateFileW aka _WinAPI_ReadFile.

Edit: tried to compare ffmpeg.exe (129 mb) with ffplay.exe (128 mb) running it as x86/x64 but it crashes (C0000005 - memory access violation).

Edited March 6 by UEZ

yuser · March 6

3 hours ago, UEZ said:

Why not? Inline assembler is also possible as you have shown but too complicated and maintaining the code is not easy but much shorter. The DLL is also assembler code but outsourced to a DLL.

As fas as I know x86 code can assign only 2GB of memory.

I will compare large files with your version and my DLL. I assume that the result should not differ much from each other as your asm code doesn't use MMX/SSE. Reading the files to memory is the same way using CreateFileW aka _WinAPI_ReadFile.

Edit: tried to compare ffmpeg.exe (129 mb) with ffplay.exe (128 mb) running it as x86/x64 but it crashes (C0000005 - memory access violation).

I recommend reading this link from microsoft, it's informative and I hope it will help you.
https://learn.microsoft.com/en-us/windows/win32/memory/memory-limits-for-windows-releases

#include <Memory.au3>
While 1
    $dwPage = _MemVirtualAlloc(Null,0x8000 * 0x1000,$MEM_COMMIT,0x40)
    ConsoleWrite(Hex($dwPage,8) & @CRLF)
    If $dwPage = 0 Then Exit
WEnd

Try running this, and you'll see that you can use more than 2 gb since LAA is enabled. You can use GlobalMemoryStatusEx to get available bytes.

As for the crash, you ran out of memory for $pOut (you can get this by debugging it or checking a crash dump). If you plan to compare files that are completely different than each other (which ruins the entire purpose for this code) then you're supposed to create enough uint_ptr for every single byte. To handle too many mismatches, the code can be rewritten so it puts even less pressure on autoit (Adding two bytes after uint_ptr as a container for byte comparison etc). Keep in mind that if you handle the output of files with very little similarity in arrays, you might reach max array size on the way. I hope I'm being clear enough

Cheers x)

UEZ · March 6

1 hour ago, yuser said:

As for the crash, you ran out of memory for $pOut (you can get this by debugging it or checking a crash dump). If you plan to compare files that are completely different than each other (which ruins the entire purpose for this code) then you're supposed to create enough uint_ptr for every single byte. To handle too many mismatches, the code can be rewritten so it puts even less pressure on autoit (Adding two bytes after uint_ptr as a container for byte comparison etc). Keep in mind that if you handle the output of files with very little similarity in arrays, you might reach max array size on the way. I hope I'm being clear enough

From an application's point of view, it doesn't matter whether you have to look at some differences or possibly all differences - the application should handle this without crashing, without questioning the meaning.

To avoid the limitation of the array size, you should also use structs, which are of course more difficult to handle.

If I use the x86 version of my DLL and the memory is written over the 2GB limit, the application crashes. With the x64 version there is no such limitation.

Of course, to minimize memory usage, you could read and compare blocks of files instead of storing everything in memory.

For me it was instructive to find out how to read out strings generated by the DLL in Autoit, especially since the comparison is much faster.

yuser · March 6

3 hours ago, UEZ said:

From an application's point of view, it doesn't matter whether you have to look at some differences or possibly all differences - the application should handle this without crashing, without questioning the meaning.

To avoid the limitation of the array size, you should also use structs, which are of course more difficult to handle.

If I use the x86 version of my DLL and the memory is written over the 2GB limit, the application crashes. With the x64 version there is no such limitation.

Of course, to minimize memory usage, you could read and compare blocks of files instead of storing everything in memory.

For me it was instructive to find out how to read out strings generated by the DLL in Autoit, especially since the comparison is much faster.

I provided how you can do it in asm, even clearing the pages created. You simply need to split the file to be compared into parts to accommodate depending on memory available you have. Otherwise on x64 you can't compare files over 2 gb each.

I see that you haven't thoroughly read the link I posted. Simply compile your DLL to handle unsigned pointers properly and it will work above 2gb on x86.

I'm trying to contribute, and my contribution doesn't necessarily have to be code that fits all sizes as I have no intention of spoon feeding anyone who doesn't want to learn anything. If you think this doesn't refer to you, please ignore this post.

UEZ · March 6

On 3/6/2024 at 5:37 PM, yuser said:

I see that you haven't thoroughly read the link I posted. Simply compile your DLL to handle unsigned pointers properly and it will work above 2gb on x86.

Thank you for the link. I haven't had time to read it in detail today, but I will. Maybe I can fix the problem with x86...

By the way, I had also coded parts in ASM to speed up the code instead of using DLLs:

Anyhow, thank you for your feedback and no need to feed me with a spoon.

Edited March 9 by UEZ
forgot "no"

UEZ · March 22

Updated the DLL (old DLL is not compatible with current version) and added UDF version.

Current DLL functions:

_WinAPI_FileCompareBinary
_WinAPI_FileCompareBinaryString
_WinAPI_FileComparePrint

If you are interessted, please checkout first post for more details and download options.

Edited March 22 by UEZ

Andreik · March 22

21 minutes ago, UEZ said:

_WinAPI_FileCompareBinaryString

_WinAPI_FileCompareBinaryString

I think it's a typo here.

UEZ · March 22

C/P issue - thx updated.

Andreik · March 22

What flavor of BASIC is the source?

UEZ · March 22

2 minutes ago, Andreik said:

What flavor of BASIC is the source?

I assume you are asking the source code of the DLL -> Freebasic.

_WinAPI_FileCompareBinary v0.70 build 2024-03-22 beta - reinventing the wheel ^^

Recommended Posts

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members

Similar Content