Duplicate Files Finder - MD5 Hash CheckSum

[ptrex]

Duplicate Files Finder - MD5 Hash CheckSum

Someone in the support forum was pointing to the Xstandard MD5 Hash COM object.

For those of you who don't know what a MD5 Hash CheckSum is :

In cryptography, MD5 (Message-Digest algorithm 5) is a widely used cryptographic hash function with a 128-bit hash value. As an Internet standard (RFC 1321), MD5 has been employed in a wide variety of security applications, and is also commonly used to check the integrity of files. An MD5 hash is typically expressed as a 32-character hexadecimal number.

MD5 was designed by Ron Rivest in 1991 to replace an earlier hash function, MD4. In 1996, a flaw was found with the design of MD5[citation needed]; while it was not a clearly fatal weakness, cryptographers began recommending the use of other algorithms, such as SHA-1 (which has since been found vulnerable itself). In 2004, more serious flaws were discovered making further use of the algorithm for security purposes questionable.

From Wikipedia, the free encyclopedia

Well this would be an example on how to use the CheckSum to find duplicated files or Scripts on your system :

Download here XStandard - MD5 Com Object

This small application is my first using the new syntax of 3.2.9.10.

It contains most of the extensive functionality that AU3 has to offer :

- DLLCalls

- SQL

- In Memory Database

- COM object

- RegisterMsg

- Native AU3 functions

- ...

DuplicateFiles_MD5_SQL_New.au3

Au3 - 3.2.13.13 version and higher

DuplicateFiles_MD5_SQL_New.au3

The speed of the application is more than reasonable.

It even scans your server if needed.

Enjoy finding duplicate scripts, audio, zip, exe, ... files

Enjoy !!

regards,

ptrex

Edited September 14, 2012 by ptrex

[speedy6]

I get an error on _GUICtrlListViewSetColumnWidth...

How com?

[ptrex]

@speedy6

This function seems to be removed in the newer versions of AU3.

You can just comment out these lines.

I will see of there is function that replaces this.

This is the one according to the new naming conventions.

_GUICtrlListView_SetColumnWidth

Thanks

regards

ptrex

Edited December 3, 2007 by ptrex

[speedy6]

And what do i have to do with a COM error?

(i get it when i start the program)

[ptrex]

@speedy6

If you tell me the error I don't have to guess ?!

regards

ptrex

[speedy6]

err.windrescription: Ongeldige klassereeks

Err.number is: 800401F3

err.scriptline is: 10

err.lastdllerror is: 0

[ptrex]

@speedy6

This means that you have not registered the MD5 Object on your system.

Regards / Groeten,

ptrex

[ptrex]

@all

Added a new update in the first post.

Enjoy fighting the digital pollution !!

Regards,

ptrex

[ptrex]

@all

Corrected a bug in the export function :

- Corrected the CSV delimiter.

- Optimized export query.

New version posted in first post.

[gseller]

ptrex... I just found this, Very NICE!! I downloaded the XMD5.dll and added these lines to your IsObj($oMD5) to autosetup the dll.. Thanks so much for sharing..

If Not IsObj($oMD5) Then
FileInstall(@ScriptDir&"\x-md5\XMD5.dll", @SystemDir & "\XMD5.dll", 0)
RunWait(@ComSpec & " /c regsvr32 /s XMD5.dll", @SystemDir, @SW_HIDE)
MsgBox(48,"Error","You don't have the MD5 COM object installed on !! We are installing it now.")
 ;Exit
EndIf

[ptrex]

@gesller

Thanks for the compliment and additions to the script.

In the meantime I have complete a version, that has no dependency on an EXTERNAL COM DLL.

It has all the hash functionaluty included in AU3.

Runs really as fast, if not faster than the previous version.

I run it to scan my networks servers (remotely). And I am amazed to see how many GB on duplicate data is around.

But unfortunately it is not open source.

Regards,

ptrex

[gseller]

ptrex...

Ahhh, Bummer... Oh well, if you ever decide you want to share we would love to see the code...

I still like this one too..

[Xenobiologist]

Hi,

I get this:

C:\Downloads\AutoIt-Skripte\Entwicklung\ForumTests\MD5Duplicates.au3 (244) : ==> Error in expression.:

$oDict1_key = $oMD5.GetCheckSumFromFile($aFilesList[$n])

$oDict1_key = ^ ERROR

You also have to check your includes. My script says you needn't Array.au3 and String.au3.

So long,

Mega

[ptrex]

@Xenobiologist

I don't know what kind of errors you get ?

On my machine Running:(3.2.9.14) (and others on which I tested it) it runs like a charm.

2. conditions

- You should use the version Running:(3.2.9.10) and higher.

- You should have the MD5 Object registered on your machine to run the Application.

Regards

ptrex

[Polyphem]

Looks really nice, but sadly I', getting a

Fatal Error

AVector []: Out of Bounds

on this one using 3.2.10.0. Has anyone an idea?

Cheers

[James]

Just a guess here but... I guess ptrex here likes SQL

Very nice! I can't wait to use it at home!

[ptrex]

@Polyphem

You need to have the SQLite DLL avaiable on the machine that runs the code.

Best way is to have the DLL somewhere in the Windows root folder.

This will solve your problem.

@JamesB

I guess ptrex here likes SQL

Indead !!

Since almost anything is accessible through SQL these day.

And it won't be long before the OS will support native SQL.

So you better get used to it.

regards,

ptrex

[Polyphem]

You need to have the SQLite DLL avaiable on the machine that runs the code.

Hmmmm, thinking about all the SQLite function calls in the script... I just can't explain why I have though about that ... guess I assumed that SQLite is already part of AutoIt's pre-installed UDF's .... thanks for your quick response ...

Cheers

Poly

[Polyphem]

Hmmmm, sorry to bother again .... installed md5 and sqllite... now the program works fine for me... except that there seems to be a problem with the passing of the values from one listview to the other (see screen attached)... I'll try to dig into the code myself...

Cheers

UPDATE:

Ups, just took me 20 seconds .... replace all occurrences of

_GUICtrlListView_DeleteAllItems ($ListView1)

with

_GUICtrlListView_DeleteAllItems (GUICtrlGetHandle($ListView1))

and

_GUICtrlListView_DeleteAllItems ($ListView2)

with

_GUICtrlListView_DeleteAllItems (GUICtrlGetHandle($ListView2))

and

_GUICtrlListView_DeleteItem($ListView2,$Index)

with

_GUICtrlListView_DeleteItem(GUICtrlGetHandle($ListView2),$Index)

Thats it

Btw, did I already say thank you for this great piece of code? Thank you ...

Cheers

Edited April 28, 2008 by Polyphem

[ptrex]

@Polyphem

You are welcome.

All depends of course on the version of AU3 you use.

Regards

ptrex