Duplicate Files Finder - MD5 Hash CheckSum

[aphesia]

could u post me the used UDF s?

i got a error for:

$WM_MENUSELECT

and

$WS_OVERLAPPEDWINDOW + $WS_VISIBLE + $WS_CLIPSIBLINGS)

should have the beta installed.. and also have the udf´s u unclided.. but no included .au3 has this parameters inside

thanks

[KaFu]

Should be in <WindowsConstants.au3> now (AU-internal change).

[aphesia]

now i get a error @line 64:

$PathLabel = GUICtrlCreateLabel("Reading Files : ",10 ,750 ,695,18,$SS_SUNKEN)

variable set without being declared.. so ss_sunken i guess

[KaFu]

Au has undergone some changes since this program has be released.

#include<StaticConstants.au3>

If a const is not declared, always search the AU3 include files first, most of the time it's hidden there somewhere.

[ptrex]

@all

Since many complain about errors when running.

I added an updated version in the first post for AU3 - 3.2.13.13 and higher.

regards

ptrex

[goldenix]

I registered the Dll but it still says Comm error com object not registered

[ptrex]

@goldenix

Check if you registered the COM object correctly.

Open the registry -> HK_Classes_Root -> Find Key "XStandard.MD5".

If you don't find it, regester the COM object again.

Rgds,

ptrex

[Zedna]

Nice script ptrex!

Because I used SQLite recently I have some optimization tips for you:

- use indexes for SQL tables (for columns used in WHERE)

- use BEGIN at start and COMMIT at end (without this each _SQLite_Exec will internally do commit transaction which slows down processing)

- don't calculate $sFileDate,$sDateStamp for small skipped files (skipped due to size filter)

- don't use Sleep (25) in main FOR NEXT loop 

- refresh progressbar and "Time Elapsed" only when it's neccessary --> avoid flicker/high CPU usage

- do INSERT INTO Duplicates in batch --> collect INSERTs into string variable and and call _SQLite_Exec() less times

- don't assign $DataCol = GUICtrlCreateListViewItem() --> it's not used

- use _GUICtrlListView_BeginUpdate / _GUICtrlListView_EndUpdate --> smoother/quicker ListView display

- don't do DROP TABLE at end of script --> it's not needed in case of memory DB

- use _SQLite_GetTable2d() instead of _SQlite_Query + _SQLite_FetchData --> I think it will be faster for big result sets - see next post from Kafu

-  use optimized MD5 hash calculation code --> search forum for examples

http://www.autoitscript.com/forum/index.ph...st&p=687067

http://www.autoitscript.com/forum/index.ph...st&p=557689

See attached optimized version of your script (except MD5 code - it's original untouched, and also _SQLite_GetTable2d not used).

Note I didn't run it so it's not tested yet.

I like that idea (search for duplicates this way) very much so I will rewrite also MD5 hash later probably and I will test it.

DuplicateFiles_MD5_SQL_New_pz.au3

Edited June 7, 2009 by Zedna

[Zedna]

Also another possible speed optimizations:

- skip small files (according to filter for size) already inside _FileListToArrayEx()

- _FileListToArrayEx() --> there are too many Redim statements inside FOR loops --> it should be before loop because Redim is relatively slow operation

Edited June 7, 2009 by Zedna

[KaFu]

- use _SQLite_GetTable2d() instead of _SQlite_Query + _SQLite_FetchData --> I think it will be faster for big result sets

I don't think so, In SMF the usage of _SQLite_GetTable2d() was in fact a large performance issue, and I switched back to use _SQlite_Query + _SQLite_FetchData to create my own array.

[Zedna]

I don't think so, In SMF the usage of _SQLite_GetTable2d() was in fact a large performance issue, and I switched back to use _SQlite_Query + _SQLite_FetchData to create my own array.

Interesting info.

In fact I used only _SQLite_GetTable2d() for small result sets in my project without any speed problems

so my above presumption was based only on idea that one DllCall should be faster than many DllCalls in cycle.

EDIT:

Coincidentally my project was for searching duplicates too.

But it was for searching duplicate objects stored inside PowerBuilder's PBL libraries (163 files with 550 MB size in summary containing 21 000 objects).

Firstly I used my own AutoIt's array to collect/search object names/sizes/dates but this way searching lasted 35 minutes.

After redesigning to use SQLite (memory database with indexes) it lasts only 20 seconds!!

So SQLite is really awesome in some cases.

Edited June 7, 2009 by Zedna

[Zedna]

Other issues of your project:

- I use 1024x768 resolution and your GUI is too big

- After closing About window by [X] button also main window closes

Edited June 7, 2009 by Zedna

[ptrex]

@Zedna,

Good to see that you debugging my old version

Which I left long time ago. And switch over to a build-in MD5 Hash version.

Anyhow some of your suggestions are valid, others are not.

I am interested to see what your new version has to bring.

Regards,

ptrex

[Rose]

It is an interesting message. I have a question. Now I use Clone Remover to find images and music files duplicates What are the differences in these two programmes and how can I use the one from this article? Thank you.

[ptrex]

@Rose

What are the differences in these two programmes

I can"t tell the differences.

Since there is very limited info on how the Clone Remover tool works, on the site.

how can I use the one from this article?

Just read the first topic on how to get started.

You need to download the MD5 COM object from the site mentioned there. And follow the installation instructions,

Then run the au3 script, open a folder, start it, and see what duplicates appear.

regards,

ptrex

[Rose]

I used to delete duplicate files duplicate music finder, which is called Clone Remover. What is your opinion about it?

Edited June 26, 2009 by Rose

[Markhugy2]

I get an error on _GUICtrlListViewSetColumnWidth...

How com?

Here is a good tool to find and delete duplicate files

www.duplicateFilesDeleter.com

[Cposture]

well done

[Kara89r]

Duplicate Files Finder - MD5 Hash CheckSum

Someone in the support forum was pointing to the Xstandard MD5 Hash COM object.

For those of you who don't know what a MD5 Hash CheckSum is :

Well this would be an example on how to use the CheckSum to find duplicated files or Scripts on your system :

Download here XStandard - MD5 Com Object

This small application is my first using the new syntax of 3.2.9.10.

It contains most of the extensive functionality that AU3 has to offer :

- DLLCalls

- SQL

- In Memory Database

- COM object

- RegisterMsg

- Native AU3 functions

- ...

DuplicateFiles_MD5_SQL_New.au3

Free Duplicate File Finder

Au3 - 3.2.13.13 version and higher

DuplicateFiles_MD5_SQL_New.au3

The speed of the application is more than reasonable.

It even scans your server if needed.

Enjoy finding duplicate scripts, audio, zip, exe, ... files

Enjoy !!

regards,

ptrex

@Polyphem

You need to have the SQLite DLL avaiable on the machine that runs the code.

Best way is to have the DLL somewhere in the Windows root folder.

This will solve your problem.

@JamesB

Indead !!

Since almost anything is accessible through SQL these day.

And it won't be long before the OS will support native SQL.

So you better get used to it.

regards,

ptrex

__________________________________________________

Thanks For This Useful Info....

[peterson]

For a duplicate files finder, Duplicate Files Deleter is one of the good software. It will give you a comprehensive list of all those files and you can decide for yourself what you want to do with them. It's easy to use. Hope it will help you.