Check-If-Same

[TheSaint]

Check-If-Same is a little program, with a big heart (window), that I have been working on since the end of August. It's a somewhat simple affair, and one of the many file checkers out there, but I have put my own slant on it. The screenshot probably says it all.

It was built to purpose for me, and you can see the MD5 stuff as an added extra. It has been suggested by my good buddy TheDcoder, that I should also add CRC32 checking, which I may do at some point. However, I just wanted a simple folder content comparison, where I was more interested in file names being matched, not the state of the files ... they having had their integrity checked already by TeraCopy.

Files only appear on either list, if they are missing in the other folder or they fail the chosen compare test. If desired, missing files can be skipped (File Name option excepted).

'Source Size' and 'Compare Size' can alternatively be 'Source Date' and 'Compare Date', as specified by the Bytes combo field.

Anyway, for what its worth, enjoy! Some of you might find it handy. Source is included in the zip.

Check-If-Same v1.1.zip  589.41 kB  (111 downloads)

Check-If-Same v1.2.zip  NEW

You have two methods of MD5 checking, one is the AutoIt Crypt hash checking, the other is using third party FSUM. In my limited number of tests, I found the crypt method was faster with small files and FSUM much faster with big files, so take your pick. If you are keen to see timing comparisons, then with the individual MD5 button option, the times are stored in the 'Settings.ini' file for the entry you selected (see the [MD5 Test] section).

Edited September 11, 2023 by TheSaint

[TheDcoder]

Looks good! A potential improvement is to add support for CRC32 (same algorithm used by TeraCopy), it is much faster than MD5

[TheSaint]

6 minutes ago, TheDcoder said:

A potential improvement is to add support for CRC32

Gawd, that means shortening the Progress Bar yet again ... already done that a few times. 

Or I could reduce the size of the font on some of the buttons as well. 

[TheSaint]

Ok, I gave into CRC32 pressure. Enjoy the addition.

See first post for the v1.2 update download.

Here's an alternate screenshot with the changes ... doing an individual check with the newly (optional) named CRC32 button.
As you can see the Progress Bar area also gets re-purposed. Dates are also shown instead of sizes.

I did some tests ... of course, and yes it seems that CRC32 is a bit faster. I got the following results with a (not shown) 3.7 Gb ISO file.

Quote

[MD5 Test]
crypt=406497.18362818
fsum-crc32=277323.445926767
fsum-md5=280878.85853384
 

Note that each result is for two files (source & compare), so divide by 2 to get how long it takes for one 3.7 Gb ISO file on my low powered PC.

Using the AutoIt crypt hashing, the MD5 process clearly takes much longer.
The CRC32 process with FSUM is marginally faster than its MD5 process.

Of course, repeated tests with same file would show variations on all three I imagine.

Edited September 5, 2020 by TheSaint

[TheDcoder]

47 minutes ago, TheSaint said:

The CRC32 process with FSUM is marginally faster than its MD5 process.

It should be a lot faster for larger files... so I am a bit surprised, maybe the CPU isn't the bottleneck here . Out of curiosity I did my own frugal test:

TheDcoder@arch /tmp> time rhash --md5 bigfile
cd573cfaace07e7949bc0c46028904ff  bigfile

________________________________________________________
Executed in    1.74 secs   fish           external 
   usr time  1679.79 millis  198.00 micros  1679.59 millis 
   sys time   62.06 millis   78.00 micros   61.98 millis 

TheDcoder@arch /tmp> time rhash --crc32 bigfile
; Generated by RHash v1.4.0 on 2020-09-05 at 17:54.26
; Written by Kravchenko Aleksey (Akademgorodok) - http://rhash.sf.net/
;
;   1073741824  17:51.23 2020-09-05 bigfile
bigfile 5B64C2B0

________________________________________________________
Executed in  724.48 millis    fish           external 
   usr time  666.49 millis  271.00 micros  666.22 millis 
   sys time   58.13 millis  107.00 micros   58.02 millis 

TheDcoder@arch /tmp> stat bigfile
  File: bigfile
  Size: 1073741824  Blocks: 0          IO Block: 4096   regular file
Device: 30h/48d Inode: 42640       Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/TheDcoder)   Gid: ( 1000/TheDcoder)
Access: 2020-09-05 17:54:27.682908480 +0530
Modify: 2020-09-05 17:51:23.904011423 +0530
Change: 2020-09-05 17:51:23.904011423 +0530
 Birth: -

CRC32 was ~2.4x faster than MD5 when tested with a 1 GB file, the file was read from RAM so the disk overhead should not exist.

[TheSaint]

5 hours ago, TheDcoder said:

CRC32 was ~2.4x faster than MD5 when tested with a 1 GB file, the file was read from RAM so the disk overhead should not exist.

Well I am not sure what you are saying here or how it helps?

To compare a file on disk to one in RAM seems pointless to me, because in the real world they are on disk. So why even do a test with that abnormal condition? What are you trying to prove that is meaningful?

Did you do an on disk test as well, to see if your results were similar to mine?

I suspect the issue may relate to RAM, and as you know the PC I am using for this only has 2 Gb RAM, whereas yours has at least 8 Gb if I recall correctly.

For further consideration, file size seems to play a part. So maybe a 1 Gb file is a sweet spot for CRC32 checking. As I said earlier, the Crypt hash method was faster with small files, quite a bit faster in fact, probably as much faster as it is slower with big files. Not that I have done exhaustive tests.

It could be interesting to know how speed for each method gradually varies as size increases.

If one were really keen, one could do a file size test first, then use the appropriate hash checking for optimum speed. I suspect though, that with quite small files, the time you could be saving probably mostly gets lost doing a size check. Still, if you were doing enough files and there was a great range of sizes, you could make up for any lost time.

P.S. It is important to note, that my earlier tests above, were with two small unpowered portable external drives. So that would have slowed things considerably too, and have a USB 2.0 bottleneck.

Edited September 5, 2020 by TheSaint

[TheDcoder]

53 minutes ago, TheSaint said:

To compare a file on disk to one in RAM seems pointless to me, because in the real world they are on disk. So why even do a test with that abnormal condition? What are you trying to prove that is meaningful?

I was trying to compare the speed of the algorithms by isolating other outside influences as much as possible, thus the abnormal conditions.

54 minutes ago, TheSaint said:

Did you do an on disk test as well, to see if your results were similar to mine?

Not in this case, but I have used CRC32 when moving large files between disks previously and it was faster than any other hashing algo.

56 minutes ago, TheSaint said:

For further consideration, file size seems to play a part. So maybe a 1 Gb file is a sweet spot for CRC32 checking.

56 minutes ago, TheSaint said:

It could be interesting to know how speed for each method gradually varies as size increases.

I am no crypto expert, but I think the results would be fairly linear, i.e time will be proportional to the size.

57 minutes ago, TheSaint said:

As I said earlier, the Crypt hash method was faster with small files, quite a bit faster in fact

I know why, it is because the WinAPI crypt functions are already loaded by your script and they don't have the overhead of calling a 3rd-party executable, which a whole-another thing, thus increasing the overhead, this will significantly impact your performance for small files, but for large files the overhead wouldn't be noticeable.

59 minutes ago, TheSaint said:

P.S. It is important to note, that my earlier tests above, were with two small unpowered portable external drives. So that would have slowed things considerably too, and have a USB 2.0 bottleneck.

Ah, that makes sense, that was the bottleneck .

[TheSaint]

1 hour ago, TheDcoder said:

I was trying to compare the speed of the algorithms by isolating other outside influences as much as possible, thus the abnormal conditions.

I know that is what you were doing, but it has to make sense with real world conditions, otherwise it is meaningless

1 hour ago, TheDcoder said:

I am no crypto expert, but I think the results would be fairly linear, i.e time will be proportional to the size.

.Neither am I, but I imagine there can be some time related factor possibilities ... perhaps like the hare and the tortoise. 

1 hour ago, TheDcoder said:

Not in this case, but I have used CRC32 when moving large files between disks previously and it was faster than any other hashing algo.

Well you should have, so we could have some meaningful meat in our sandwich.

1 hour ago, TheDcoder said:

I know why, it is because the WinAPI crypt functions are already loaded by your script and they don't have the overhead of calling a 3rd-party executable

Obviously, but still very important when doing lots of files, especially if many are small. And like I said, could be interesting to know when the benefit in speed is eventually lost due to file size.

[TheDcoder]

13 hours ago, TheSaint said:

but it has to make sense with real world conditions, otherwise it is meaningless

I don't follow, why is it meaningless to compare the speed of the algorithm? In fact, it is the only meaningful way to do any objective benchmark, otherwise you'd be clumping in all sorts of "real-world" factors which are out of the control and skew the results, just like what happened in your case, where MD5 and CRC32 took the same amount of time due to disk bottleneck .

13 hours ago, TheSaint said:

so we could have some meaningful meat in our sandwich.

Meaningless and subjective meat you mean

13 hours ago, TheSaint said:

And like I said, could be interesting to know when the benefit in speed is eventually lost due to file size.

Impossible to do a good objective test, too many factors involved. It would differ greatly even on the same computer, so it is not worth persuing this. The best way is to use a DLL always when possible... but in this case, it looks like WinAPI MD5 Algo is less optimized, so bad luck.

Edited September 6, 2020 by TheDcoder

[TheSaint]

4 hours ago, TheDcoder said:

I don't follow, why is it meaningless to compare the speed of the algorithm? In fact, it is the only meaningful way to do any objective benchmark, otherwise you'd be clumping in all sorts of "real-world" factors which are out of the control and skew the results, just like what happened in your case, where MD5 and CRC32 took the same amount of time due to disk bottleneck .

Yes bud, but if you test in an unreal environment, it has no real value.
What you should do, is make sure the environment you test in is real and has suitable factors ... then you really can get an objective benchmark ... it's never objective if the conditions are unreal.

So in our situation, that would be eliminating external drives, and maybe ensuring plenty of RAM.

4 hours ago, TheDcoder said:

Impossible to do a good objective test, too many factors involved.

I disagree. The intent is to use them in the real world, so only a real world test is meaningful ... objective.

If you reduce the limiting factors as much as feasible, you will get a fairly objective test result.

Otherwise you are just in a fantasy land that never equates to reality, and what's the point of that.

What's the point of knowing CRC32 is faster in an unreal environment, if it cannot be shown to be so in a real environment? All very well for theory, but it needs to translate to real usage.

If I haven't convinced you yet, then I suggest we stop here and move on. Especially as I am clearly not nerdy enough. 

[TheDcoder]

1 hour ago, TheSaint said:

What's the point of knowing CRC32 is faster in an unreal environment, if it cannot be shown to be so in a real environment?

I think you are going about this in a wrong way, especially how you think it is "unreal". It is as real as the bottleneck in your USB

My intention was only to compare the speed of the algorithms, and only that. My test was just a proof that CRC32 is faster than MD5, obviously there are other factors which will limit the speed in a certain situation, and if the algorithm does not get the data as fast as it can process them, then it is of course going to be slower... which is what happened in your case, the speed was limited by the disk transfer rate.

For a "real" test, you can try comparing MD5 and CRC32 for the same file on your internal hard disk, this should eliminate the bottleneck you had in your previous test.

[TheSaint]

6 hours ago, TheDcoder said:

I think you are going about this in a wrong way, especially how you think it is "unreal". It is as real as the bottleneck in your USB

Gawd bud, it is not me going about it the wrong way, it is you. What possible value can your test be. You haven't proved anything in regard to the real world conditions, you have only done a test in RAM, which is not relevant, as it doesn't help or solve anything. You have proved what happens in RAM, and that is all, and seeing as that is not where any real world usage occurs, it is pointless. You are getting lost in a hypothetical, that is only a hypothetical, and while no doubt of interest to nerds or geeks, serves no practical purpose.

It is of no comparison to the real effect of USB speed limitation, which is a real world thing. That is real, because it is where reality meets usage.

As some would say, you need to take your test out of the laboratory and into the real world, where results have actual meaning beyond theoretical, where real life factors play a part, not perfect pristine conditions that just don't occur in the real world. How is your perfect test going to help us in the real world.

'The Real World' is defined by where you actually use the MD5 or CRC32 checking.

And remember, it was you who suggested I should be using CRC32 in the real world instead of MD5, where I will supposedly gain a benefit. A RAM test achieves nothing in that regard, except to make you feel better ... it doesn't help or convince me.

6 hours ago, TheDcoder said:

For a "real" test, you can try comparing MD5 and CRC32 for the same file on your internal hard disk, this should eliminate the bottleneck you had in your previous test.

Haven't I already suggested that in my previous post, along with acknowledging where the likely bottleneck was. That is however a bottleneck that I have to put up with, unless you or I etc come up with an improvement with that real world scenario.

Now I can keep repeating myself and saying things in as many different ways that I can think up, but it is clearly as pointless to do so as your RAM test, so I suggest we give up. I would much prefer we discuss something useful.

I cannot see the value of your test, and you cannot understand the point I make ... let's leave it at that, because clearly, by this point, neither of us are going to be convinced by the other, and I've grown bored and tired of repeating myself, even if slightly couched differently. Let's leave our arguments in HexChat.

[TheSaint]

8 hours ago, TheSaint said:

... it doesn't help or convince me.

@TheDcoder - Not that I need convincing, as I took you at your word, and so incorporated a CRC32 option into my program, without doing a single comparison test first.

So far I haven't made any speed claims about CRC32 versus MD5, just reported on my real world result. Without definitive proof to the contrary, I believe you when you say CRC32 is faster, I accept you have the experience and knowledge to backup that claim.

All that remains really, is how much under real world conditions, is CRC32 faster, and what can we maybe do to improve any conditional limitations.

With that in mind, I decided to try out something, that occurred to me as I was waking today ... a kind of thinking outside the box thought.

I gave my program, what I call a 'Local' option. What it does, is copy 'fsum.exe' to the root of Source and Compare drives, and use it from there, to see if that improves the checking speed. It seemed to, going by the following results, using the exact same 3.7 Gb file as before, with the same exact portable drives.

New Results

Quote

fsum-crc32=268341.99517717
fsum-md5=277841.330732487

Previous Results

Quote

fsum-crc32=277323.445926767
fsum-md5=280878.85853384
 

Not a huge difference, but there appears to be some improvement with my Local method ... but no real change with CRC32 versus MD5.

Now at this point, I don't know what variations might exist naturally anyway, so I am not claiming the Local method actually does improve things.

NOTE - The copying of fsum.exe occurs before the timing begins, as with multiple files on the same drives, that would indeed be the case.

Edited September 7, 2020 by TheSaint

[TheSaint]

Well there you go, another test, with the Local option off, shows that natural variation, and now CRC32 looks worse than MD5. 

Quote

fsum-crc32=291519.075982613
fsum-md5=270283.544319351

The exact same 3.7 Gb file as before, with the same exact portable drives.

I rest my case.

Spoiler

Maybe. 

P.S. There was one difference with my usage. This latest test. the MD5 test occurred first. Same short break between tests though.

Edited September 7, 2020 by TheSaint

[TheDcoder]

9 hours ago, TheSaint said:

Gawd bud, it is not me going about it the wrong way, it is you. What possible value can your test be.

Well bud, I believe my test is valuable because it shows which algorithm is faster, thus simpler. Note that the algorithm does not change how it works, it always functions the same, both in laboratory and in the real world.

1 hour ago, TheSaint said:

Not that I need convincing, as I took you at your word, and so incorporated a CRC32 option into my program, without doing a single comparison test first.

1 hour ago, TheSaint said:

I accept you have the experience and knowledge to backup that claim.

Thanks, I knew you would see the merit in my suggestion. This discussion was started because I was a bit "bamboozled" by your benchmark, however now I understand why.

2 hours ago, TheSaint said:

a kind of thinking outside the box thought.

I gave my program, what I call a 'Local' option.

It is indeed outside the box, good one. But it may not be much of a help... and it might make things worse as well.

1 hour ago, TheSaint said:

Well there you go, another test, with the Local option off, shows that natural variation, and now CRC32 looks worse than MD5. 

Yeah, all thanks to the "warm-up" phenomenon, this happens in almost all benchmarks, the first test is usually performs worse compared to when it is done in the middle.

2 hours ago, TheSaint said:

I rest my case.

Not so fast, here is a real test with my real Seagate 5400RPM SATA disk:

expandcollapse popup

TheDcoder@arch /m/d/D/W/Windows 10> stat -c %b Win10_1909_EnglishInternational_x64.iso
10594256
TheDcoder@arch /m/d/D/W/Windows 10> time rhash --crc32 Win10_1909_EnglishInternational_x64.iso
; Generated by RHash v1.4.0 on 2020-09-07 at 15:32.28
; Written by Kravchenko Aleksey (Akademgorodok) - http://rhash.sf.net/
;
;   5424252928  17:49.07 2020-01-13 Win10_1909_EnglishInternational_x64.iso
Win10_1909_EnglishInternational_x64.iso 6B7B6CB5

________________________________________________________
Executed in   42.95 secs   fish           external 
   usr time    3.42 secs  267.00 micros    3.42 secs 
   sys time    1.49 secs  108.00 micros    1.49 secs 

TheDcoder@arch /m/d/D/W/Windows 10> time rhash --md5 Win10_1909_EnglishInternational_x64.iso
d1f08aea37586702f6fbe2fe3ea8c3fd  Win10_1909_EnglishInternational_x64.iso

________________________________________________________
Executed in   44.20 secs   fish           external 
   usr time    8.33 secs  195.00 micros    8.33 secs 
   sys time    1.73 secs   77.00 micros    1.73 secs 

TheDcoder@arch /m/d/D/W/Windows 10> time rhash --sha1 Win10_1909_EnglishInternational_x64.iso
31549049b73cef72456af0bff71f494e76e1f506  Win10_1909_EnglishInternational_x64.iso

________________________________________________________
Executed in   46.09 secs   fish           external 
   usr time    6.01 secs  377.00 micros    6.01 secs 
   sys time    1.84 secs  152.00 micros    1.84 secs 

TheDcoder@arch /m/d/D/W/Windows 10> time rhash --sha256 Win10_1909_EnglishInternational_x64.iso
b7a4a0786ea9ed51ac7874490885400f9cf3bbc71ab3dad468282da25147a29e  Win10_1909_EnglishInternational_x64.iso

________________________________________________________
Executed in   45.17 secs   fish           external 
   usr time   12.78 secs    0.00 micros   12.78 secs 
   sys time    1.84 secs  464.00 micros    1.84 secs

And it looks like my disk is being the bottleneck, even the SHA1 and SHA256 cryptographic algorithms take the same amount of time as CRC and MD5. Perhaps my processor is too fast

In fact, if you observe closely, SHA1 was one second slower than SHA256... and SHA1 is not even secure anymore. So really, we can't find out the ideal hashing algorithm if we use real world tests like these, because there will be hidden bottlenecks.

In the end, all I want to say is that CRC32 is best because it was designed for the purpose of checking differences in files, as opposed to the other cryptographic algorithms, which  are designed to test the integrity of the file, thus they will be slower.

I rest my case

[TheSaint]

43 minutes ago, TheDcoder said:

Well bud, I believe my test is valuable because it shows which algorithm is faster, thus simpler. Note that the algorithm does not change how it works, it always functions the same, both in laboratory and in the real world.

Well bud, like I have said, I don't see almost any value. Not disputing the algorithm doesn't change, because of course it doesn't. What does change however, is the end result, and the end result is the all important bit. I don't care about the algorithm, unless it is a more reliable guarantee, I just care about the end result. That said, all things being relatively the same, the faster method in real conditions gets my vote.

43 minutes ago, TheDcoder said:

Yeah, all thanks to the "warm-up" phenomenon, this happens in almost all benchmarks, the first test is usually performs worse compared to when it is done in the middle.

Gawd. You misread me. 

In my last round, it was the first test (MD5) that performed the best. Ha ha ha ha.

Previously it had been the CRC32 that got to be first test, and it seems to have fared worse as second.

It's hard to escape the conclusion, that MD5 performed better overall (more reliable timing), even though the CRC32 is better if you let it go first ... apparently .... maybe ... perhaps. 

Do 50 tests will ya, and tell me your conclusions ... anyone. Me, I haven't go that much time to waste. 

P.S. And I like to be kind to my drives, with no pointless testing on them ... or kept to a bare minimum. 

Edited September 7, 2020 by TheSaint

[RTFC]

@TheSaint & @TheDcoder: so how long have you two been married now?

[TheSaint]

50 minutes ago, RTFC said:

@TheSaint & @TheDcoder: so how long have you two been married now?

Forever it seems.

It's also one of those long distance things.

We are still hoping that science will allow him to have my babies ... can you just imagine how cute they would be ... a bit of him a bit of me. And we intend to do that, without ever physically meeting one another. We are destined to be forever apart, especially thanks to COVID ... and it is better that way anyway, as I would have to reject him on the grounds he is male, and I ain't Gay. Currently though, I can pretend he is some exotic Indian Princess ... such is the beauty of our digital connection.

Spoiler

As if. 

P.S. However rosie things might seem, we still cannot get over the fact he doesn't laugh at my jokes, especially the constipated accountant pencil one, and for me, that's a deal-breaker.

Edited September 7, 2020 by TheSaint

[TheDcoder]

7 hours ago, TheSaint said:

(...)  unless it is a more reliable guarantee

Yeah, that's the important bit, and as my tests have shown, when you remove the bottleneck, CRC is the clear winner which is more than twice as fast as MD5. The current differences are minuscule, and perhaps it is just plain bad luck due to several factors (including a potentially bad implementation of CRC in fsum.exe), so you cannot rule out that MD5 is magically better "in the real world"

In the end, CRC is your fastest horse, but in a muddy road the speed doesn't matter, horses will slip and there will be luck involved.

@RTFC Proudly married since the inception of the AutoIt Chatbox

[TheSaint]

20 hours ago, TheDcoder said:

In the end, CRC is your fastest horse, but in a muddy road the speed doesn't matter, horses will slip and there will be luck involved.

Or ..... CRC is a fine pedigree horse that is the fastest around the Race Track, but just can't cut the mustard running anywhere else ... in other words, he (or she) is a real show pony, but ask him to do some real work and he/she runs the other way. 

P.S. Alas, life is full of bottlenecks that we have very little choice about .... poor CRC.

P.S.S. Don't hesitate to suggest some other free alternative to FSUM.

Edited September 8, 2020 by TheSaint