Check-If-Same

[TheDcoder]

1 hour ago, TheSaint said:

CRC is a fine pedigree horse that is the fastest around the Race Track, but just can't cut the mustard running anywhere else ... in other words, he (or she) is a real show pony, but ask him to do some real work and he/she runs the other way.

Horse $#!T

I don't understand horses, and it was just a stupid analogy that I came up in the heat of the moment

In reality both CRC and MD5 are mathematical functions, number goes in and number comes out, there wouldn't be much variation because the input isn't parsed... it is just processed with the same algorithm over and over.

1 hour ago, TheSaint said:

P.S.S. Don't hesitate to suggest some other free alternative to FSUM.

Think I already did way back in some other thread, and also used it in my tests... it is called rhash and it is cross-platform. I recall FSUM had some shortcomings that I pointed out in the other thread, can't recall them right now, but FSUM definitely isn't the perfect pony you might have had in your mind

Edited September 8, 2020 by TheDcoder

[TheSaint]

20 hours ago, TheDcoder said:

Horse $#!T

Prove it Donkey (A$$) expert ... horse by another name. 

20 hours ago, TheDcoder said:

I don't understand horses

Clearly, so you fell into a trap of your own making. 

20 hours ago, TheDcoder said:

it is called rhash and it is cross-platform.

Damn, I was hoping you couldn't come up with anything ... now I have to pretend I never saw that bit or check it out. 

P.S. I do recall checking that program out way back when ... maybe even before you suggested it, and I'm not using it ... so I wonder what that means? 

[TheDcoder]

13 minutes ago, TheSaint said:

Prove it Donkey (A$$) expert ... horse by another name. 

Already did you stupid a$$ donkey, see my benches

14 minutes ago, TheSaint said:

I do recall checking that program out way back when ... maybe even before you suggested it, and I'm not using it ... so I wonder what that means? 

It means you forgot to check it out properly

[TheDcoder]

If you want a modern, really fast (and FOSS) hashing algorithm, try xxHash. It is neither CRC32 or MD5, but it is explicitly designed for testing data integrity between transfers. It is the one of the fastest algorithms out there. You can download the command-line tool from GitHub.

Edit: Just did a benchmark, it calculated the hash of a 1GB file within ~115ms!

TheDcoder@arch /tmp> time xxhsum bigfile
fd1792309f7da722  bigfile
                                                                      
________________________________________________________
Executed in  115.66 millis    fish           external 
   usr time   75.83 millis  220.00 micros   75.61 millis 
   sys time   39.89 millis   97.00 micros   39.79 millis

 

Edited September 9, 2020 by TheDcoder

[TheSaint]

On 9/10/2020 at 12:11 AM, TheDcoder said:

If you want a modern, really fast (and FOSS) hashing algorithm, try xxHash.

Sounds promising, I'll give it a whirl ... along with rhash.

Looking at the list of the other programs involved in comparative benchmarking, I note CRC32 is very conspicuous by its absence, while MD5 is listed. Just saying.

EDIT

Gawd, trust you to recommend something useless to me .... only 64 bit ... where's the other flavors (XXH32 especially)? 

Edited September 10, 2020 by TheSaint

[TheDcoder]

53 minutes ago, TheSaint said:

Looking at the list of the other programs involved in comparative benchmarking, I note CRC32 is very conspicuous by its absence, while MD5 is listed. Just saying.

I believe "CityHash" is a variant of the CRC algorithm... I might be wrong, just guessing. Who knows what those nerds are up-to anyway

54 minutes ago, TheSaint said:

Gawd, trust you to recommend something useless to me .... only 64 bit ... where's the other flavors (XXH32 especially)? 

Oops, no idea how I missed the obvious win64 bit... looks like there aren't official builds for the 32-bit version, but you are in luck because I found an alternative version built by the MSYS2 project, this should be 32-bit: http://repo.msys2.org/mingw/i686/mingw-w64-i686-xxhash-0.7.4-1-any.pkg.tar.zst

Use 7-zip because zst is a relatively new compression algorithm. Let me know how it goes.

[TheSaint]

24 minutes ago, TheDcoder said:

but you are in luck because I found an alternative version built by the MSYS2 project, this should be 32-bit: http://repo.msys2.org/mingw/i686/mingw-w64-i686-xxhash-0.7.4-1-any.pkg.tar.zst

Use 7-zip because zst is a relatively new compression algorithm. Let me know how it goes.

As if. Clearly my version of 7-Zip is either out-of-date or that file type isn't supported.

Only a nerd would think it sensible to release something in a ZST file. Why for dog's sake? What's wrong with all the most used common formats?

It doesn't bode well for bothering with it, when the mindset behind it is like that.

Anyway thanks bud ... I'm off to look at rhash now ... or perhaps I should just give up.

Me and Tech aren't getting along well today. I had to uninstall Signal because it somehow screwed up one of my contacts, such that I could not send them a normal text message ... and no Signal was not set as the default, and I haven't used it in ages ... a really weird situation. Nerds aren't my favorite people right now.

Edited September 10, 2020 by TheSaint

[TheDcoder]

1 minute ago, TheSaint said:

As if. Clearly my version of 7-Zip is either out-of-date or that file type isn't supported.

Probably the latter, just remembered that ZST is also developed by Yann Corbet (not sure if the name is correct), the same person who made xxHash.

2 minutes ago, TheSaint said:

Only a nerd would think it sensible to release something in a ZST file. Why for dog's sake? What's wrong with all the most used common formats?

This is not a release... it is part of the MSYS2 eco-system of packages, not something your typical joe would be using. The package file is not meant for normal usage, it is automatically download and installed into your computer.

I gave you a link to it because that is the only place where I could find a compiled binary for the 32-bit version... short of compiling it myself .

I repackaged it in a ZIP, enjoy: https://mega.nz/file/OY4wWICJ#e90d_7K-mdN-iqTCbJ7W3ZGNaeRpvfDZHhrMcusmXUE (You can find the executables under the bin folder)

10 minutes ago, TheSaint said:

I had to uninstall Signal because it somehow screwed up one of my contacts

RIP Signal, sad to hear that... and frankly, I don't buy it, it should not have that kind of power... unless you had it set it as your messaging app (you didn't). So I blame your SMS app, OEM apps are shit like that, use something sensible like QKSMS as your SMS app.

[TheSaint]

12 minutes ago, TheDcoder said:

I repackaged it in a ZIP, enjoy: https://mega.nz/file/OY4wWICJ#e90d_7K-mdN-iqTCbJ7W3ZGNaeRpvfDZHhrMcusmXUE (You can find the executables under the bin folder)

Thanks bud, despite my reservations I just downloaded it. EDIT - It ran without complaint, so I guess that means 32 bit.

Wasn't gonna go into detail about my phone issue here. Come join me and Xandy in HexChat.

Edited September 10, 2020 by TheSaint

[TheSaint]

Well I temporarily adapted my program to use XXHASH, and the first test gave the following result.

Quote

[Checksum Test]
xxh32=270638.444581249

Compared to my last test with FSUM.

Quote

fsum-crc32=291519.075982613
fsum-md5=270283.544319351

Once again, this was with same drives and file.

So as can be seen, no real gain of any significance ... certainly not with 3.7 Gb source and compare files.
I suspect there might be more variation on my PC, which only has 2 GB of RAM, if the file size was something like 500 MB or a bit less. I'll have to dig up a file like that from somewhere and test with it. It isn't what I built my program to test though for my personal usage, which is mostly large ISO files (DVD and Blu-ray) ... not forgetting that I won't be using checksum tests myself, unless a rare error surfaced ... in which case I would just probably use TeraCopy to do the comparison anyway.

I should now download RHASH and try that, though I would be surprised if we get much variation with results using that.

Edited September 11, 2020 by TheSaint

[TheDcoder]

@TheSaint Yup, exactly as shown in my test, currently our HDDs are very slow that even the most advanced algorithms are hitting their top speeds waiting for new data to arrive. That is why benchmarks with real HDDs are not really ideal, they have bottlenecks, the speed of the program can't exceed the speed of the data transfer.

So ideally you would be using xxHash since it offers the most performant hash algorithm... it even hits the bottleneck for RAM .

In any case, thanks for playing along... glad to see that xxhash for win32 is working, I was a bit doubtful about that.

[TheSaint]

6 minutes ago, TheDcoder said:

That is why benchmarks with real HDDs are not really ideal, they have bottlenecks, the speed of the program can't exceed the speed of the data transfer.

Yes bud, but surely you realize, that the result in the real world that has those bottlenecks is all that counts, because that is where you will be using it. The rest is just nice theory in essence, because you cannot capitalize on it ... at least for me.

And gawd, benchmarks are meaningless if not based in reality. Who gives a damn about results in perfect Fantasy Land ... only a Nerd ... ideal my ASS.

9 minutes ago, TheDcoder said:

So ideally you would be using xxHash since it offers the most performant hash algorithm... it even hits the bottleneck for RAM .

Honestly bud, I don't see how using it makes any difference, on a practical level ... at least for me.
FSUM is much easier to obtain and essentially works at roughly the same speed ... at least for me.

Maybe if I was using on a PC with much more RAM, I might get much better comparative results, but while that would be good for others, or me in the future, it doesn't help me right now.

So, as things stand, I will happily continue to use FSUM with either MD5 or CRC32 ... roll the dice for which.

[TheDcoder]

3 minutes ago, TheSaint said:

So, as things stand, I will happily continue to use FSUM with either MD5 or CRC32 ... roll the dice for which.

I'd suggest at-least using rhash seeing as how FSUM has some shortcomings... and did I mention rhash is open-source?

[TheDcoder]

I think we missed another important aspect while discussing disk bottlenecks... a faster algorithm also means less CPU usage, which should especially be evident in your low-powered netbook, can you maybe check this? Something like SHA256 hash computing would probably use one of your cores in full-speed but CRC32 might only use less than 25% of it.

[TheSaint]

4 hours ago, TheDcoder said:

Something like SHA256 hash computing would probably use one of your cores in full-speed but CRC32 might only use less than 25% of it.

I am busy for a good while, but I may test that later. It makes me sad, that I paid for 100%, and you are now telling me I only got 25%.

[TheSaint]

So anyway, I did an MD5 and CRC32 test using RHASH.

Quote

[Checksum Test]
rhash-MD5=285820.534324224
rhash-CRC=285800.309776325

Just not much in it.

[TheSaint]

And as requested, FSUM using SHA256.

Quote

[Checksum Test]
fsum-sha256=490923.851671084

@TheDcoder - Oh boy bud, your street cred just plummeted. 

Edited September 16, 2020 by TheSaint

[TheDcoder]

On 9/17/2020 at 1:49 AM, TheSaint said:

So anyway, I did an MD5 and CRC32 test using RHASH.

On 9/17/2020 at 1:49 AM, TheSaint said:

Just not much in it.

Results look much more consistent than fsum, so I suggest replacing fsum with rhash.

On 9/17/2020 at 2:34 AM, TheSaint said:

And as requested, FSUM using SHA256.

I didn't request SHA256, but thanks for showing us the results, since your processor is slower, the disk is no longer a bottleneck. There is no competition between your Intel Atom and my Intel Core i7 really.

On 9/17/2020 at 2:34 AM, TheSaint said:

@TheDcoder - Oh boy bud, your street cred just plummeted. 

For what reason I wonder?

[TheSaint]

On 9/13/2020 at 9:56 PM, TheDcoder said:

.... can you maybe check this? .....

So that wasn't a request then?

Wonder no longer SHA256 Poster Boy. 

Edited September 18, 2020 by TheSaint

[TheDcoder]

7 hours ago, TheSaint said:

So that wasn't a request then?

Wonder no longer SHA256 Poster Boy. 

I said check for CPU usage, not run time, two different things

Since your disk is a bottleneck, your usage should be less than 100% for CRC32 or xxHash, and MD5 should use more of your CPU than those two. Don't check with SHA256, because your processor is not fast enough to hit the disk bottle neck.