MD5,SHA1,CRC32,RC4,BASE64,XXTEA machine code version

[KaFu]

Just that the info isn't lost:

Trancexx mentioned that he had DEP (= Data Execution Prevention "Feature" on Vista) issues with his functions on several systems.

Monoceres reply was:

I haven't got problems with DEP so I can't test, but shouldn't protecting the memory with the EXECUTE_READWRITE flag with the VirtualProtect function solve that issue?

Will take a deeper look when I've got some spare time... or someone else takes this task ...

Edited May 27, 2009 by KaFu

[KaFu]

Inspired by trancexx post here: #704656

Just a try , but maybe this will prevent DEP issues on Vista+. Used _MemVirtualAlloc() once in header, _MemVirtualFree() seems unnecessary, hopefully this will not result in the universe to collapse ...

Edit: Here's an update of the example...

Edited July 23, 2009 by KaFu

[Venom007]

Thank you for this one! Looks good and sooooooooo fast!

[scandisk]

Very nice!

[hawky358]

Hey

I've been having some trouble with the code above.

When you've hashed around 508 files in a single session (i.e. without closing and reopening the program) the output of all files is returned as:

d41d8cd98f00b204e9800998ecf8427e which seems to be the MD5 of nothing...

I've tried unloading the memory and clearing the variables, can anyone help?

[KaFu]

I've been having some trouble with the code above.

I've hashed tens of thousands of files with this code in SMF... you'll have to provide an example to see what's going wrong.

[hawky358]

I've hashed tens of thousands of files with this code in SMF... you'll have to provide an example to see what's going wrong.

Okay

As an example I just did an MD5 on a single file 1000 times, the same happens on multiple files

I took the exact code as above and just changed the example part:

; ====================================================
; Example Start
; ====================================================
$Hash_BufferSize = 1024 * 1024
for $count = 1 to 1000

;$sFile = FileOpenDialog("Choose file", "", "All files (*)")
;If @error Then Exit
$sfile = "c:\test.file"
$hFile = FileOpen($sFile, 16)

$MD5CTX = _MD5Init()
For $i = 1 To Ceiling(FileGetSize($sFile) / $Hash_BufferSize)
    _MD5Input($MD5CTX, FileRead($hFile, $Hash_BufferSize))
Next
$hashed = _MD5Result($MD5CTX)
ConsoleWrite($count & ": " & $hashed & @CR)
Next
; ====================================================
; Example End
; ====================================================

My console output was:

1: 0x0D804C52E491127DD1AFCED29434A0F9
2: 0x0D804C52E491127DD1AFCED29434A0F9
3: 0x0D804C52E491127DD1AFCED29434A0F9
.
.
.
508: 0x0D804C52E491127DD1AFCED29434A0F9
509: 0x0D804C52E491127DD1AFCED29434A0F9
510: 0xD41D8CD98F00B204E9800998ECF8427E
511: 0xD41D8CD98F00B204E9800998ECF8427E
.
.
.
998: 0xD41D8CD98F00B204E9800998ECF8427E
999: 0xD41D8CD98F00B204E9800998ECF8427E
1000: 0xD41D8CD98F00B204E9800998ECF8427E

From 510 it shows the "Nothing" MD5.

With a GUI this happens until you physically close the program and start it again

Just to confirm I have tried this on four machines.

Three running XP and one running Vista all with the same result

edit: Clarified problem

Edited July 22, 2009 by hawky358

[KaFu]

My fault ...

Add FileClose($hFile) directly before the next... updated my example...

[KaFu]

Here's a small update of my MD5 example. Example now checks same file 10 times (file closed correctly )... just added/changed some minor things after a PM discussion with trancexx (thanks for the input!).

expandcollapse popup

#include <Memory.au3> ; for _MemVirtualAlloc(0, 512, $MEM_COMMIT, $PAGE_EXECUTE_READWRITE), to prevent DEP issues

; http://www.autoitscript.com/forum/index.php?showtopic=76976
; By Ward

Global $_MD5Opcode = '0xC85800005356576A006A006A008D45A850E8280000006A00FF750CFF75088D45A850E8440000006A006A008D45A850FF7510E8710700005F5E5BC9C210005589E58B4D0831C0894114894110C70101234567C7410489ABCDEFC74108F'
$_MD5Opcode &= 'EDCBA98C7410C765432105DC21000C80C0000538B5D088B4310C1E80383E03F8945F88B4510C1E0030143103943107303FF43148B4510C1E81D0143146A40582B45F88945F4394510724550FF750C8B45F88D44031850E8A00700008D4'
$_MD5Opcode &= '3185053E84E0000008B45F48945FC8B45FC83C03F39451076138B450C0345FC5053E8300000008345FC40EBE28365F800EB048365FC008B45102B45FC508B450C0345FC508B45F88D44031850E84D0700005BC9C21000C840000053565'
$_MD5Opcode &= '76A40FF750C8D45C050E8330700008B45088B088B50048B70088B780C89D021F089D3F7D321FB09D801C1034DC081C178A46AD7C1C10701D189C821D089CBF7D321F309D801C7037DC481C756B7C7E8C1C70C01CF89F821C889FBF7D32'
$_MD5Opcode &= '1D309D801C60375C881C6DB702024C1C61101FE89F021F889F3F7D321CB09D801C20355CC81C2EECEBDC1C1C21601F289D021F089D3F7D321FB09D801C1034DD081C1AF0F7CF5C1C10701D189C821D089CBF7D321F309D801C7037DD48'
$_MD5Opcode &= '1C72AC68747C1C70C01CF89F821C889FBF7D321D309D801C60375D881C6134630A8C1C61101FE89F021F889F3F7D321CB09D801C20355DC81C2019546FDC1C21601F289D021F089D3F7D321FB09D801C1034DE081C1D8988069C1C1070'
$_MD5Opcode &= '1D189C821D089CBF7D321F309D801C7037DE481C7AFF7448BC1C70C01CF89F821C889FBF7D321D309D801C60375E881C6B15BFFFFC1C61101FE89F021F889F3F7D321CB09D801C20355EC81C2BED75C89C1C21601F289D021F089D3F7D'
$_MD5Opcode &= '321FB09D801C1034DF081C12211906BC1C10701D189C821D089CBF7D321F309D801C7037DF481C7937198FDC1C70C01CF89F821C889FBF7D321D309D801C60375F881C68E4379A6C1C61101FE89F021F889F3F7D321CB09D801C20355F'
$_MD5Opcode &= 'C81C22108B449C1C21601F289D021F889FBF7D321F309D801C1034DC481C162251EF6C1C10501D189C821F089F3F7D321D309D801C7037DD881C740B340C0C1C70901CF89F821D089D3F7D321CB09D801C60375EC81C6515A5E26C1C60'
$_MD5Opcode &= 'E01FE89F021C889CBF7D321FB09D801C20355C081C2AAC7B6E9C1C21401F289D021F889FBF7D321F309D801C1034DD481C15D102FD6C1C10501D189C821F089F3F7D321D309D801C7037DE881C753144402C1C70901CF89F821D089D3F'
$_MD5Opcode &= '7D321CB09D801C60375FC81C681E6A1D8C1C60E01FE89F021C889CBF7D321FB09D801C20355D081C2C8FBD3E7C1C21401F289D021F889FBF7D321F309D801C1034DE481C1E6CDE121C1C10501D189C821F089F3F7D321D309D801C7037'
$_MD5Opcode &= 'DF881C7D60737C3C1C70901CF89F821D089D3F7D321CB09D801C60375CC81C6870DD5F4C1C60E01FE89F021C889CBF7D321FB09D801C20355E081C2ED145A45C1C21401F289D021F889FBF7D321F309D801C1034DF481C105E9E3A9C1C'
$_MD5Opcode &= '10501D189C821F089F3F7D321D309D801C7037DC881C7F8A3EFFCC1C70901CF89F821D089D3F7D321CB09D801C60375DC81C6D9026F67C1C60E01FE89F021C889CBF7D321FB09D801C20355F081C28A4C2A8DC1C21401F289D031F031F'
$_MD5Opcode &= '801C1034DD481C14239FAFFC1C10401D189C831D031F001C7037DE081C781F67187C1C70B01CF89F831C831D001C60375EC81C622619D6DC1C61001FE89F031F831C801C20355F881C20C38E5FDC1C21701F289D031F031F801C1034DC'
$_MD5Opcode &= '481C144EABEA4C1C10401D189C831D031F001C7037DD081C7A9CFDE4BC1C70B01CF89F831C831D001C60375DC81C6604BBBF6C1C61001FE89F031F831C801C20355E881C270BCBFBEC1C21701F289D031F031F801C1034DF481C1C67E9'
$_MD5Opcode &= 'B28C1C10401D189C831D031F001C7037DC081C7FA27A1EAC1C70B01CF89F831C831D001C60375CC81C68530EFD4C1C61001FE89F031F831C801C20355D881C2051D8804C1C21701F289D031F031F801C1034DE481C139D0D4D9C1C1040'
$_MD5Opcode &= '1D189C831D031F001C7037DF081C7E599DBE6C1C70B01CF89F831C831D001C60375FC81C6F87CA21FC1C61001FE89F031F831C801C20355C881C26556ACC4C1C21701F289F8F7D009D031F001C1034DC081C1442229F4C1C10601D189F'
$_MD5Opcode &= '0F7D009C831D001C7037DDC81C797FF2A43C1C70A01CF89D0F7D009F831C801C60375F881C6A72394ABC1C60F01FE89C8F7D009F031F801C20355D481C239A093FCC1C21501F289F8F7D009D031F001C1034DF081C1C3595B65C1C1060'
$_MD5Opcode &= '1D189F0F7D009C831D001C7037DCC81C792CC0C8FC1C70A01CF89D0F7D009F831C801C60375E881C67DF4EFFFC1C60F01FE89C8F7D009F031F801C20355C481C2D15D8485C1C21501F289F8F7D009D031F001C1034DE081C14F7EA86FC'
$_MD5Opcode &= '1C10601D189F0F7D009C831D001C7037DFC81C7E0E62CFEC1C70A01CF89D0F7D009F831C801C60375D881C6144301A3C1C60F01FE89C8F7D009F031F801C20355F481C2A111084EC1C21501F289F8F7D009D031F001C1034DD081C1827'
$_MD5Opcode &= 'E53F7C1C10601D189F0F7D009C831D001C7037DEC81C735F23ABDC1C70A01CF89D0F7D009F831C801C60375C881C6BBD2D72AC1C60F01FE89C8F7D009F031F801C20355E481C291D386EBC1C21501F28B4508010801500401700801780'
$_MD5Opcode &= 'C5F5E5BC9C20800C814000053E840000000800000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000008F45EC8B5D0C6A088D43105'
$_MD5Opcode &= '08D45F850E8510000008B4310C1E80383E03F8945F483F838730B6A38582B45F48945F0EB096A78582B45F48945F0FF75F0FF75ECFF750CE831F8FFFF6A088D45F850FF750CE823F8FFFF6A1053FF7508E8050000005BC9C210005589E'
$_MD5Opcode &= '55156578B7D088B750C8B4D10FCF3A45F5E595DC20C00'


; Allocating memory with $PAGE_EXECUTE_READWRITE, to prevent DEP (= Data Execution Prevention "Feature" on Vista) errors
; ConsoleWrite(BinaryLen($_MD5Opcode) & @crlf) ; = 2161
Global $pRemoteCodeMD5 = _MemVirtualAlloc(0, 2161, $MEM_COMMIT, $PAGE_EXECUTE_READWRITE)

Global $_MD5CodeBuffer = DllStructCreate("byte[2161]", $pRemoteCodeMD5)
DllStructSetData($_MD5CodeBuffer, 1, $_MD5Opcode)

; ====================================================
; Example Start
; ====================================================
Global $Hash_BufferSize = 1024 * 170 ; good size for performance after some testing, at least for me...

$sFile = FileOpenDialog("Choose file", "", "All files (*)")
If @error Then Exit

For $n = 1 To 10

    $timer = TimerInit()
    $hFile = FileOpen($sFile, 16)

    $MD5CTX = _MD5Init()
    For $i = 1 To Ceiling(FileGetSize($sFile) / $Hash_BufferSize)
        _MD5Input($MD5CTX, FileRead($hFile, $Hash_BufferSize))
    Next

    FileClose($hFile)

    $MD5Result = _MD5Result($MD5CTX)
    
    If $MD5Result = '0xD41D8CD98F00B204E9800998ECF8427E' Then $MD5Result = "0 Byte File"
    ConsoleWrite('MD5 for ' & $sFile & ' = ' & $MD5Result & ', calculated in ' & TimerDiff($timer) & 'ms' & @CRLF)

Next

;MsgBox(0,'','MD5 for ' & $sFile & ' = ' & _MD5Result($MD5CTX))
; ====================================================
; Example End
; ====================================================

Func _MD5Init()

    Local $OpcodeStart = 62

    Local $MD5CTX = DllStructCreate("dword[22]")
    DllCall("user32.dll", "none", "CallWindowProcW", "ptr", DllStructGetPtr($_MD5CodeBuffer) + $OpcodeStart, _
            "ptr", DllStructGetPtr($MD5CTX), _
            "int", 0, _
            "int", 0, _
            "int", 0)

    Return $MD5CTX

EndFunc   ;==>_MD5Init

Func _MD5Input(ByRef $MD5CTX, $Data)

    ;Local $OpcodeStart = (StringInStr($_MD5Opcode, "C80C0000538B5D08") - 1) / 2 - 1
    Local $OpcodeStart = 107

    Local $Input = DllStructCreate("byte[" & BinaryLen($Data) & "]")
    DllStructSetData($Input, 1, $Data)

    DllCall("user32.dll", "none", "CallWindowProcW", "ptr", DllStructGetPtr($_MD5CodeBuffer) + $OpcodeStart, _
            "ptr", DllStructGetPtr($MD5CTX), _
            "ptr", DllStructGetPtr($Input), _
            "int", BinaryLen($Data), _
            "int", 0)

    $Input = 0

EndFunc   ;==>_MD5Input

Func _MD5Result(ByRef $MD5CTX)

    Local $OpcodeStart = 1960

    Local $Digest = DllStructCreate("byte[16]")

    DllCall("user32.dll", "none", "CallWindowProcW", "ptr", DllStructGetPtr($_MD5CodeBuffer) + $OpcodeStart, _
            "ptr", DllStructGetPtr($Digest), _
            "ptr", DllStructGetPtr($MD5CTX), _
            "int", 0, _
            "int", 0)

    Local $Ret = DllStructGetData($Digest, 1)
    $Digest = 0
    $MD5CTX = 0
    Return $Ret

EndFunc   ;==>_MD5Result

Edited July 23, 2009 by KaFu

[Lyubo]

Hello,

I need to implement DES ( Triple DES ) in my application.

Can anyone help me?

Thank you in advance

[Lyubo]

Thanks to Lazycat's great idea. He told us how to use the CallWindowProc API to run machine code.

So I wrote some codes in Sphinx C--, and run them in AutoIt.

Here are MD5, SHA1, CRC32, RC4, and BASE64 algorithm in the attachment. (My CRC32 Code is a bit faster then Lazycat's version)

SHA1_MD5_RC4_BASE64_CRC32.zip

If someone needs other algorithm, please let me know, and I will try my best.

If you are looking for SHA2 or AES, here and here.

2008/08/02 Update Note:

Corrected Block TEA (XXTEA) block cipher is added. Take a look at Wikipedia for detail.

SHA1_MD5_RC4_BASE64_CRC32_XXTEA.zip

2008/08/24 Update Note:

Add a parameter $LineBreak to _Base64Encode(). This function will break output line in every 76 characters by default. Thanks to trancexx.

SHA1_MD5_RC4_BASE64_CRC32_XXTEA.zip

2008/09/27 Update Note:

Optimize machine code in _Base64Encode() and _Base64Decode().

Fix _Base64Encode() crash problem. Thanks to trancexx.

SHA1_MD5_RC4_BASE64_CRC32_XXTEA.zip

Great works! Thank You!

I need DES algorithm encrypt/decrypt functionality.

Can You help me?

[skyteddy]

Hello Kafu, hello everybody,

Here's a small update of my MD5 example. Example now checks same file 10 times (file closed correctly )... just added/changed some minor things after a PM discussion with trancexx (thanks for the input!).

I used your example and it works pretty good, when I use the "normal" x32-compiler.

If I use the x64-compiler, the program crashes after the fileselection. Independed if I use the compiler of the version 3.3.0.0 or the compiler from the actual beta 3.3.1.5

Problemsignatur:

Problemereignisname: APPCRASH

Anwendungsname: md5-1.exe

Anwendungsversion: 3.3.1.5

Anwendungszeitstempel: 4af350fa

Fehlermodulname: StackHash_581e

Fehlermodulversion: 0.0.0.0

Fehlermodulzeitstempel: 00000000

Ausnahmecode: c0000005

Ausnahmeoffset: 0000000000a40046

Betriebsystemversion: 6.1.7600.2.0.0.256.1

Gebietsschema-ID: 1031

Zusatzinformation 1: 581e

Zusatzinformation 2: 581ee8acabc9f34eead0f4d1eb4ede4d

Zusatzinformation 3: 7ed6

Zusatzinformation 4: 7ed6b481e599cd06b45676cd74321085

Lesen Sie unsere Datenschutzbestimmungen online:

http://go.microsoft.com/fwlink/?linkid=104288&clcid=0x0407

Wenn die Onlinedatenschutzbestimmungen nicht verfügbar sind, lesen Sie unsere Datenschutzbestimmungen offline:

C:\Windows\system32\de-DE\erofflps.txt

I try also then _MD5-function from the posting number 1, but it crashes too.

At least, I try the other md5-udf from your footer. They don't crash, but on each second file, I don't get a md5-checksum. :-(

What can I do?

Many thanks in advance und viele Gruesse an das andere Ende Deutschlands!

R@iner

PS: I need the md5-calculation for files and for strings.

[KaFu]

Hi R@iner,

I assume the reason why it crashes on 64bit is that the ASM Opcode is native 32bit. The only way to solve this would be to write an 64bit Opcode... and I don't have a clue how to do this . But as a work-around, is it really necessary to compile the code as 64bit? Compile it for 32bit and run it in "{SystemDrive} & "\Program Files (x86)". The other code in my sig is a Windows API, don't have a clue why it fails for you. Post an example in that thread, maybe trancexx will take a look herself.

[MvGulik]

Your post is empty Edited February 12, 2011 by iEvKI3gv9Wrkd41u

[monoceres]

Hi R@iner,

I assume the reason why it crashes on 64bit is that the ASM Opcode is native 32bit. The only way to solve this would be to write an 64bit Opcode... and I don't have a clue how to do this . But as a work-around, is it really necessary to compile the code as 64bit? Compile it for 32bit and run it in "{SystemDrive} & "\Program Files (x86)". The other code in my sig is a Windows API, don't have a clue why it fails for you. Post an example in that thread, maybe trancexx will take a look herself.

Or even better: Use the standardized functions available in the latest beta. They're safe, fast, small, easy to use and guarantied to work on a every system autoit works on. Unless you need the small perfomance gain or the algorithms it doesn't support (CRC32 for exaple) the standard UDF is better to use.

[skyteddy]

Hello KaFu, hello monoceres,

thank you for your help!

Or even better: Use the standardized functions available in the latest beta.

Great, this works if I use the 3.3.1.5 compiler too. My idea was, only to copy the crypt.au3 in me workingdir and include this local crypt.au3 and compile my script with 3.3.0.0 but it doesn't work. I get an error 1 and don't know why.

They're safe, fast, small, easy to use and guarantied to work on a every system autoit works on.

Yes, it is easy to use, but with md5 its slower. It works on x32 und x64 operating systems. This is the most important point.

My challenge is now, to modify my script, that I can compile it with the new beta-compiler. There a many changes between the final and the latest beta, e.g. the Inet functions.

many thanks again!

R@iner

[ngskicker]

can someone give a simple of huffman coding

[IchBistTod]

Hi R@iner,

I assume the reason why it crashes on 64bit is that the ASM Opcode is native 32bit. The only way to solve this would be to write an 64bit Opcode... and I don't have a clue how to do this . But as a work-around, is it really necessary to compile the code as 64bit? Compile it for 32bit and run it in "{SystemDrive} & "\Program Files (x86)". The other code in my sig is a Windows API, don't have a clue why it fails for you. Post an example in that thread, maybe trancexx will take a look herself.

even this does not make it work for me win 7 ultimate 64 bit....

=[

The small performance improvement is nice, especially when being used several thousands of times in a row.

[KaFu]

even this does not make it work for me win 7 ultimate 64 bit....

It works fine for me... I use this code in SMF (32bit compiled), and I'm running Win7-64bit myself.

[Electon]

the base64 code doesnt seem to be working anymore. im running win7 64bit. your example script just crashes