Help Make This Slow Function Faster

[CrypticKiwi]

Hey guys, 
I’m stuck with speed again with StringReplace() but this time it’s as slow as a snail. I realize I might have posted a seemingly similar post before but I choose to start a new topic because circumstances are different.
The function replaces 2 numbers with each other based on another number in another file: 

 

 

$Handle1 = FileOpen("ExampleFile01.txt", 0)
$Example1 = FileRead($Handle1)

$Handle2 = FileOpen("ExampleFile02.txt", 0)
$Example2 = FileRead($Handle2)

_Scramble($Example1,$Example2)
Func _Scramble($CycleData,$Data)
    For $N = 1 To StringLen($CycleData) - 10
        Local $SMid = StringMid($Data, $N, 1)
        If $SMid <> 0 Then
            Local $Right = StringMid($CycleData, ($N + $SMid) - 1, 1)
            Local $Left = StringMid($CycleData, $N, 1)
            $CycleData = StringReplace($CycleData, $N, $Right)
            $CycleData = StringReplace($CycleData, Number(($N + $SMid) - 1), $Left)
        EndIf
    Next
    Return $CycleData
EndFunc   ;==>_Scramble

Now, this function takes around 1 hour 30 mins to finish a ~1mb file  on a core i5. the bigger problem is that I need to loop this function ~100000 times :-/
Any help would be greatly appreciated.  

ExampleFile01.txt

ExampleFile02.txt

[Mip101]

Hey CrypticKiwi,
I have rewritten your function to gain some performance, hope it helps.

$Handle1 = FileOpen("ExampleFile01.txt", 0)
$Example1 = FileRead($Handle1)

$Handle2 = FileOpen("ExampleFile02.txt", 0)
$Example2 = FileRead($Handle2)

_Scramble($Example1,$Example2)
Func _Scramble($CycleData,$Data)
    $CycleDataArray = StringSplit($CycleData,"")
    $DataArray = StringSplit($Data,"")
    $ReturnString = ""
    For $i = 1 To StringLen($CycleData) - 10
        If $DataArray[$i] <> 0 Then
            $Temp = $CycleDataArray[$i]
            $CycleDataArray[$i] = $CycleDataArray[$i + $DataArray[$i] - 1]
            $CycleDataArray[$i + $DataArray[$i] - 1] = $Temp
        EndIf
    Next
    For $i = 1 To StringLen($CycleData)
        $ReturnString &= $CycleDataArray[$i]
    Next
    return $ReturnString
EndFunc   ;==>_Scramble

 

[mikell]

Even with a regex looping 1 million times will be long...

Edited August 25, 2015 by mikell

[CrypticKiwi]

Thank you guys for your reply,

Special thanks to Mip101, this is a huge performance boost!! the file got _scrambled in less than 20 secs!!  

Even with a regex looping 1 million times will be long...

thats why im thankful its 100000 not a million.

 

Edited August 25, 2015 by CrypticKiwi

[jchd]

Is that for an encryption scheme? Because 20*10000 seconds (> 23 days) for processing 1Mb file isn't too sexy.

[Danyfirex]

to be faster make de replace part directly in memory. I rewrite your code using structure and making son changes. I should be at least 96% faster based on 100%.

$Handle1 = FileOpen("ExampleFile01.txt", 0)
$Example1 = FileRead($Handle1)

$Handle2 = FileOpen("ExampleFile02.txt", 0)
$Example2 = FileRead($Handle2)

Local $hTimer = TimerInit()
Local $Salida=_Scramble($Example1,$Example2)
Local $fDiff = TimerDiff($hTimer)
MsgBox(0, "Time Difference", $fDiff)

FileWrite("Output.txt",$Salida)

FileClose($Handle1)
FileClose($Handle2)
Func _Scramble($CycleData,$Data)
    Local $Right=""
    Local $Left=""
    Local $SMid=""
    Local $iSize=StringLen($CycleData)
    Local $tCycleData=DllStructCreate("char Data[" & $iSize & "]")
    $tCycleData.Data=$CycleData
    $CycleData=""
    $iSize=$iSize-10
    For $N = 1 To $iSize
        $SMid = StringMid($Data, $N, 1)
        If $SMid <> 0 Then
            $Right=DllStructGetData($tCycleData,1,($N + $SMid) - 1)
            $Left=DllStructGetData($tCycleData,1,$N)
            DllStructSetData($tCycleData,1,$Right,$N)
            DllStructSetData($tCycleData,1,$Left,Number(($N + $SMid) - 1))
        EndIf
    Next
    Return $tCycleData.Data
EndFunc   ;==>_Scramble

Saludos

 

 

Edited August 25, 2015 by Danyfirex

[CrypticKiwi]

 

Is that for an encryption scheme? Because 20*10000 seconds (> 23 days) for processing 1Mb file isn't too sexy.

Nope theoretical  data compression algo. and indeed it aint sexy at all considering ill wait 23 days check result fix stuff repeat!  

yet Mip101's code halves the 23 days with a whooping 39% on my desktop.

on old core i5 laptop
Mip101's = 22604.22
Dany's = 42909.00

on old core 2 duo Desktop
Mip101's = 9040.88 Best So far
Dany's = 20454.23

 

Edited August 25, 2015 by CrypticKiwi

[Danyfirex]

I forgot that Autoit Memory access is slow. so I write this just for fun and for x86. I should take less that 1 second even old Pentium Machine as mine one.

$Handle1 = FileOpen("ExampleFile01.txt", 0)
$Example1 = FileRead($Handle1)
$Handle2 = FileOpen("ExampleFile02.txt", 0)
$Example2 = FileRead($Handle2)
Local $hTimer = TimerInit()
Local $Salida = _Scramble($Example1, $Example2)
Local $fDiff = TimerDiff($hTimer)
MsgBox(0, "Time Difference", $fDiff)
FileWrite("Output.txt", $Salida)
FileClose($Handle1)
FileClose($Handle2)


Func _Scramble($CycleData, $Data)
    Local $lenCycleData = StringLen($CycleData)
    Local $lenData = StringLen($CycleData)
    Local $tCycleData = DllStructCreate("byte Data[" & $lenCycleData & "]")
    Local $tData = DllStructCreate("byte Data[" & $lenData & "]")
    Local $pData = DllStructGetPtr($tData)
    Local $pCycleData = DllStructGetPtr($tCycleData)
    Local $sByteArray = "0x5589E5BE000000008B4D088B550C8B7D10B800000000BB000000008A04372C303C0074253C0072063C097702EB05B801000000BB0000000001F083E8018A1C028A3C32881C32883C0246E2C55DC20C00"
    Local $tByteArray = DllStructCreate("byte Data[" & BinaryLen($sByteArray) & "]")
    $tByteArray.Data = $sByteArray
    $tData.Data = $Data
    $tCycleData.Data = $CycleData
    DllCallAddress("none", DllStructGetPtr($tByteArray), "int", $lenCycleData - 10, "ptr", $pCycleData, "ptr", $pData)
    Return $tCycleData.Data

EndFunc   ;==>_Scramble

Saludos

 

Edited August 26, 2015 by Danyfirex

[UEZ]

@Danyfirex: Can you show us the details for

Local $sByteArray = "0x5589E5BE000000008B4D088B550C8B7D10B800000000BB000000008A04372C303C0074253C0072063C097702EB05B801000000BB0000000001F083E8018A1C028A3C32881C32883C0246E2C55DC20C00"

please?

 

Thx.

Edited August 26, 2015 by UEZ

[Danyfirex]

@UEZ   it is awful assembly code (is not optimized). but if you insist (lol) look:

push ebp
mov ebp, esp
mov esi, 0
mov ecx, [ebp + 08]
mov edx, [ebp + 12]
mov edi, [ebp + 16]
Bucle :
mov eax, 0
mov ebx, 0
mov al, byte[edi + esi]
sub al, 48
cmp al, 0
je Incre
cmp al, 0
jb Clear
cmp al, 9
ja Clear
jmp Mid
Clear :
mov eax, 1
Mid :
mov ebx, 0
add eax, esi
sub eax, 1
mov bl, byte[edx + eax]
mov bh, byte[edx + esi]
mov byte[edx + esi], bl
mov byte[edx + eax], bh
Incre :
inc esi
loop Bucle
pop ebp
ret 12

Saludos 

[jchd]

Here again, native AutoIt strings are UTF16-LE (wchar[N]), not ANSI (char[N]). The distinction doesn't seem to make a difference with the little that the OP told us but if one somehow someday has to handle non ASCII digits, then bytes are not enough.

Edited August 26, 2015 by jchd

[UEZ]

@Danyfirex: When I saw this thread my idea was to code also some ASM lines to increase my skills but I saw that you did it already and I was curious about the code.

I'm also an ASM rookie...

Thanks for sharing!

Edited August 26, 2015 by UEZ

[Danyfirex]

@jchd what's your point?  I got it.

You're wellcome @UEZ.You' have nice assembly posts. I had months without write anything in assembly. 

 

Saludos

Edited August 26, 2015 by Danyfirex

[Mip101]

@Danyfirex

For an ASM noob like me, that ASM function is great! Thanks for sharing

[CrypticKiwi]

OMG Danyfirex this is what they call magic... from 90 minutes to 18 ms im impressed!!

[Danyfirex]

Yes. Assembly look magic. That would be more or less the result if you do it in C / C ++. Autoit is a little slow for this type o things. (That does not take away the love I have for it.)  maybe I'll write some assembly alternative for some string funtions as replace when we need to do some in  very long strings.

Saludos

 

[jchd]

A more flexible alternative if you accept either installing first or FileInstall-ing things, you could submit your C source to TCC, compile + link it on the fly (to .exe or DLL) and invoke it for a (relatively) comparable speed.

More steps, slightly slower, but finally much more versatile than hardcoding machine code in the application.

[LarsJ]

But then you also miss all the fun by writing assembler code.

[jchd]

True, but you gain the joy of C obfuscation.