andygo

and: when version's before stuck with memory allocate error because input file was too big, this here eat's them all 😁 a friend of mine did an extreme test with 6.3 million line input... tooks a few moments but it finish with no errors:

Hello, a long time passed (had to do some things in real life) and since a few days ago i focused again a little more intense on this script. the Last code from LarsJ (postet on 10.August-2015) is realy nice because it is so simple and effective, that i as a non-pro also could unterstand it. i implemented it into my gui-based tool, it is now 4 times faster as my last version. realy nice!!!!!! i hope it is ok for you that i use your code for the main internal solving / sorting part.

hello, i changed variable-names to only two.digit long. this dont change the speed. default messageloop test follows

i will try those hints and see if its more fast ?

inspired from your codes, i could make my one also a bit faster. and theres room for more i think. but i dont want to use just yours (even if its much more complex and faster), i want to understand it and code it by myself. learning by doing, hope you dont see this as snooty

i have a 3.000 lines (1.028 differents) example and a 26.553 lines (7.605 differents). results for both with your 3 examples in seconds: 2.3 seconds (example 1 with 3.000 / 1.028) 1.0 seconds (example 2 with 3.000 / 1.028) 0.8 seconds (example 3 with 3.000 / 1.028) 82.0 seconds (example 1 with 26.553 / 7.605) 17.5 seconds (example 2 with 26.553 / 7.605) 3.7 seconds (example 3 with 26.553 / 7.605) your script in your first post here: 1.3 seconds (3.000 / 1.028) 10.0 seconds (26.553 / 7.605) jchd's sql script: 1.7 seconds (3.000 / 1.028) 3.8 seconds (26.553 / 7.605) executed them all a few times to get exact results.

Manadar, i got your point and my ambition is aroused to find my own solution i am not that pro that your all are, but maybe i come closer :)

Good evening, here are the results with a 25.000 lines input. LarsJ code needs ~ 10 seconds jchd sql code needs ~ 4,5 seconds. special test with a 156.000 lines file (34mb input): solved in 21 seconds. crazy fast

realisitc input size in most cases is ~ 3.000 to 5.000 lines. to test a 25.000 lines file was a pal's idea just to see the scripts result and time. but i like it and no matter if a user dont recognize timediffernce between yours and jchd sql solution, i will test it too feedback follows.

LarsJ, this is pretty cool . tried it and works. also with a 25.000 lines file. now i will try to use it with GUI. Feedback follows in 2 days latest.

#include <File.au3> #include <StaticConstants.au3> #include <EditConstants.au3> #include <ComboConstants.au3> #include <GuiConstants.au3> #include <WindowsConstants.au3> #include <ButtonConstants.au3> #include <IE.au3> #include <Timers.au3> #include <Process.au3> #Include <Misc.au3> #include <Constants.au3> Break(0) Opt("TrayAutoPause",0) Opt("GuiOnEventMode", 1) if msgbox(65, "Hinweis", "Das Tool wird ggf. von einigen Schutzprogrammen negativ erkannt."& @CRLF & @CRLF &"Ich distanziere mich hiermit ausdrücklich von Schadsoftware"& @CRLF &"in jeglicher Form!"& @CRLF & @CRLF &"©2015 andygo") = 2 then exit global $loopstop = 0, $v = 0, $what, $writecount, $aRecords, $starttime $pre = GuiCreate("EMM.Tool", 290, 150) $laden = GUICtrlCreateButton ("Datei laden", 4, 4, 90, 20) GUICtrlSetOnEvent($laden, "_verarbeiten") $breakup = GUICtrlCreateCheckbox("STOP", 100, 4, 90, 20) GUICtrlSetState($breakup, $GUI_DISABLE) $save = GUICtrlCreateButton ("Datei speichern", 195, 4, 90, 20) GUICtrlSetState($save, $GUI_DISABLE) GUICtrlSetOnEvent($save, "_ausgabe") $inf1 = GUICtrlCreateLabel("", 4, 55, 282, 20, $SS_CENTER) $inf2 = GUICtrlCreateLabel("", 4, 75, 282, 20, $SS_CENTER) $label = GUICtrlCreateLabel("Bereit...", 4, 110, 282, 40, $SS_CENTER) GUICtrlSetFont($label, 16, 500) GUISetOnEvent($GUI_EVENT_CLOSE, "_Quit2") GUISetState(@SW_SHOW) while 1 Sleep(50) wend Func _Quit2() if msgbox(68,"EMM.Tool","Wirklich beenden?") = 6 then Exit EndFunc func addup($a, $b, $c, $d) GUICtrlSetData($label, $what & $v) GUICtrlSetData($inf2, "In: "&$aRecords[0] & " Datensätze ** " & "Out: " &$writecount & " Datensätze") if GUICtrlRead($breakup) = 1 then $loopstop = 1 endfunc func addsave($a, $b, $c, $d) GUICtrlSetData($label, "Schreibe... " & $writecount) endfunc func _verarbeiten() Opt("GuiOnEventMode", 0) GUICtrlSetState($laden, $GUI_DISABLE) GUICtrlSetState($save, $GUI_DISABLE) GUICtrlSetState($breakup, $GUI_UNCHECKED) GUICtrlSetData($label, "Bereit...") GUICtrlSetData($inf1, "") GUICtrlSetData($inf2, "") $infile = FileOpenDialog("Datei", @ScriptDir & "\", "emm-log (*.*)", 1) If @error Then ;MsgBox(4096, "", "No File(s) chosen") Else If Not _FileReadToArray($infile, $aRecords) Then MsgBox(4096, "Fehler", "Fehlerhafte Datei") Else $writecount = 0 $v = StringSplit($infile, "\") GUICtrlSetData($inf1, $infile) GUICtrlSetData($inf2, $aRecords[0] & " Datensätze gelesen") $v = $aRecords[0] $what = "Prüfe... " GUICtrlSetState($breakup, $GUI_enABLE) global $dRecords[$aRecords[0]+1], $eRecords[$aRecords[0]+1], $i, $x, $y, $date _Timer_SetTimer($pre,50,"addup") For $x = 1 To $aRecords[0] if $loopstop = 1 then exitloop $v -= 1 if StringLen ($aRecords[$x]) > 0 Then $bRecords = StringLeft($aRecords[$x], 59) $bRecords = StringReplace($bRecords, StringMid ( $bRecords, 22, 20), " ") $dRecords[$x] = StringReplace($bRecords, 33, "xxxxxx") $bRecords = StringSplit($aRecords[$x], " ", 1) $eRecords[$x] = $bRecords[3] endif Next $v = 0 for $x = 1 to ($aRecords[0] - 1) $v += $x next $what = "Vergleiche... " $i = 1 For $y = 1 To $aRecords[0]-1 if $loopstop = 1 then ExitLoop For $x = $y+1 To $aRecords[0] if $loopstop = 1 then exitloop 2 $v -= 1 if $eRecords[$y] = $eRecords[$x] and StringLen ($eRecords[$y]) > 0 then $i += 1 $eRecords[$x] = "" $date = StringLeft($dRecords[$x], 19) endif next if StringLen($eRecords[$y]) > 0 then $writecount += 1 if $i = 1 then $dRecords[$y] = $dRecords[$y] &" "& $i if $i > 1 then $dRecords[$y] = $dRecords[$y] &" "& $i &" "& $date endif $i = 1 next _Timer_KillAllTimers($pre) GUICtrlSetData($inf2, "In: "&$aRecords[0] & " Datensätze ** " & "Out: " &$writecount & " Datensätze") GUICtrlSetState($breakup, $GUI_DISABLE) if $loopstop = 1 Then GUICtrlSetData($label, "Abbruch durch Benutzer") GUICtrlSetState($breakup, $GUI_UNCHECKED) GUICtrlSetData($inf1, "") GUICtrlSetData($inf2, "") $loopstop = 0 Else GUICtrlSetData($label, "Fertig :-)") GUICtrlSetState($save, $GUI_enABLE) endif endif endif GUICtrlSetState($laden, $GUI_enABLE) Opt("GuiOnEventMode", 1) endfunc func _ausgabe() GUICtrlSetState($save, $GUI_DISABLE) GUICtrlSetState($laden, $GUI_disABLE) Local $file = FileSaveDialog("Datei", @ScriptDir, "emm-log (*.txt)", 2+16) If @error Then GUICtrlSetData($label, "Fertig :-)") Else if StringInStr(StringRight($file, 4), ".") = 0 then $file = $file & ".txt" FileOpen($file, 2) If $file = -1 Then GUICtrlSetData($label, "Fertig :-)") FileClose($file) MsgBox(0, "Fehler", "Speichern nicht möglich.") else _Timer_SetTimer($pre,50,"addsave") GUICtrlSetState($breakup, $GUI_enABLE) FileWriteLine($file, "[code]") For $x = 1 To $aRecords[0] if StringLen($eRecords[$x]) > 0 then FileWriteLine($file, $dRecords[$x]) $writecount -= 1 endif next FileWriteLine($file, "[/code]") FileClose($file) _Timer_KillAllTimers($pre) GUICtrlSetState($breakup, $GUI_DISABLE) if $loopstop = 1 Then GUICtrlSetData($label, "Abbruch durch Benutzer") GUICtrlSetState($breakup, $GUI_UNCHECKED) $loopstop = 0 Else GUICtrlSetData($label, "Datei gespeichert") endif Endif endif GUICtrlSetState($save, $GUI_enABLE) GUICtrlSetState($laden, $GUI_enABLE) endfunc The main loop-inloop is between line 91 and line 108. this is what takes time. here is an example of 5 lines, how a file looks like to feed the script with. you can create a 3.000 lines file from it. the lines always start with date-time-stamp in always same format. then 3 spaces, then a block of 16 chars, then again 3 spaces. in each line this is always the same count of chars. now it follows a long long string, this could be different long in each line. it is followed always by 3 spaces and then the word 'written' or 'blocked' so, now what my script does: it counts how many times the long long string exists in a file, so each line will be compared with each. it counts the same lines, cut them of a little bit, writes the count behind the cutted string and behind the count, the date-time from the last same string. the output file then looks like this: 2015/02/15 04:48:55 8270870102xxxxxx81 1 2015/02/15 04:48:56 8270300102xxxxxx0F 1 2015/02/15 04:49:03 8270870102xxxxxx16 1 2015/02/15 04:49:07 8270870102xxxxxx70 1 2015/02/15 04:49:07 8270200102xxxxxx0F 1 2015/02/15 04:39:57 8270870102xxxxxx81 142 2015/02/20 12:01:20 2015/02/15 04:39:58 8270870102xxxxxx81 144 2015/02/20 11:51:56 2015/02/15 04:39:58 8270310102xxxxxx0F 1 2015/02/15 04:40:05 8270870102xxxxxx81 142 2015/02/20 11:51:57 2015/02/15 04:40:09 8270970102xxxxxx91 77 2015/02/20 11:57:40 inexample.txt

i will post the code today afternoon, still at work now.

ok, i tried 64 compile and process priority, both didnt change the speed. icleaned my loop to the needed times:$totalloops = 0for x = 1 to array[0] - 1$totalloops += 1nextfor a 3.000 Items array i need (2.999 + 2.998 + 2.997.... + 1) loops and this takes now only 13 seconds.i can deal with that

hello, i have a script which can handle huge data. i feed it with textfiles, it reads each line into array, then compare the lines, do some string operations. this all happens in a for.to.next loop. the script only use 15% cpu of my 8core.amd can i force the script to use more cpuload an therefore being faster? would it make a speed difference to compile it as 64bit exe on 64bit systems? thank you for commemts :)

simple own msgbox: #include <GUIConstantsEx.au3> #include <WindowsConstants.au3> #include <StaticConstants.au3> $action = "" Opt("GUIOnEventMode", 0) $gui_3 = GUICreate("msgbox", 326, 145, -1, -1, $WS_CAPTION) $gui_3text = GUICtrlCreateLabel("text", 10, 50, 306, 20, $SS_CENTER) $gui_3ja = GUICtrlCreateButton("Yes", 75, 90, 50, 20) $gui_3nein = GUICtrlCreateButton("No", 201, 90, 50, 20) GUISetState() while 1 sleep(20) $mMsg = GUIGetMsg(1) Switch $mMsg[0] case $gui_3ja $action = "yes" exitloop case $gui_3nein $action = "no" exitloop EndSwitch wend msgbox(64, "info", "you clicked "&$action)