make script faster

[jvanegmond]

@andygo, the reason their scripts are much faster is because their algorithms, their methods for solving the problem, are much more efficient. It's not necessarily because LarsJ thinks arrays should be structs or that jhcd likes to bundle SQLite as a dependency. That's in jest to illustrate the point, but the respective solutions do need to be objectively considered.

If you were to write out in plain English the concepts behind both of their solutions, you could see even with a layman's view quite clearly why their solutions are much faster. Their total complexity, the total number of operations required, are signifcantly lower. Thus resulting in lower runtimes. You could use your weapons of choice, string and array manipulation, to approach similar results.

Currently you are searching the entire dataset to see if it matches with every other element in the dataset. If your dataset is 1000 big, your total number of comparisons will be 1000x1000. If you keep track of the unique elements you processed, you would for each element first check if you actually need to check it against every other element. Because the set of unique elements is very small, the total number of comparisons would go down dramatically. It would be closer to 1000x10, which is already a 100 fold improvement. That's the difference between 5 minutes and 3 seconds and then you're only just getting started.

[LarsJ]

I'll bet I can beat the SQL-code with a few optimizations. I'll show some code tomorrow.

[jchd]

Quite probably. I was just illustrating a different avenue.

[andygo]

Manadar, i got your point and my ambition is aroused to find my own solution  i am not that pro that your all are, but maybe i come closer :)

[LarsJ]

Here are three examples. The difference between the examples is the sort function.

For all three examples duplicate strings are not included in the sorting. Duplicates are counted and timestamps are stored. That's all. It's assumed that timestamps for duplicates is in ascending order.

The first example is not even a sorting but only a listing of the different strings. A simple For-loop is used to create the list. Because of simplicity this eksemple is fast for a very very small number of different strings.

expandcollapse popup

#include <Array.au3>

Opt( "MustDeclareVars", 1 )

Example()


Func Example()
  ; 1. Read file
  Local $aArray1 = FileReadToArray( "inexample1.txt" )
  ;_ArrayDisplay( $aArray1 )

  ; 2. Create 2D-array
  Local $iRows = UBound( $aArray1 )
  Local $aArray2[$iRows][6], $aLine ; Col 4 : Last time, Col 5 : Count
  For $i = 0 To $iRows - 1
    $aLine = StringSplit ( $aArray1[$i], "   ", 3 ) ; 3 = $STR_ENTIRESPLIT + $STR_NOCOUNT
    $aArray2[$i][0] = $aLine[0] ; Col 0 : Time
    $aArray2[$i][1] = $aLine[1] ; Col 1 : 16 chars
    $aArray2[$i][2] = $aLine[2] ; Col 2 : Long string
    $aArray2[$i][3] = $aLine[3] ; Col 3 : blocked/written
  Next
  ;_ArrayDisplay( $aArray2 )

  ; 3. Index based list of col 2 (long string) in $aArray2
  ; (Rows with duplicate strings are counted and times stored)
  Local $aIndex[$iRows]
  Local $nIndex = 0 ; Number of rows in index
  ListArray( $aArray2, $aIndex, $nIndex )
  ;_ArrayDisplay( $aArray2 )

  ; 4. Extract results
  Local $aResults[$nIndex], $k
  For $i = 0 To $nIndex - 1
    $k = $aIndex[$i]
    $aResults[$i] = $aArray2[$k][0] & "   " & _ ; First time
                    StringLeft( $aArray2[$k][2], 10 ) & "xxxxxx" & StringMid ( $aArray2[$k][2], 17, 2 ) & "   " & _
                    StringFormat( "%3d", $aArray2[$k][5] ) & "   " & _ ; Count
                    $aArray2[$k][4] ; Last time
    ConsoleWrite( $aResults[$i] & @CRLF )
  Next
EndFunc


Func ListArray( ByRef $aItems, ByRef $aIndex, ByRef $n ) ; $n = number of rows in index
  For $i = 0 To UBound( $aItems ) - 1
    For $j = 0 To $n - 1
      If Not StringCompare( $aItems[$i][2], $aItems[$aIndex[$j]][2] ) Then ExitLoop
    Next
    If $j < $n Then ; Found
      $aItems[$aIndex[$j]][4] = $aItems[$i][0] ; Last time
      $aItems[$aIndex[$j]][5] += 1 ; Count
    Else ; Not found
      $aItems[$i][4] = $aItems[$i][0] ; Last time
      $aItems[$i][5] += 1 ; Count
      $aIndex[$n] = $i
      $n += 1
    EndIf
  Next
EndFunc

For a medium number of different strings a binary sorting of the strings is probably faster. The index is just a normal array. A For-loop is used to make room in the index for new rows.

expandcollapse popup

#include <Array.au3>

Opt( "MustDeclareVars", 1 )

Example()


Func Example()
  ; 1. Read file
  Local $aArray1 = FileReadToArray( "inexample1.txt" )
  ;_ArrayDisplay( $aArray1 )

  ; 2. Create 2D-array
  Local $iRows = UBound( $aArray1 )
  Local $aArray2[$iRows][6], $aLine ; Col 4 : Last time, Col 5 : Count
  For $i = 0 To $iRows - 1
    $aLine = StringSplit ( $aArray1[$i], "   ", 3 ) ; 3 = $STR_ENTIRESPLIT + $STR_NOCOUNT
    $aArray2[$i][0] = $aLine[0] ; Col 0 : Time
    $aArray2[$i][1] = $aLine[1] ; Col 1 : 16 chars
    $aArray2[$i][2] = $aLine[2] ; Col 2 : Long string
    $aArray2[$i][3] = $aLine[3] ; Col 3 : blocked/written
  Next
  ;_ArrayDisplay( $aArray2 )

  ; 3. Index based sort of $aArray2 by col 2 (long string)
  ; (Rows with duplicate strings are counted and times stored)
  Local $aIndex[$iRows]
  Local $nIndex = 0 ; Number of rows in index
  ; Sort 2D-array by column 2 (long string)
  SortArray( $aArray2, $aIndex, $nIndex )
  ;_ArrayDisplay( $aArray2 )

  ; 4. Extract results
  Local $aResults[$nIndex], $k
  For $i = 0 To $nIndex - 1
    $k = $aIndex[$i]
    $aResults[$i] = $aArray2[$k][0] & "   " & _ ; First time
                    StringLeft( $aArray2[$k][2], 10 ) & "xxxxxx" & StringMid ( $aArray2[$k][2], 17, 2 ) & "   " & _
                    StringFormat( "%3d", $aArray2[$k][5] ) & "   " & _ ; Count
                    $aArray2[$k][4] ; Last time
    ConsoleWrite( $aResults[$i] & @CRLF )
  Next
EndFunc


Func SortArray( ByRef $aItems, ByRef $aIndex, ByRef $n )
  Local $lo, $hi, $mi, $f = 0 ; $n = number of rows in index
  For $i = 0 To UBound( $aItems ) - 1
    $lo = 0
    $hi = $n - 1
    While $lo <= $hi ; Binary search
      $mi = Int( ( $lo + $hi ) / 2 )
      Switch StringCompare( $aItems[$i][2], $aItems[$aIndex[$mi]][2] )
        Case -1
          $hi = $mi - 1
        Case  1
          $lo = $mi + 1
        Case  0 ; Equal
          $f = 1
          ExitLoop
      EndSwitch
    WEnd
    If $f Then
      $aItems[$aIndex[$mi]][4] = $aItems[$i][0] ; Last time
      $aItems[$aIndex[$mi]][5] += 1 ; Count
      $f = 0
      ContinueLoop
    EndIf
    For $j = $n - 1 To $mi Step -1 ; Make space for the row number in index
      $aIndex[$j+1] = $aIndex[$j]
    Next
    $aIndex[$mi+($lo=$mi+1)] = $i ; Insert row number $i at position $mi+($lo=$mi+1) in index
    $aItems[$i][4] = $aItems[$i][0] ; Last time
    $aItems[$i][5] += 1 ; Count
    $n += 1
  Next
EndFunc

For a larger number of different strings the time used to make room in the index for new rows becomes important. A For-loop is inefficient. If a structure is used as index, an entire block of memory can be moved with a single command without the need to use a loop.

expandcollapse popup

#include <Array.au3>

Opt( "MustDeclareVars", 1 )

Example()


Func Example()
  ; 1. Read file
  Local $aArray1 = FileReadToArray( "inexample1.txt" )
  ;_ArrayDisplay( $aArray1 )

  ; 2. Create 2D-array
  Local $iRows = UBound( $aArray1 )
  Local $aArray2[$iRows][6], $aLine ; Col 4 : Last time, Col 5 : Count
  For $i = 0 To $iRows - 1
    $aLine = StringSplit ( $aArray1[$i], "   ", 3 ) ; 3 = $STR_ENTIRESPLIT + $STR_NOCOUNT
    $aArray2[$i][0] = $aLine[0] ; Col 0 : Time
    $aArray2[$i][1] = $aLine[1] ; Col 1 : 16 chars
    $aArray2[$i][2] = $aLine[2] ; Col 2 : Long string
    $aArray2[$i][3] = $aLine[3] ; Col 3 : blocked/written
  Next
  ;_ArrayDisplay( $aArray2 )

  ; 3. Index based sort of $aArray2 by col 2 (long string)
  ; (Rows with duplicate strings are counted and times stored)
  Local $tIndex = DllStructCreate( "uint[" & $iRows & "]" )
  Local $pIndex = DllStructGetPtr( $tIndex )
  Local $nIndex = 0 ; Number of rows in index
  ; Sort 2D-array by column 2 (long string)
  SortArray( $aArray2, $pIndex, $tIndex, $nIndex )
  ;_ArrayDisplay( $aArray2 )

  ; 4. Extract results
  Local $aResults[$nIndex], $k
  For $i = 0 To $nIndex - 1
    $k = DllStructGetData($tIndex,1,$i+1)
    $aResults[$i] = $aArray2[$k][0] & "   " & _ ; First time
                    StringLeft( $aArray2[$k][2], 10 ) & "xxxxxx" & StringMid ( $aArray2[$k][2], 17, 2 ) & "   " & _
                    StringFormat( "%3d", $aArray2[$k][5] ) & "   " & _ ; Count
                    $aArray2[$k][4] ; Last time
    ConsoleWrite( $aResults[$i] & @CRLF )
  Next
EndFunc


Func SortArray( ByRef $aItems, $pIndex, $tIndex, ByRef $n )
  Local $lo, $hi, $mi, $f = 0 ; $n = number of rows in index
  For $i = 0 To UBound( $aItems ) - 1
    $lo = 0
    $hi = $n - 1
    While $lo <= $hi ; Binary search
      $mi = Int( ( $lo + $hi ) / 2 )
      Switch StringCompare( $aItems[$i][2], $aItems[DllStructGetData($tIndex,1,$mi+1)][2] )
        Case -1
          $hi = $mi - 1
        Case  1
          $lo = $mi + 1
        Case  0 ; Equal
          $f = 1
          ExitLoop
      EndSwitch
    WEnd
    If $f Then
      $f = DllStructGetData($tIndex,1,$mi+1)
      $aItems[$f][4] = $aItems[$i][0] ; Last time
      $aItems[$f][5] += 1 ; Count
      $f = 0
      ContinueLoop
    EndIf
    If $n > $mi Then _ ; Make space for the row number in index
      DllCall( "kernel32.dll", "none", "RtlMoveMemory", "struct*", $pIndex+($mi+1)*4, "struct*", $pIndex+$mi*4, "ulong_ptr", ($n-$mi)*4 )
    DllStructSetData( $tIndex, 1, $i, $mi+1+($lo=$mi+1) ) ; Insert row number $i at position $mi+1+($lo=$mi+1) in index
    $aItems[$i][4] = $aItems[$i][0] ; Last time
    $aItems[$i][5] += 1 ; Count
    $n += 1
  Next
EndFunc

All three examples uses approximately the same amount of time to count the strings in inexample1.txt (the zip in post 13) with 3000 rows and 15 different strings. And about the same time as the SQLite example. But only half as much time as tst00.au3 in the zip.

(The reason why the SQLite example is fast is probably also the small number of different strings. Not much time is spend on slow INSERT commands.)

andygo, will you test the examples with your large data files? You should run each example a few times until the time measurement more or less becomes constant.

Edited July 28, 2015 by LarsJ

[andygo]

i have a 3.000 lines (1.028 differents) example and a 26.553 lines (7.605 differents). results for both with your 3 examples in seconds:

2.3 seconds (example 1 with 3.000 / 1.028)
1.0 seconds (example 2 with 3.000 / 1.028)
0.8 seconds (example 3 with 3.000 / 1.028)

82.0 seconds (example 1 with 26.553 / 7.605)
17.5 seconds (example 2 with 26.553 / 7.605)
3.7 seconds (example 3 with 26.553 / 7.605)

 

your script in your first post here:

1.3 seconds (3.000 / 1.028)

10.0 seconds (26.553 / 7.605)

 

jchd's sql script:

1.7 seconds (3.000 / 1.028)

3.8 seconds (26.553 / 7.605)

executed them all a few times to get exact results.

Edited July 29, 2015 by andygo

[LarsJ]

Thank you, very interesting.

I noticed that your script uses OnEvent mode. It may be advantageous to use the default MessageLoop mode. See here.

It may also be advantageous to use Au3Stripper with the /RM-option. See here.

[andygo]

inspired from your codes, i could make my one also a bit faster.

and theres room for more i think. but i dont want to use just yours (even if its much more complex and faster), i want to

understand it and code it by myself. learning by doing,  hope you dont see this as snooty

expandcollapse popup

#include <File.au3>
#include <StaticConstants.au3>
#include <EditConstants.au3>
#include <ComboConstants.au3>
#include <GuiConstants.au3>
#include <WindowsConstants.au3>
#include <ButtonConstants.au3>
#include <IE.au3>
#include <Timers.au3>
#include <Process.au3>
#Include <Misc.au3>
#include <Constants.au3>
Break(0)
Opt("TrayAutoPause",0)
Opt("GuiOnEventMode", 1)
if msgbox(65, "Hinweis", "Das Tool wird ggf. von einigen Schutzprogrammen negativ erkannt."& @CRLF & @CRLF &"Ich distanziere mich hiermit ausdrücklich von Schadsoftware"& @CRLF &"in jeglicher Form!"& @CRLF & @CRLF &"©2015 andygo") = 2 then exit
global $loopstop = 0, $v = 0, $what, $aRecords, $starttime

$pre = GuiCreate("EMM.Tool", 290, 150)
$laden = GUICtrlCreateButton ("Datei laden", 4, 4, 90, 20)
GUICtrlSetOnEvent($laden, "_verarbeiten")
$breakup = GUICtrlCreateCheckbox("STOP", 100, 4, 90, 20)
GUICtrlSetState($breakup, $GUI_DISABLE)
$save = GUICtrlCreateButton ("Datei speichern", 195, 4, 90, 20)
GUICtrlSetState($save, $GUI_DISABLE)
GUICtrlSetOnEvent($save, "_ausgabe")
$inf1 = GUICtrlCreateLabel("", 4, 55, 282, 20, $SS_CENTER)
$inf2 = GUICtrlCreateLabel("", 4, 75, 282, 20, $SS_CENTER)
$label = GUICtrlCreateLabel("Bereit...", 4, 110, 282, 40, $SS_CENTER)
GUICtrlSetFont($label, 16, 500)
GUISetOnEvent($GUI_EVENT_CLOSE, "_Quit2")
GUISetState(@SW_SHOW)

while 1
    Sleep(50)
wend
Func _Quit2()
    if msgbox(68,"EMM.Tool","Wirklich beenden?") = 6 then Exit
EndFunc

func addup($a, $b, $c, $d)
    GUICtrlSetData($label, $what & $v)
    if GUICtrlRead($breakup) = 1 then $loopstop = 1
endfunc
func addsave($a, $b, $c, $d)
    GUICtrlSetData($label, "Schreibe... " & $v)
endfunc

func _verarbeiten()
    Opt("GuiOnEventMode", 0)
    GUICtrlSetState($laden, $GUI_DISABLE)
    GUICtrlSetState($save, $GUI_DISABLE)
    GUICtrlSetState($breakup, $GUI_UNCHECKED)
    GUICtrlSetData($label, "Bereit...")
    GUICtrlSetData($inf1, "")
    GUICtrlSetData($inf2, "")
    $infile = FileOpenDialog("Datei", @ScriptDir & "\", "emm-log (*.*)", 1)
    if NOT @error then
        If Not _FileReadToArray($infile, $aRecords) Then
            MsgBox(4096, "Fehler", "Fehlerhafte Datei")
        Else
            GUICtrlSetData($inf1, $infile)
            GUICtrlSetData($inf2, "In: "&$aRecords[0] & " Datensätze ** " & "Out: ... Datensätze")
            $v = $aRecords[0]
            $what = "Prüfe... "
            GUICtrlSetState($breakup, $GUI_enABLE)
            global $bRecords[$aRecords[0]+1], $cRecords, $i, $x, $y, $date
            _Timer_SetTimer($pre,50,"addup")
            _ArraySort($aRecords, 0, 1)
            For $x = 1 To $aRecords[0]
                if $loopstop = 1 then exitloop
                $v -= 1
                if StringLen ($aRecords[$x]) > 0 Then
                    $bRecords[$x] = StringMid($aRecords[$x], 42, StringInStr ( $aRecords[$x], "   " , 0, -1)-42)
                endif
            Next
            $cRecords = _ArrayUnique($bRecords, 0, 1)
            $bRecords[0] = $aRecords[0]
            GUICtrlSetData($inf2, "In: "&$aRecords[0] & " Datensätze ** " & "Out: " &$cRecords[0] & " Datensätze")
            $v = $cRecords[0]
            $what = "Vergleiche... "
            global $outRecords[$cRecords[0]+2]
            $outRecords[0] = "[code]"
            $outRecords[$cRecords[0]+1] = "[/code]"
            For $x = 1 To $cRecords[0]
                if $loopstop = 1 then ExitLoop
                $dRecords = _ArrayFindAll($bRecords, $cRecords[$x])
                if  UBound($dRecords) = 1 then
                    $outRecords[$x] = StringReplace(StringReplace(StringLeft($aRecords[$dRecords[0]], 59), StringMid ( StringLeft($aRecords[$dRecords[0]], 59), 22, 20), " "), 33, "xxxxxx") &"   " & "  1"
                else
                    $outRecords[$x] = StringReplace(StringReplace(StringLeft($aRecords[$dRecords[0]], 59), StringMid ( StringLeft($aRecords[$dRecords[0]], 59), 22, 20), " "), 33, "xxxxxx") &"   " & StringTrimLeft("   ", StringLen(UBound ($dRecords))) & UBound ($dRecords) & "   " & StringLeft($aRecords[$dRecords[UBound ($dRecords)-1]], 19)
                    _ArrayDelete($bRecords, _ArrayToString ( $dRecords, ";" ))
                    _ArrayDelete($aRecords, _ArrayToString ( $dRecords, ";" ))
                endif
                $v -= 1
            next
            _Timer_KillAllTimers($pre)
            GUICtrlSetState($breakup, $GUI_DISABLE)
            if $loopstop = 1 Then
                GUICtrlSetData($label, "Abbruch durch Benutzer")
                GUICtrlSetState($breakup, $GUI_UNCHECKED)
                GUICtrlSetData($inf1, "")
                GUICtrlSetData($inf2, "")
                $loopstop = 0
            Else
                GUICtrlSetData($label, "Fertig :-)")
                GUICtrlSetState($save, $GUI_enABLE)
            endif
        endif
    endif
    GUICtrlSetState($laden, $GUI_enABLE)
    Opt("GuiOnEventMode", 1)
endfunc

func _ausgabe()
    GUICtrlSetState($save, $GUI_DISABLE)
    GUICtrlSetState($laden, $GUI_disABLE)
    Local $file = FileSaveDialog("Datei", @ScriptDir, "emm-log (*.txt)", 2+16)
    If @error Then
        GUICtrlSetData($label, "Fertig :-)")
    Else
        if StringInStr(StringRight($file, 4), ".") = 0 then $file = $file & ".txt"
        If $file = -1 Then
            GUICtrlSetData($label, "Fertig :-)")
            FileClose($file)
            MsgBox(0, "Fehler", "Speichern nicht möglich.")
        else
            _Timer_SetTimer($pre,50,"addsave")
            GUICtrlSetState($breakup, $GUI_enABLE)
            $v = UBound ($outRecords)
            _FileWriteFromArray($file, $outRecords)
            _Timer_KillAllTimers($pre)
            GUICtrlSetState($breakup, $GUI_DISABLE)
            if $loopstop = 1 Then
                GUICtrlSetData($label, "Abbruch durch Benutzer")
                GUICtrlSetState($breakup, $GUI_UNCHECKED)
                $loopstop = 0
            Else
                GUICtrlSetData($label, "Datei gespeichert")
            endif
        Endif
    endif
    GUICtrlSetState($save, $GUI_enABLE)
    GUICtrlSetState($laden, $GUI_enABLE)
endfunc

 

[LarsJ]

Not at all.

[jvanegmond]

@@andygo well done! Much improved!

[andygo]

Thank you, very interesting.

I noticed that your script uses OnEvent mode. It may be advantageous to use the default MessageLoop mode. See here.

It may also be advantageous to use Au3Stripper with the /RM-option. See here.

i will try those hints and see if its more fast ?

[andygo]

hello, i changed variable-names to only two.digit long. this dont change the speed.

 

default messageloop test follows

[LarsJ]

Last week, I was using the Dictionary object for some code. Immediately I remembered this thread. I don't know why I forgot the Dictionary object in the first place. It appears to be an obvious method to use.

Here is some code based on the Dictionary object: 

expandcollapse popup

#include <Array.au3>

Opt( "MustDeclareVars", 1 )

Example()


Func Example()
  ; 1. Read file
  Local $aArray1 = FileReadToArray( "inexample1.txt" )
  ;_ArrayDisplay( $aArray1 )

  ; 2. Create 2D-array
  Local $iRows = UBound( $aArray1 )
  Local $aArray2[$iRows][6], $aLine ; Col 4 : Last time, Col 5 : Count
  For $i = 0 To $iRows - 1
    $aLine = StringSplit ( $aArray1[$i], "   ", 3 ) ; 3 = $STR_ENTIRESPLIT + $STR_NOCOUNT
    $aArray2[$i][0] = $aLine[0] ; Col 0 : Time
    $aArray2[$i][1] = $aLine[1] ; Col 1 : 16 chars
    $aArray2[$i][2] = $aLine[2] ; Col 2 : Long string
    $aArray2[$i][3] = $aLine[3] ; Col 3 : blocked/written
  Next
  ;_ArrayDisplay( $aArray2 )

  ; 3. Count strings in $aArray2 based on Dictionary object
  ; (Rows with duplicate strings are counted and times stored)
  Local $oDict = ObjCreate( "Scripting.Dictionary" )
  CountArray( $aArray2, $oDict )
  ;_ArrayDisplay( $aArray2 )

  ; 4. Extract results
  Local $nDict = $oDict.Count()
  Local $aResults[$nDict], $aItems = $oDict.Items(), $k
  For $i = 0 To $nDict - 1
    $k = $aItems[$i]
    $aResults[$i] = $aArray2[$k][0] & "   " & _ ; First time
                    StringLeft( $aArray2[$k][2], 10 ) & "xxxxxx" & StringMid ( $aArray2[$k][2], 17, 2 ) & "   " & _
                    StringFormat( "%3d", $aArray2[$k][5] ) & "   " & _ ; Count
                    $aArray2[$k][4] ; Last time
    ConsoleWrite( $aResults[$i] & @CRLF )
  Next
EndFunc


Func CountArray( ByRef $aItems, ByRef $oDict )
  Local $idx
  For $i = 0 To UBound( $aItems ) - 1
    $idx = $oDict( $aItems[$i][2] )
    If $idx Then
      $aItems[$idx][4] = $aItems[$i][0] ; Last time
      $aItems[$idx][5] += 1 ; Count
    Else
      $oDict( $aItems[$i][2] ) = $i
      $aItems[$i][4] = $aItems[$i][0] ; Last time
      $aItems[$i][5] += 1 ; Count
    EndIf
  Next
EndFunc

If you compare the function CountArray with the function SortArray in last code box in post 25, CountArray seems to be very simple. This does not necessarily mean that the code is also faster. But there is a possibility. It would be interesting to see the results of your two test files with this code. Remember to run the code a few times.

[andygo]

Hello,

 

a long time passed (had to do some things in real life)   and since a few days ago i focused again a little more intense on this script.

the Last code from LarsJ (postet on 10.August-2015) is realy nice because it is so simple and effective, that i as a non-pro also could unterstand it.

i implemented it into my gui-based tool, it is now 4 times faster as my last version. realy nice!!!!!!

i hope it is ok for you that i use your code for the main internal solving / sorting part.

[LarsJ]

Interesting. You can use the code. That's why I have posted it.

[andygo]

and: when version's before stuck with memory allocate error because input file was too big, this here eat's them all 😁 a friend of mine did an extreme test with 6.3 million line input... tooks a few moments but it finish with no errors: