Code Gotcha! + Bonus Fast, Low Memory StringInFile()

[corz]

Running this function here (in this simple test script) takes 16 seconds. The exact same function running inside my program takes over 22 seconds! WT*?!? This bamboozled me for a bit.

After some trial and error I discovered what was making the difference and thought I might share this nugget which I will keep in mind for future intensive operations and if possible, set/reset at each side of said operation!

In short, one, or rather two (each has an effect) common AutoIt script settings will cause some intensive code operations to run more slowly. The example searches in millions of lines of text from a file.

I wasn't quite sure where to post this, so I thought I'd leave in the actual function and put it in example scripts as a bonus! It's a fast, low-memory way to find a string inside any size of file.
 

expandcollapse popup

; First, nice and fast..

;AutoItSetOption("TrayOnEventMode", 0)
;AutoItSetOption("GUIOnEventMode", 0)

;;---------- RUN x64 ----------
;;Searching for : ->one million a characters.txt
;;Time =>16.23 seconds
;;.\StringInFileSimpleTest.au3 (46) : ==> $found_array:
;;[0]   [0] = 100084        [1] = This is a line which contains the string "one million a characters.txt"
;;[1]   [0] = 100085        [1] = This is another line which contains the string "one million a characters.txt"




; uncomment the following two lines for almost 40% slower times!

;AutoItSetOption("TrayOnEventMode", 1)
;AutoItSetOption("GUIOnEventMode", 1)

;;---------- RUN x64 ----------
;;Searching for : ->one million a characters.txt
;;Time =>22.63 seconds
;;.\StringInFileSimpleTest.au3 (46) : ==> $found_array:
;;[0]   [0] = 100084        [1] = This is a line which contains the string "one million a characters.txt"
;;[1]   [0] = 100085        [1] = This is another line which contains the string "one million a characters.txt"



; init..
$string = "one million a characters.txt"        ; a string to search for (this happens to be a file name)
$test_file = "B:\Test\2,040,000 lines.data"     ; 2,040,000 longish (ave' 70 chr) lines of text in this file
$begin_time = TimerInit()



;; this would fix it, if your program UI design allows..
;local $old_tray_mode = AutoItSetOption("TrayOnEventMode", 0)
;local $old_menu_mode = AutoItSetOption("GUIOnEventMode", 0)

$found_array = StringInFileSimple($test_file, $string, default, true) ; (must search to end of file)
ConsoleWrite("Time =>" & Round(TimerDiff($begin_time)/1000, 2) & " seconds")

;; revert back to previous settings..
;AutoItSetOption("TrayOnEventMode", $old_tray_mode)
;AutoItSetOption("GUIOnEventMode", $old_menu_mode)


;if $do_debug = $ON then PrintArray($found_array,  "$found_array", 0, @ScriptLineNumber);debug





;
; StringInFileSimple()
;
; A low memory way to find a string inside any size of file.
; This is also fairly fast and handles UTFx strings & files.
;
; In its simplest form, you send a file path and a string and it returns the
; line number where the string was first found, or false if not found in the file.
;
; You can force this function to return an array by passing the third param as
; true. It is a 2D array with one main element which looks like this:
;
;        [0][0] = {int} line number of matching line, [0][1] = {string} entire line containing found string
;
; This is handy when you need to return the entire found line.
;
; It will also return an array if you set the fourth parameter, return_all=true,
; which returns ALL the matches found in the file, in the same two-dimensional
; array format as before, elements starting at 0, e.g.:
;
;
;        [0][0] = line number of matching line,        [0][1] = entire line containing found string
;        [1][0] = line number of 2nd matching line,    [1][1] = entire line containing found string
;        [2][0] = line number of 3rd matching line,    [2][1] = entire line containing found string
;        [3][0] = etc.
;
; This was designed to search within a file for relative or absolute file paths
; using a single search operation, but works great for any string.
;
func StringInFileSimple($file_path, $string_to_find, $return_array=false, $return_all=false)

    local $line_number = false
    local $file_handle = FileOpen($file_path, 0) ; or make a param out of the "0", or omit.

    if $return_all then $return_array = true
    if $return_array then
        local $array_count = 1
        local $ret_array[$array_count][2]
    endif

    local $this_line

    while true

        local $found = false

        $this_line = FileReadLine($file_handle)
        if @error then
            $line_number = false
            exitloop
        endif

        $line_number += 1

        if StringInStr($this_line, $string_to_find) then

            $found = true

            if $return_array then
                $ret_array[$array_count-1][0] = $line_number
                $ret_array[$array_count-1][1] = $this_line
            endif

            if $return_all then
                $array_count += 1
                redim $ret_array[$array_count][2]
            else
                exitloop
            endif

        endif
    wend

    FileClose($file_handle)

    if $return_array then
        if $return_all then redim $ret_array[$array_count-1][2] ; chop off last (empty) element
        return $ret_array
    endif

    if $found then
        return $line_number
    endif

    return false

endfunc

; I also have a StringInFile() which allows you to optionally pass an array of
; search terms. But even asking if $x = true on each iteration of a 1,000,000
; lines can cost you a second, so this is the "simple" version. All speed-up
; tips for this code warmly welcomed!

So there you have it!

;o) Cor

Edited January 1, 2015 by corz

[corz]

Hey, I didn't expect a Nobel for this, but BEJEEEZUS!FuxnFuk! Not only one but two pretty startling AutoIt discoveries (and workarounds) but a free super-useful, super-fast function to boot! Six months, nada.

You are a dry bunch!

;o) Cor

[Xandy]

What can I say?  I have a limited intelligence and I've used it in a particular direction.

I try not to do exhaustive searches for strings.  There are reasons to do them, but don't normally write that type of Application.  We're all busy.  I'm busy, you're hopefully busy.  Actually I'll try to remember this if I find a use for it.  Thank you for sharing.

[JohnOne]

If it's comments you're after, then it's quite standard code and nothing about it is either startling or new.

Of course I'm delighted you like it.

[guinness]

There is a far better approach than this and before you ask, no I won't.

[corz]

JohnOne, the last time I looked, the functions on the forum used gobs of memory to do the exact same thing. If I missed another, better method, feel free to point me in that direction.

And I've not come across any mention of these gotchas before, ergo, they are new.

guinness, why not?

 

;o) Cor

Edited July 23, 2015 by corz

[JohnOne]

It's a function for searching through a file line by line, there are many example of that in the forum.

No idea what you mean by a gotcha, unless you are again implying that your code is super special.

It's not, it's bog standard code.

[Danyfirex]

There is a faster option.  rutine in assembly.

 

Saludos

Edited July 23, 2015 by Danyfirex

[KaFu]

I guess nobody really saw what you were referring to... interesting discovery ...

Local $test, $timer
For $z = 1 To 10
    For $y = 0 To 1
        For $x = 0 To 1
            AutoItSetOption("TrayOnEventMode", $y)
            AutoItSetOption("GUIOnEventMode", $x)
            $timer = TimerInit()
            For $i = 0 To 100000
                $test = 100 ^ 2
            Next
            ConsoleWrite($y & "x" & $x & @TAB & TimerDiff($timer) & @CRLF)
        Next
    Next
    ConsoleWrite(@CRLF)
Next

 

Edited July 23, 2015 by KaFu

[MuffinMan]

Wow, very interesting indeed.  I need to go check a few of my scripts...

[Melba23]

corz,

Perhaps if you had bothered to explain your "discovery" in the OP, you might have got more responses. Now KaFu has pointed it out I must say that I find it very interesting and makes me glad I use MessageLoop mode almost exclusively.

M23

[JohnOne]

So what this means is that if you set either TrayOnEventMode or GUIOnEventMode you're loops will be slower.

[BrewManNH]

I think that the difference in speed of the loops is minimal, but measurable. I've also noticed that how you set the OnEvent mode alters the loop differences, if you change the two inner loops ($x and $y) to 1 to 0 Step -1 sometimes OnEvent mode goes faster than with it turned off.

AutoIt has a problem with making accurate timing measurements of script execution below a certain point, the order in which you do things can show differences where there aren't any or they're a lot smaller than if you run the exact same code but move things around in it.