speed up INIREAD

[mohdumar]

My script reads values from 50000 keys from a .ini file (in a loop) and compares it with a user-entered value, then displays the value of the matching key to the user.

Func READINI()
Local $startpos = 0
Local $loop = 50000
For $i = $startpos To $loop Step 1
Local $var = IniRead("database.ini", "db", $i, "Not Found")
If $var = $some_user_entered_value Then  ; equals it
MsgBox(4096, "Found", $var & " was found at " & $i) ;was found!
ExitLoop 1 ;no use for further compare
EndIf
GUICtrlSetData($progress_bar, $i / $loop * 100) ; set progress bar value
Next
GUICtrlSetData($progress_bar, 0)  ; set it back to zero
EndFunc

Is there anyway to speed it up? does high process priority work?

I also tried from this forum but they were slower than builtin disk I/O functions........

[JohnOne]

I don't think ini is very practical for speed with 50,000 records, I suggest you think about using a dadabase

or maybe even xml.

I'm not very clever at all with either, so cant help much other to say, they will both improve with regards

to speed you get now with ini files.

[jchd]

Switch to an SQLite database, it's a perfect fit for such simple application. Start with SQLite doc found in helpfile then chime in again if you're stuck.

[MilesAhead]

The fact that you use an .ini file leads me to believe you want a string key for a value. In that case, if 50000 is really the outer limit, you may find Scripting.Dictionary is sufficient. Not as robust as a database but for quick and dirty it's a lot easier to use. The data file could be a flat file alternating key value lines, if the keys and data can be expressed as a single line of text.

You could read the entire data file with _FileReadToArray(). Test the number of items read > 0 and not odd( with BitAND($array[0],1) ). Since these are key/data pairs an odd number of lines would indicate and error.

50000 may be stretching it a bit. I'm not sure if you can find any docs how Scripting.Dictionary relates to system memory or whatnot. MS seems to be kind of mum about it. I just ran a test on a 2 GB system, inside SciTE, loading 50000 lines of dummy data with a stringized number as key and it didn't balk.

Depends on what you are writing. If it's for money transfer between banks then you are better off using an engine. But if it's for a quickie user utility AssocArray may be enough.

These 2 functions are all you need. Also when getting an item using the key make sure to test with Exists method. As in If $array.Exists("key") then blah blah or else if you read with a key that's not there, the dictionary will create one with empty string as data. My AssocArray uses string key with case insensitive compare. But of course you can just write your own wrapper with a different CompareMode setting.

;use Scripting.Dictionary object for simple associative arrays
Func _AssocArray()
    Local $aArray = ObjCreate("Scripting.Dictionary")
    If @error Then
        Return SetError(1, 0, 0)
    EndIf
    $aArray.CompareMode = 1
    Return $aArray
EndFunc   ;==>_AssocArray

Func _AssocArrayDestroy(ByRef $aArray)
    If Not IsObj($aArray) Then
        Return False
    EndIf
    $aArray.RemoveAll()
    $aArray = 0
    Return True
EndFunc   ;==>_AssocArrayDestroy

Here's a simple example for creating an array of colors with the names as key.

expandcollapse popup

Func _ColorList()
    Local $colorList = _AssocArray()
    If @error Then
        Return SetError(1, 0, 0)
    EndIf
    $colorList("AntiqueWhite") = 0xFAEBD7
    $colorList("Black") = 0x000000
    $colorList("Blue") = 0x0000FF
    $colorList("Brown") = 0xA52A2A
    $colorList("CadetBlue") = 0x5F9EA0
    $colorList("Chocolate") = 0xD2691E
    $colorList("Coral") = 0xFF7F50
    $colorList("CornflowerBlue") = 0x6495ED
    $colorList("DarkBlue") = 0x00008B
    $colorList("DarkCyan") = 0x008B8B
    $colorList("DodgerBlue") = 0x1E90FF
    $colorList("ForestGreen") = 0x228B22
    $colorList("Gold") = 0xFFD700
    $colorList("Gray") = 0x808080
    $colorList("Green") = 0x008000
    $colorList("HotPink") = 0xFF69B4
    $colorList("LightGreen") = 0x90EE90
    $colorList("LightPink") = 0xFFB6C1
    $colorList("LightSeaGreen") = 0x20B2AA
    $colorList("Lime") = 0x00FF00
    $colorList("Magenta") = 0xFF00FF
    $colorList("Maroon") = 0xB03060
    $colorList("MediumTurquoise") = 0x48D1CC
    $colorList("MediumVioletRed") = 0xC71585
    $colorList("MistyRose") = 0xFFE4E1
    $colorList("Navy") = 0x000080
    $colorList("Olive") = 0x808000
    $colorList("Orchid") = 0xDA70D6
    $colorList("PaleGreen") = 0x98FB98
    $colorList("PaleVioletRed") = 0xDB7093
    $colorList("Pink") = 0xFFC0CB
    $colorList("Plum") = 0xDDA0DD
    $colorList("Purple") = 0x800080
    $colorList("Red") = 0xFF0000
    $colorList("RoyalBlue") = 0x4169E1
    $colorList("Sienna") = 0x00A0522D
    $colorList("Silver") = 0xC0C0C0
    $colorList("SkyBlue") = 0x87CEEB
    $colorList("SteelBlue") = 0x004682B4
    $colorList("Tan") = 0xD2B48C
    $colorList("Teal") = 0x008080
    $colorList("Violet") = 0xEE82EE
    $colorList("Wheat") = 0xF5DEB3
    $colorList("White") = 0xFFFFFF
    $colorList("Yellow") = 0xFFFF00
    Return $colorList
EndFunc   ;==>_ColorList

Func _ColorListDestroy(ByRef $cList)
    Return _AssocArrayDestroy($cList)
EndFunc   ;==>_ColorListDestroy

Edited January 17, 2012 by MilesAhead

[Spiff59]

Here's an "old-fashioned" way of loading the data.

You could do an _ArraySearch() and limit the start and end lines by using values from the $aSectionIndex[] array.

It has the advantage of being simple with low overhead and infinitely faster than 50,000 separate I/O opens, reads, and closes (via IniRead). It would not be as fast as other suggestions that have been made.

#include <Array.au3>
#include <File.au3>

Global $aRawfile, $iSections = 0, $x = 0
_FileReadToArray(@ScriptDir & "test.ini", $aRawfile)
Global $aSectionIndex[$aRawfile[0] + 1][3] ; section name, start line, end line
Global $aFile[$aRawfile[0] + 1][2] ; section or key name, key value (optional)

For $y = 1 to $aRawfile[0]
  If $aRawfile[$y] Then ; ignore blanks
    $x += 1
    If StringLeft($aRawfile[$y], 1) = "[" Then ; section header
     If $iSections > 0 Then $aSectionIndex[$iSections][2] = $x - 1
       $aFile[$x][0] = $aRawfile[$y]
       $iSections += 1
       $aSectionIndex[$iSections][0] = $aRawfile[$y]
       $aSectionIndex[$iSections][1] = $x + 1
    Else
       $temp = StringSplit($aRawfile[$y], "=")
       $aFile[$x][0] = $temp[1]
       $aFile[$x][1] = $temp[2]
    EndIf
  EndIf
Next
$aSectionIndex[$iSections][2] = $x
ReDim $aSectionIndex[$iSections + 1][3]
$aSectionIndex[0][0] = $iSections
ReDim $aFile[$x + 1][2]
$aFile[0][0] = $x

_ArrayDisplay($aRawfile, "Rawfile")
_ArrayDisplay($aSectionIndex, "Section Index")
_ArrayDisplay($aFile)

This assumes a "clean" ini file and has no edits to discard comments, strip unwanted whitespace, or handle keys with no value.

typo

Edited January 17, 2012 by Spiff59

[Bowmore]

@MilesAhead

Just for information. I've used scriptingDictionary without any problems with 4 million key data pairs.

[jchd]

If it will definitely be read-only data, then you may perhaps use a scripting dictionary, provided your application copes with it. But if ever you have to update it in a multi-process use case, I cast strong doubts about robustness. Also a DB will allow for partial search key, etc.

[MilesAhead]

@MilesAhead

Just for information. I've used scriptingDictionary without any problems with 4 million key data pairs.

Thanks for the info. I don't know what the OP's app is exactly. But if it's just key/data pairs I don't see the need for an engine unless there's some low latency requirement. Processing 200 records a minute is one thing. If a user is typing in a string and hitting Enter to get some value, then simpler is often better. After all C++ has map STL. I used it for flat file key/data lookup for programs I wrote in 2001 and I never got user feedback about the program being unable to fetch the data. And that was for a shell extension. Never hung up Explorer that I ever heard.

The nice thing about the Dictionary is as long as you check with Exists, if you make a mistake it's usually a typo. Nice and neat.

$val = $array("Key")

simple.

Edited January 18, 2012 by MilesAhead

[mohdumar]

I also think that an SQLite database would be better suited for this situation. If you have to use the INI file though, I would go with a combination of FileRead and StringRegExp instead of the INI functions.

If you pass the path to your INI file and the search value into this function, it will return the key that corresponds to the value. The search value is case sensitive.

ConsoleWrite(_ReadINI(@ScriptDir & "Test.ini", "C") & @CRLF)

Func _ReadINI($s_IniPath, $s_Value)
    Local $a_RegEx = StringRegExp(StringRegExpReplace(FileRead($s_IniPath), "(rn|r)", @LF), "n(.*?)=" & $s_Value, 3)
    If Not IsArray($a_RegEx) Then Return SetError(1,0,0)
    Return $a_RegEx[0]
EndFunc

This was my Test.ini.

[Test]
Test1=A
Test2=B
Test3=C
Test4=D
Test5=E
Test6=F
Test7=G
Test8=H
Test9=I

this solution is really fast and easy.. by the way my script is really simple ..thx