Read file to (huge) array

[Geir1983]

Hi

After the new 3.3.10 discontiuation of the PluginOpen i need to change my script a bit. I guess i can just use dllcall, but would be cool to just convert my dll into autoit.

What i need to do is read a big file containing an array of 32487834 32bit integers and place them in an array. It seems autoit only allows 16 million elements in an array so i was thinking splitting it over 2 or more arrays. My problem is how to do this, is there a function like memcopy or something i could use?

What i have so far:

; Create 32 bit int array
; Local $HR[32487834] Too big! 
Local $HR1[10829278]
Local $HR2[10829278]
Local $HR3[10829278]

For $idx=0 to UBound($HR1)-1
    Int($HR1[$idx], 1)
Next
$HR2 = $HR1
$HR3 = $HR1

; Read data from disk ¨123mb
$Data = FileReadToArray(FileOpen(@ScriptDir & "\HandRanks.dat" , $FO_BINARY)) ;

;Now insert $Data to $HR arrays

For reference this is the DLL:

int HR[32487834];

int InitData()
{
    memset(HR, 0, sizeof(HR));
    FILE * fin = fopen(“HANDRANKS.DAT”, “rb”);
    size_t bytesread = fread(HR, sizeof(HR), 1, fin);
    fclose(fin);
}

int GetHandValue(int* pCards)
{
    int p = HR[53 + *pCards++];
    p = HR[p + *pCards++];
    p = HR[p + *pCards++];
    p = HR[p + *pCards++];
    p = HR[p + *pCards++];
    p = HR[p + *pCards++];
    return HR[p + *pCards++];
}

Edited February 28, 2014 by Geir1983

[JohnOne]

You might be better off using DllStructs, don't thing they have a limit.

for example. if your file is structured like so....

Test.txt

123
456
789
123
456
789
123
456
789
123
456
789

Then...

$strStruct = "int[32487834]"

$intStruct = DllStructCreate($strStruct)
$sFile = "Test.txt"

_FileToStruct($sFile, $intStruct)

For $i = 1 To 20
    ConsoleWrite(DllStructGetData($intStruct, 1, $i) & @LF)
Next

Func _FileToStruct($file, ByRef $struct)

    If Not IsDllStruct($struct) Then
        Exit MsgBox(0, "Error", "Not struct")
    EndIf

    $size = FileGetSize($file)
    $open = FileOpen($file)
    $index = 1

    Do
        $line = FileReadLine($open)
        DllStructSetData($struct, 1, Int($line), $index)
        $index += 1
    Until FileGetPos($open) >= $size

    FileClose($open)

EndFunc   ;==>_FileToStruct

[Geir1983]

Thanks, this worked

My file is all binary so had to do some small changes to your example, reading 4 bytes at a time instead of line by line and then placing it into the array. However, it takes alot of time to read all the elements, about 130 seconds. Reading just the file itselft takes about 3-4 seconds. Is there a smarter way to populate the dllstruct instead of one element at the time?

Reading the array is not super fast, but I can do 140k quries (pluss some additional code) in about 2 seconds so im pretty happy with that. It is pretty close to what I had with PluginOpen (maybe even faster, but comparing on different machines).

$strStruct = "int[32487834]"

Global $intStruct = DllStructCreate($strStruct)
$sFile = @scriptdir & "\HandRanks.dat"

_FileToStruct($sFile, $intStruct)

Func _FileToStruct($file, ByRef $struct)

    If Not IsDllStruct($struct) Then
        Exit MsgBox(0, "Error", "Not struct")
    EndIf

    $size = FileGetSize($file)
    $open = FileOpen($file, $FO_BINARY)
    $index = 0
    $begin = TimerInit()
    Do

        $line = FileRead($open, 4)
        FileSetPos($line, $index*4, $FILE_BEGIN)
        DllStructSetData($struct, 1, Int($line), $index)
        $index += 1
    Until FileGetPos($open) >= $size

    $dif = TimerDiff($begin)
    ConsoleWrite("File placed in array in (ms): " & $dif & ", Number of elements read: " & $index & @CRLF)
    FileClose($open)

EndFunc   ;==>_FileToStruct

[JohnOne]

Is every 4 bytes you read in binary mode, exactly 1 integer?

[Geir1983]

yes, 32 bit integer always occupy 4 bytes in memory (binary mode).

Edit: This means my file is identical to the way the array is stored as a variable in memory. It will always have allocted the same amount of bytes (elements * size) in memory. (This gives also the same filesize no matter the content in the array, all elements with zero value still occupies the same 123mb (32487834 elements * 4 bytes). Some languages have functions for copying the memory contents directly to another area, but have not seen this in Autoit, it also seems different to this DllStruct.

Edited March 1, 2014 by Geir1983

[JohnOne]

Then have you thought of just accessing the data directly from file in binary mode employing the filesetpos function?

[Geir1983]

Hmm, that might work, ill test it.

I tried it now, but it is alot slower to read the data from the file than from the dllstruct. It is also alot more complex to calculate the address to read.with 140k queries (this is the most important part)timer went from 2 seconds to 8 seconds so its no good.

Is it possible to read all of the file to a variable and access it like fileread does? (with an offset and size?)

Edit: Actually i could remove some statements and improve the fileread method to 2.9s by removing INT() and redundant multiplying (from 1.9 when using the dllstruct). Maybe you have some more tips on optimizing?

Edit2: Nvm, I think i need the INT().. when i put it back in there im back to 8,5 sec

DllStruct:

Func AnalyzeHand(Byref $Card1)
    $p = DllStructGetData($intStruct, 1, 53 + $Card1[0])
    $p = DllStructGetData($intStruct, 1, $p + $Card1[1])
    $p = DllStructGetData($intStruct, 1, $p + $Card1[2])
    $p = DllStructGetData($intStruct, 1, $p + $Card1[3])
    $p = DllStructGetData($intStruct, 1, $p + $Card1[4])
    $p = DllStructGetData($intStruct, 1, $p + $Card1[5])
    $p = DllStructGetData($intStruct, 1, $p + $Card1[6])
    Return DllStructGetData($intStruct, 1, $p)
EndFunc

FileRead:

Func AnalyzeHand(Byref $Card1)
    FileSetPos($open, (53 + $Card1[0])*4, $FILE_BEGIN)
    $p = Int(FileRead($open, 4))
    FileSetPos($open, ($p+$Card1[1])*4, $FILE_BEGIN)
    $p = Int(FileRead($open, 4))
    FileSetPos($open, ($p+$Card1[2])*4, $FILE_BEGIN)
    $p = Int(FileRead($open, 4))
    FileSetPos($open, ($p+$Card1[3])*4, $FILE_BEGIN)
    $p = Int(FileRead($open, 4))
    FileSetPos($open, ($p+$Card1[4])*4, $FILE_BEGIN)
    $p = Int(FileRead($open, 4))
    FileSetPos($open, ($p+$Card1[5])*4, $FILE_BEGIN)
    $p = Int(FileRead($open, 4))
    FileSetPos($open, ($p+$Card1[6])*4, $FILE_BEGIN)
    Return Int(FileRead($open, 4))
EndFunc

Edited March 1, 2014 by Geir1983

[guinness]

The Dll*() functions should only be used for APIs and not internally in AutoIt.

[Geir1983]

Then do you have any other suggestion for the 32487834 element array?

[guinness]

SQL database?

[JohnOne]

Take no notice of guinness, he's an old fuddy duddy.

Have you considered converting your dat file to SQLite db?

That's pretty quick.

[guinness]

Take no notice of guinness, he's an old fuddy duddy.

Have you considered converting your dat file to SQLite db?

That's pretty quick.

[JohnOne]

Meh, she's an old codger too.

You're right about the db though.

[Geir1983]

Hmm, i guess it could be done, but i doubt it is faster to query a SQLite db than an DLLStruct from autoit? It would eliminate the long time needed to populate the array at startup though..

[JohnOne]

I don't know, they're pretty swift those SQLite db's like.

[jchd]

Looks like the need to evaluate card hands (poker?).

Even assuming this is legit, an array won't do since AutoIt arrays are limited to 2²⁴ entries.

Again assuming legit application, DllStruct is the fastest way here.

[JohnOne]

Indeed, but populating that struct probably quicker from a db than flat text file.

EDIT:

I'm sure this his for his own created poker game.

Edited March 1, 2014 by JohnOne

[LarsJ]

Geir1983, have you tried something like this:

 

$hFile = FileOpen( "file", 16 ) ; 16 = binary mode
$sBinData = FileRead( $hFile )
$iBytes = @extended
FileClose( $hFile )

$tInts = DllStructCreate( "int[" & $iBytes/4 & "]" )
$pInts = DllStructGetPtr( $tInts )
$tBytes = DllStructCreate( "byte[" & $iBytes & "]", $pInts )
DllStructSetData( $tBytes, 1, $sBinData )

Now should $tInts contain your integers.

[Geir1983]

Lars, that is exactly the magic i was looking for, array populated in no time. Thanks

John, i did a test populating the DllStruct (one by one) without FileRead operations (just inserted a constant value in every element), it still took nearly a minute. Thanks for all your help!   

Oh and btw this is legit:

'?do=embed' frameborder='0' data-embedContent>>

[JohnOne]

Yeah that's a cool tip from Lars, which just helped me to get past a problem myself.