Pre-compile include-file into an autoit readable binary?

[yyywww]

I have an au3 include file which contains just one big 2D array with 3 elements in each line, and around 120 000 lines in total. When I include this file in my script and load it, it takes roughly 4 minutes until it's been fully loaded into memory and my main script starts.

Is there any way I could pre-compile, or somehow turn this file (11 MB) containing the array into a binary which is then loaded (faster I assume since it would be in binary format?) by my main script?

[Danp2]

Can you explain the context in which you use this array? Perhaps it's time to consider another storage option, such as a database.

[jchd]

Can you post a short sample of how you declare and  initialize your array?

4 minutes is really a huge time for loading 11Mb of data.  This example, including computed init data, runs in under 2s from Scite on my very, really very slow PC:

Local $a[120000][3]

For $i = 0 To UBound($a) - 1
    $a[$i][0] = $i * $i
    $a[$i][1] = ChrW($i + 256)
    $a[$i][2] = Random(0,1)
Next

To answer your last question, AutoIt doesn't offer a way to build object files, like true compilers do.

[Thopaga]

Maybe, but I do not know.

 

[junkew]

Sounds really long to read a text file and split it into columns

1. Read the full textfile in a variable
2. Split it with a regex into an array

example like below based on fixed columns of size 2,2,3 and a CRLF at end of each line

#include <StringConstants.au3>
#include <array.au3>

$strData="ABCDEFG" & @CRLF
$strData&="HIJKLMN" & @CRLF
$strData&="OPQRSTU" & @CRLF
$strData&="VWXYZAB" & @CRLF

;~ $strData=fileread("data.txt")
consolewrite($strdata)

;~ Split fixed line
$aArray=StringRegExp($strData,"(.{2})(.{2})(.{3})(\R)",$STR_REGEXPARRAYGLOBALFullMATCH)
consolewrite(@error&@CRLF)

Local $aMatch = 0

For $iRow = 0 To UBound($aArray) - 1
    $aMatch = $aArray[$iRow]

    For $iCol = 1 To UBound($aMatch) - 1
       consolewrite($iRow & ',' & $icol & ":" & $aMatch[$icol] &@CRLF)
    Next
Next

 

[jchd]

17 hours ago, jchd said:

Can you post a short sample of how you declare and  initialize your array?

Round 2

[yyywww]

20 hours ago, jchd said:

Can you post a short sample of how you declare and  initialize your array?

In my main script I do an include:

#include "_bigarray.au3"

 

And the contents of this file are as follows:

; _bigarray.au3
    #include-once
    global $aArray1 = [ _
['String1', 'String2', 0.71711], _
['String1', 'String2', 0.70803], _
['String1', 'String2', 0.70042], _
; ... and so on, around 10000 lines more ...
['String1', 'String2', 0.44563]] ; last line
    ; i have around 12 of the above arrays

 

I tried out your sample code and it only took me about one second.

[yyywww]

@Danp2 Basically I am using the data stored in the array like a database. But I would really like to avoid using a database to keep my script free of dependencies.

 

@Thopaga @junkew

Thanks, I think this could be a good solution for what I am trying to do.

I tried junkew's script for a text file with 120k sample lines and it took only 1 second to process the file and turn it into an array, which would be way faster than how I am doing it right now with an include.

Edited August 13, 2019 by yyywww

[jchd]

When the array dimension(s) are not explicitely specified, the interpretor has to parse the whole init part in full detail to determine the actual dimension(s), before actually loading data into memory.

For instance, this:

Global $aArray1 = [ _
    ['String1', 'String2', 0.71711], _
    ['String1', 'String2', 0.70803], _
    ['String1', 'String2', 0.44563]] ; last line

results in:

Array[3][3]
      [0][0] => String (7)         'String1'
      [0][1] => String (7)         'String2'
      [0][2] => Double             0.71711
      [1][0] => String (7)         'String1'
      [1][1] => String (7)         'String2'
      [1][2] => Double             0.70803
      [2][0] => String (7)         'String1'
      [2][1] => String (7)         'String2'
      [2][2] => Double             0.44563

But this:

Global $aArray2 = [ _
    ['String1', 'String2', 0.71711], _
    ['String1', 'String2', 0.70803], _
    ['String1', 'String2', 0.44563, "Gotcha", "bigger than you thought", "and much bigger really!"]] ; last line

yields:

Array[3][6]
      [0][0] => String (7)         'String1'
      [0][1] => String (7)         'String2'
      [0][2] => Double             0.71711
      [0][3] => String (0)         ''
      [0][4] => String (0)         ''
      [0][5] => String (0)         ''
      [1][0] => String (7)         'String1'
      [1][1] => String (7)         'String2'
      [1][2] => Double             0.70803
      [1][3] => String (0)         ''
      [1][4] => String (0)         ''
      [1][5] => String (0)         ''
      [2][0] => String (7)         'String1'
      [2][1] => String (7)         'String2'
      [2][2] => Double             0.44563
      [2][3] => String (6)         'Gotcha'
      [2][4] => String (23)        'bigger than you thought'
      [2][5] => String (23)        'and much bigger really!'

That implies much more work at init time than this:

Global $aArray2[][6] = [ _
    ['String1', 'String2', 0.71711], _
    ['String1', 'String2', 0.70803], _
    ['String1', 'String2', 0.44563, "Gotcha", "bigger than you thought", "and much bigger really!"]] ; last line

because by finding the column dimension, the interpreter only has to count rows and can store data on the fly.
Better still, provide the actual dimensions at declaration [10000][2], since values are known in advance or easily counted.

This slowdown isn't noticeable when arrays are small but can waste enough cycles to make a difference in your use case.

 

[yyywww]

@jchd Thank you, that was enlightening to me.

But it is strange, when I change each Array to its specific size [10000][3] like you suggested, it still takes roughly the same amount of time? 143 seconds with that size declaration for each array and 143.3 seconds without.

[jchd]

You're right. The following code takes around 65 s on my slow, slow PC.

Local $h = FileOpen("arr12x10kx3.au3", 2)
FileWriteLine($h, '#include-once' & @CRLF)
For $i = 1 To 12
    FileWriteLine($h, 'Local $a' & $i & ' = [ _')
    For $j = 1 To 9999
        FileWriteLine($h, ' ["abcdefghijklm", "nopqrstuvwxyz", 0.123456789], _')
    Next
    FileWriteLine($h, ' ["abcdefghijklm", "nopqrstuvwxyz", 0.123456789] _')
    FileWriteLine($h, ']' & @CRLF)
Next

Local $t = TimerInit()
#include "arr12x10kx3.au3"
ConsoleWrite(TimerDiff($t))

Anyway consider SQLite. Remember you can FileInstall the SQLite library.  Of course accessing an array via index will be faster than a database, but the advantage of a DB goes way beyond accessing a single item thru individual indices.  It all depends on how you'll be accessing data and what you intend to compute (e.g. statistical or partial results).

Edited August 13, 2019 by jchd
Paste error

[junkew]

• it will really depend on your usecase requirements if include au3 file array or database usage or txtfilebased or ...  approach is best
• resources can be included in your compiled exe approcah:  so data.txt could be directly get with function _ResourceGetAsString
• I would suggest to use some kind of integer constant value to indicate string1 or string2 or was it just in your example any random character string?