How to free up memory after using _FileReadToArray

[BryanWall]

I need to read multiple files and run through each line to look for a gap in a time column from one row to another. For this, I use _FileReadToArray then loop through the array since that UDF works far better than FileReadLine. It works fine for the first file or two, but then I get a "Error Allocating Memory" error. If I start from the file it errors on, it works fine then throws an error again a few files later. The files are generally around half a gig with millions of rows.

I have tried to free up memory by setting the array variable to "", using _ArrayDelete, redeclaring the array variable, and _ReduceMemory() from 

Below is a simple test process I've been using to try to figure out how to solve this problem.

#include <File.au3>
#include <Array.au3>


Global $Files[6]


$Files[0] = "D:\Bryan\Applanix test\AutomationTest\Exports\PWL00217_SG\RealTime_PWL00217_SG.txt"
$Files[1] = "D:\Bryan\Applanix test\AutomationTest\Exports\PWL00230_AZ\RealTime_PWL00230_AZ.txt"
$Files[2] = "D:\Bryan\Applanix test\AutomationTest\Exports\PWL00240_AZ\RealTime_PWL00240_AZ.txt"
$Files[3] = "D:\Bryan\Applanix test\AutomationTest\Exports\PWL00217_SG\RealTime_PWL00217_SG.txt"
$Files[4] = "D:\Bryan\Applanix test\AutomationTest\Exports\PWL00230_AZ\RealTime_PWL00230_AZ.txt"
$Files[5] = "D:\Bryan\Applanix test\AutomationTest\Exports\PWL00240_AZ\RealTime_PWL00240_AZ.txt"

For $i = 0 to 5
    wLog($i)
    ;_FileReadToArray($Files[$i],$File)
    $check = Test($Files[$i])
    wLog($check)
Next
wLog("Got here")
Exit

Func Test(ByRef $Path)
    Local $File
    Local $Output
    _FileReadToArray($Path,$File)
    $Output = $File[0]
    ;_ArrayDelete($File,("0-" & $File[0]))
    ;Local $File = ""
    ;$File = ""
    ;_ReduceMemory()
    
    Return $Output
    
EndFunc

If it matters, I am running the script from a .au3 file, not done up as an .exe. 

The only next step I can think of doing is to take the entire function I use to read the files, find the time gaps, and populate a different array accordingly into a seperate .au3 or even a .exe and call it, and have it return the needed gap array. 

 

Any suggestions? Also, if I have to go the seperate code route do I have to publish/compile it fully as an .exe or would puting it in a .au3 and including that cause the memory to free up when it is done?

[BryanWall]

And I can confirm the separate .au3 and include still triggers the memory error.

[Jos]

Why do you need to read them into an Array first, in stead of processing them one at a time record by record?

[BryanWall]

Because whenever I tried FileReadLine checking millions of rows was taking many, many minutes as opposed to _FileReadToArray which was only taking ~20 seconds per file while iterating through the whole thing to find gaps takes less than a second. As I recall that was a known thing with FileReadLine, it gets slower the further into a file you get because it is reading every line before the one you want every time, or at least that is how I recall the description. 
Is there another option I'm not aware of?

 

[Jos]

Don't think it does that unless you specify the linenumber with a filename. Doing an openfile and then using the returned filehandle in a filereadline till the eof should not do that as far as I know.

Edited December 12, 2023 by Jos

[Nine]

I agree with Jos.  What taking so much time with FileReadLine is when you DO NOT use file handle.  As for memory consumption, it will be far less than using _FileReadToArray.

[jchd]

Also AutoIt arrays are capped at 16M cells.

[BryanWall]

I am trying to FileReadLine but it is extremely slow. Here is the simple test code I'm using 

For $i = 0 to 5
    wLog($i)
    Test($Files[$i])
Next
Exit

Func Test(ByRef $File)
    Local $FileHandle
    Local $LineNum = 1
    Local $loopCount = 0
    
    $FileHandle = FileOpen($File)
    
    While 1
        $line = FileReadLine($FileHandle,$LineNum)
        if @error = -1 Then ExitLoop
        $loopCount+=1
        if $loopCount > 100 Then
            wLog($LineNum)
            $loopCount = 0
        EndIf
        $LineNum+=100
    WEnd
    
EndFunc

 

Basically I'm just reading every hundred lines and writing to the console every hundred loops which is every 10,000 lines. 

Here is what the is in the console. First 20,000 rows in 3 seconds, but it gets slower the longer it runs.

If this is supposed to be just as quick as reading to array, what am I doing wrong?

 

 

 

 

[BryanWall]

Well, aside from forgetting to include the FileClose at the end of the function but that isn't the issue causing the slowness.

[Jos]

You are still using a linenumber, don't!

try:

For $i = 0 to 5
    wLog($i)
    Test($Files[$i])
Next
Exit

Func Test(ByRef $File)
    Local $FileHandle
    Local $LineNum = 1
    Local $loopCount = 0
    
    $FileHandle = FileOpen($File)
    
    While 1
        $line = FileReadLine($FileHandle)
        if @error Then ExitLoop
        $loopCount+=1
        if $loopCount > 100 Then
            wLog($LineNum)
            $loopCount = 0
        EndIf
        $LineNum+=100
    WEnd
    FileClose($FileHandle)
EndFunc

what does this give?

 

quote from the helpfile: 

Quote

If a filehandle is passed to the function and no line number is specified, the "next" line will be read - for a newly opened file this will be the first line. If lines are read in a loop, it is recommended to let the count increase automatically and not to use the "line" parameter, as doing so will force AutoIt to re-read the file from the beginning each time, significantly slowing execution time for large files.

 

Edited December 12, 2023 by Jos

[BryanWall]

That goes far quicker. about 6 seconds per file. Thank you.

That said, it seems surreal to me that reading every single line in a file is quicker than any method of reading only every 100 lines or so.  

[Jos]

Understood, but it isn't as to know what the line number is, you have to read it from scratch each time!  

[Jos]

by the way: It is also good practice to do a fileclose(FileHandle) after you are done as else the handles will remain open till AutoIt closes.
see updated code I posted.

Edited December 12, 2023 by Jos

[BryanWall]

Yeah I had noted that in an additional reply myself.  I was going to edit it to have FileClose just before EndFunc but I can't edit my posts. Either I'm missing where it is or I'm in a sort of probation status as a new account.

[Melba23]

BryanWall,

You should see the "edit" menu item now.

M23

[BryanWall]

Indeed I can now see an edit option. Thank you.