Truncated read in FileReadToArrray()

[_JL_]

Hi,

I am using FileReadToArray() to read a text file with over 8000 lines:

                $updDataSet = FileReadToArray($useOutputFilename & ".txt")

Each line is about a few hundred characters and the whole file is about 1.2MB.

What I found is that the resultant array only has 6448 elements, and the last line in the array is clearly truncated halfway compared to the text file. The function call didn't return any error - it just quietly stopped reading the rest of the lines from the file.

From the documentation this should be well-within the limits of array dimension and string size so I am not sure what the problem is.

Any pointers or suggestions are most appreciated.

Thanks.

[water]

Welcome to AutoIt and the forum!

You could use function _FileReadToArray from the File UDF to check if you get the same (wrong) result.
If yes, then there might be a problem with the file content.

[_JL_]

Thanks water for the suggestion.

I just tried _FileReadToArray() on the same file. It read about 200 more lines (~6600 vs ~6400) but the results are similar, i.e., incomplete read and the last line is truncated in the middle.

This suggests that the file content is not the problem as the two functions stopped at different points in the file. I also checked the file contents using notepad and found no problem.

Any other ideas?

Thanks.

[water]

You could use Notepad++ or a similar tool to display invisible characters. Maybe this sheds some light onto the issue.

[_JL_]

I believe it is not due to the file content because if that's the case the functions should always stop at the same point.

I just tried running the script in another PC with more memory (16GB vs 4 GB) and the function read ~8800 lines, still short of the 9000+ lines in the file but this strongly suggests that it is a memory-related issue. The host is far from running out of memory though (~7GB out of 16GB used with the script running) ...

[water]

Would you mind to post the file in this thread so we can play with it and see if we can reproduce the problem?

[water]

BTW: Which version of AutoIt do you run?

[_JL_]

I used the latest version of Autoit. I also just tried the beta version which is still no luck.

Attached the text file. Thanks for the help.

StockPrice-ALL.txt

[water]

I saved your file to C:\temp and get 9278 entries (rows 0-9277) in the returned array when running the following code.
Last row in the array is:

Quote

VHSI HSI Volatility Index;2017/05/31;11.78;12.16;11.78;11.92;;;HKD;2.7586;;;;;;;;2.7213;;;;;

My script:

#include <Array.au3>
Global $aFile = FileReadToArray("C:\temp\StockPrice-ALL.txt")
_ArrayDisplay($aFile)

Maybe you could download the file from your post above and run my script on this file?

[jchd]

If you remove the two first useless lines, then you have a SSV (semicolumn separated value) file which can be easily imported into an SQLite database, building a 9275-row table. You can query this DB using SQLite support proposed by AutoIt.

Note that the last column has no explicit name in the header line. I believe it's due to an extraneous semicolumn at the end of each data line.

[_JL_]

Your script works perfectly.

I found the problem - the txt file is being generated dynamically and the codes run into race condition where it may call FileReadtoArray() before the file is completely saved to disk. If I add some extra delay for the save to finish then problem solved!

Thanks to water for the help! This is a great community.

[water]

Glad to be of service  

[Malkey]

"StockPrice-ALL.txt" file into a 2D array.

#include <Array.au3>
#include <Date.au3>

Local $aFile
Local $aFile = FileReadToArray("StockPrice-ALL.txt")
;_ArrayDisplay($aFile)

Local $iUB = UBound($aFile)
Local $iMaxCols = 1 ; Must be at least 1 column and less than 50.

Local $tCur = _Date_Time_GetLocalTime()
Local $aRetArray[$iUB][50] = [[$aFile[0] & " - " & _Date_Time_SystemTimeToDateTimeStr($tCur)], [$aFile[1]]] ; First 2 lines and unknown number of columns (less than 50).

For $i = 2 To $iUB - 1 ; Start on 3rd line
    $aTemp = StringSplit($aFile[$i], ";", 2)
    If UBound($aTemp) > $iMaxCols Then $iMaxCols = UBound($aTemp) ; Find maximum number of columns.
    For $j = 0 To UBound($aTemp) - 1
        $aRetArray[$i][$j] = $aTemp[$j]
    Next
Next
ReDim $aRetArray[$iUB][$iMaxCols]

; Add extra row in array if last line in file is blank.
StringReplace(FileRead("StockPrice-ALL.txt"), @LF, "")
If @extended = $iUB Then _ArrayAdd($aRetArray, "", 0, ";", $iUB) ; Add extra row to array.

_ArrayDisplay($aRetArray)

 

[_JL_]

Thanks to jchd for the SQLite pointer and Malkey for the code example. Very useful to a AutoIT noob like me.

 

[jchd]

If you want I can make an example DB with what I believe is a better design. I'll do that in the (my) morning. Done.

You can use SQLite Expert (free for home version) to play with it.

Stock.sq3

Edited June 7, 2017 by jchd
DB added