Does FileReadToArray() function correctly?

[Spiff59]

Some crazy manipulations to the input file in this thread got me looking at FileReadToArray() and questioning whether it behaves properly. What seems to be implied in that thread is that the input file is line-delimited by @CRLF, but that individual lines may contain @CR characters.

FileReadToArray(), as presently written, tries to identify line delimiters as being either @CRLF, @CR or @LF. It's process flow is one of the following routes:

1. If a @CRLF or @LF is present, split on @CRLF and remove all @CR's from the file.

2. If a @CR is present, split on @CR

Should it remove stray @CR's when it has determined to break on @CRLF?

It does not bother removing stray @LF's. I wonder if it might be written more like this, which does not remove other characters when it splits on @CRLF, and also happens to be measurably faster:

Func _FileReadToArray($sFilePath, ByRef $aArray)
    Local $aFile = FileRead($sFilePath)
    If @error Then Return SetError(1, 0, 0);; unable to open the file
    Select
     Case StringInStr($aFile, @CRLF)
         If StringRight($aFile, 2) = @CRLF Then $aFile = StringTrimRight($aFile, 2)
         $aArray = StringSplit($aFile, @CRLF, 1)
     Case StringInStr($aFile, @CR)
         If StringRight($aFile, 1) = @CR Then $aFile = StringTrimRight($aFile, 1)
         $aArray = StringSplit($aFile, @CR)
     Case StringInStr($aFile, @LF)
         If StringRight($aFile, 1) = @LF Then $aFile = StringTrimRight($aFile, 1)
         $aArray = StringSplit($aFile, @LF)
     Case StringLen($aFile)
         Dim $aArray[2] = [1, $aFile]
     Case Else
         Return SetError(2, 0, 0) ; empty file
    EndSelect
    Return 1
EndFunc ;==>_FileReadToArray

(a part of the speed increase is due to the removal of a FileGetSize() on the FileRead() statement which seems superfluous. Maybe it is not?)

Edit: Another question might be how to handle a file with both lone @CR's and lone @LF's. The production version gives preference to @LF. The version above to @CR (although that could be easily changed). You'd likely need to pass another function parameter to allow the user to override that hierarchy by specifiying a preferred delimiter.

Edited October 6, 2012 by Spiff59

[water]

Why should it remove stray @CR's when it has determined to break on @CRLF?

The code doesn't remove all @CRLF, @CR or @LF. It only removes the rightmost to cut off emtpy lines at the end of the file.

The code has been discussed in trac ticket #198.

Edited October 6, 2012 by water

[BrewManNH]

What _FileReadToArray does is it checks to see if there are any LF in the file, if there are, it strips all CRs from the file then splits the lines by the LF characters. If there aren't any LFs in the file it splits on the CR only.

[kylomas]

Spiff59, et al.,

The following code demonstrates what you are finding, and what Valik pointed out 5 years ago in the trac ticket. Func _my_filelisttoarray is one way that I might choose to fix the problem.

I added a parm to trim trailing blank lines.

expandcollapse popup

;
;
;
#include <array.au3>
#include <file.au3>
$str =  'line 1' & @cr & _
  'line 2' & @lf & _
  'line 3' & @crlf & _
  'line 4' & @cr & _
  'line 5' & @lf & _
  'line 6' & @crlf & _
  'line 7' & @crlf & _
  'line 8' & @crlf & @crlf & @lf & @cr
if fileexists('k:temptest010.txt') then filedelete('k:temptest010.txt')
filewrite("k:temptest010.txt", $str)
local $a10
_filereadtoarray("k:temptest010.txt",$a10)
_arraydisplay($a10)
_my_filereadtoarray("k:temptest010.txt",$a10,true)
_arraydisplay($a10)
Func _my_FileReadToArray($sFilePath, ByRef $aArray, $trim_trailing_blank_lines = false)
 Local $aFile = FileRead($sFilePath)
 ; change all @crlf and @cr to @lf
 $aFile = stringregexpreplace($aFile,@crlf, @lf)
 $aFile = stringregexpreplace($aFile,@cr, @lf)
 ; trim blank lines if required
 if $trim_trailing_blank_lines then
  while stringleft(stringright($aFile,2),1) = @lf
   $aFile = stringtrimright($aFile,1)
  wend
  $aFile = stringtrimright($aFile,1)
 endif
 $aArray = stringsplit($aFile,@lf)
 if isarray($aArray) then
  return $aArray
 Else
  return $aFile
 endif
EndFunc   ;==>_FileReadToArray

I am sure that there are better ways to do the stringregexpreplace.

kylomas

[JohnOne]

I am sure that there are better ways to do the stringregexpreplace.

Like stringreplace for example.

[Spiff59]

That's probably the oldest Trac ticket I've ever looked over!

I'm not referring to the behavior regarding the last line of the file.

The ticket is useful is giving an idea of what sort of files are out there, when it states:

@CRLF ; Windows

@LF ; Unix

@CR ; Mac

My questions are:

Is it proper that when the function detects a file that is either @LF or @CRLF delimited that it also removes all single @CR's?

If a file contains both single @CR's and single @LF's, which should take precedence?

(there is also the question whether files with these character combinations even exist)

Edited October 7, 2012 by Spiff59

[Melba23]

Hi,

I use this RegEx to force all line endings into @CRLF format when I need to be sure:

#include <Array.au3>

$sBase_Text = "1" & @CRLF & "2" & @CR & "3" & @LF & "4" & @CRLF & "5"
$aArray = StringSplit($sBase_Text, @CRLF, 1)
_ArrayDisplay($aArray, "Not all @CRLF")

$sNew_Text = StringRegExpReplace($sBase_Text, "((?<!x0d)x0a|x0d(?!x0a))", @CRLF)

$aArray = StringSplit($sNew_Text, @CRLF, 1)
_ArrayDisplay($aArray, "Now all @CRLF")

It basically looks for single @CR or @LF characters not in a pair with the other.

M23

[guinness]

Interesting regular expression Melba23.

[Melba23]

guinness,

It certainly took a while to get it to work properly!

M23

[guinness]

What about this as an improvement?

expandcollapse popup

#include <Array.au3>
#include <FileConstants.au3>

Local $aArray = 0
_FileReadToArray(@ScriptFullPath, $aArray, Default)
_ArrayDisplay($aArray)

; #FUNCTION# ====================================================================================================================
; Name...........: _FileReadToArray
; Description ...: Reads the specified file into an array.
; Syntax.........: _FileReadToArray($sFilePath, ByRef $aArray [, $iFlag = 1])
; Parameters ....: $sFilePath           - Path and filename of the file to be read.
;                  $aArray              - The array to store the contents of the file.
;                  $iFlag               - [optional] 1 - Return the array count in the 0th index or 0 - Don't return the array count. Default is 1.
; Return values .: Success - Returns a 1
;                  Failure - Returns a 0
;                  @error  - 0 = No error.
;                  |1 = Error opening specified file
;                  |2 = Unable to Split the file
; Author ........: Jonathan Bennett <jon at hiddensoft dot com>, Valik - Support Windows Unix and Mac line separator
; Modified.......: Jpm - fixed empty line at the end, Gary Fixed file contains only 1 line, guinness - Optional flag to return the array count.
; Remarks .......: $aArray[0] will contain the number of records read into the array.
; Related .......: _FileWriteFromArray
; Link ..........:
; Example .......: Yes
; ===============================================================================================================================
Func _FileReadToArray($sFilePath, ByRef $aArray, $iFlag)
    Local $hFileOpen = FileOpen($sFilePath, $FO_READ)
    If $hFileOpen = -1 Then Return SetError(1, 0, 0)
    Local $sFileRead = FileRead($hFileOpen)
    FileClose($hFileOpen)

    ; Check if to return the array count in the 0th index
    If $iFlag = Default Or $iFlag = 1 Then
        $iFlag = 1
    Else
        $iFlag = 3
    EndIf

    $sFileRead = StringRegExpReplace($sFileRead, "((?<!x0d)x0a|x0d(?!x0a))", @CRLF)
    $aArray = StringSplit($sFileRead, @CRLF, $iFlag)
    If @error Then
        If StringLen($sFileRead) Then
            Local $aReturn[2] = [1, $sFileRead]
            $aArray = $aReturn
        Else
            Return SetError(2, 0, 0)
        EndIf
    EndIf
EndFunc   ;==>_FileReadToArray

[DXRW4E]

hi guinness, does not seem to make sense, if you use the StringRegExpReplace (passing twice), then why not use the StringRegExp (passing once, very much faster) ??

expandcollapse popup

#include <Array.au3>
#include <FileConstants.au3>

Local $aArray = 0
_FileReadToArray(@ScriptFullPath, $aArray, Default)
_ArrayDisplay($aArray)

; #FUNCTION# ====================================================================================================================
; Name...........: _FileReadToArray
; Description ...: Reads the specified file into an array.
; Syntax.........: _FileReadToArray($sFilePath, ByRef $aArray [, $iFlag = 1])
; Parameters ....: $sFilePath          - Path and filename of the file to be read.
;                 $aArray             - The array to store the contents of the file.
;                 $iFlag               - [optional] 1 - Return the array count in the 0th index or 0 - Don't return the array count. Default is 1.
; Return values .: Success - Returns a 1
;                 Failure - Returns a 0
;                 @error  - 0 = No error.
;                 |1 = Error opening specified file
;                 |2 = Unable to Split the file
; Author ........: Jonathan Bennett <jon at hiddensoft dot com>, Valik - Support Windows Unix and Mac line separator
; Modified.......: Jpm - fixed empty line at the end, Gary Fixed file contains only 1 line, guinness - Optional flag to return the array count.
; Remarks .......: $aArray[0] will contain the number of records read into the array.
; Related .......: _FileWriteFromArray
; Link ..........:
; Example .......: Yes
; ===============================================================================================================================
Func _FileReadToArray($sFilePath, ByRef $aArray, $iFlag)
    Local $hFileOpen = FileOpen($sFilePath, $FO_READ)
    If $hFileOpen = -1 Then Return SetError(1, 0, 0)
    Local $sFileRead = FileRead($hFileOpen)
    FileClose($hFileOpen)

    ; Check if to return the array count in the 0th index
    If $iFlag = Default Or $iFlag = 1 Then
        $iFlag = @LF
    Else
        $iFlag = ""
    EndIf
    ; or $iFlag optional, so _FileReadToArray($sFilePath, ByRef $aArray, $iFlag = "")
    ; If $iFlag <> "" Then $iFlag = @LF
   
    $aArray = StringRegExp($iFlag & $sFileRead & @LF, "([^rn]*)(?:rn|n|r)", 3)
    If @error Then
        If StringLen($sFileRead) Then
            Local $aReturn[2] = [1, $sFileRead]
            $aArray = $aReturn
        Else
            Return SetError(2, 0, 0)
        EndIf
    Else
        If $iFlag Then $aArray[0] = UBound($aArray) - 1
    EndIf
EndFunc   ;==>_FileReadToArray

and so 200% faster, or equally as fast as the StringSplit in fuction default

Ciao.

Edited October 7, 2012 by DXRW4E

[guinness]

DXRW4E,

You make a valid point. Nice function by the way.

[kylomas]

Spiff59,

The point of what I posted was to normalize end of line characters, as M23 and DXRW4E have done. As I alluded to, my regexpreplace was weak and has since been improved. I put in the trailing blank line option to avoid the old, tired discussions about the validity and handling of trailing blanks (there was recently a long, tedious thread devoted to this).

kylomas

[kylomas]

DXRW4E,

Nice, I missed the whole "byref" thing!

kylomas

@spiff59 - You had to know that Ohio St. was going to roll considering that the Cornhuskers had trouble with Wisconsin !

[Spiff59]

Spiff59,

The point of what I posted was to normalize end of line characters, as M23 and DXRW4E have done. As I alluded to, my regexpreplace was weak and has since been improved. I put in the trailing blank line option to avoid the old, tired discussions about the validity and handling of trailing blanks (there was recently a long, tedious thread devoted to this).

kylomas

I might have been involved in that old, tired discussion! If it was the one about FileCountLines() where the function would strip all the blank lines off the end of the file whether there was 1 or 1000. It bothered me that FileCountLines() rarely agreed with any editor on the planet as to how many lines were in a file.

I think these are all good examples. The point of this thread was that it seemed odd to me that in some cases FileReadToArray() could simply strip bytes from a file. They aren't treated as line delimiters. Just poof! And they're gone! It's seems the consensus that regardless of the combinations of @CRLF's, @CR's and @LF's in a file that they should all be treated as line delimiters. That does seem a more standardized rule than the exisitng logic that can cause some @CR's to simply vanish.

Edited October 7, 2012 by Spiff59

[DXRW4E]

Nice function by the way.

Thank You

DXRW4E,

Nice, I missed the whole "byref" thing!

I was editing the post, because as use the REGEXP, the options are endless, example

expandcollapse popup

; #FUNCTION# ====================================================================================================================
; Name...........: _FileReadToArray
; Description ...: Reads the specified file into an array.
; Syntax.........: _FileReadToArray($sFilePath, ByRef $aArray [, $iFlag = 1])
; Parameters ....: $sFilePath         - Path and filename of the file to be read.
;                $aArray             - The array to store the contents of the file.
;                $iFlag            - Optional: (add the flags together for multiple operations):
;                |$iFlag = 0 Return the array count in the 0th index (Default)
;                |$iFlag = 1 Don't return the array count
;                |$iFlag = 2 Don't return Line empty (ignores @CRLF & @CRLF & @CRLF ect ect)
;                |$iFlag = 4 Don't return Line empty or contain only whitespace character (ignores @CRLF & " " & @CRLF & @Tab & @CRLF ect ect)
;                |$iFlag = 8 Strip Line leading white space
; Return values .: Success - Returns a 1
;                Failure - Returns a 0
;                @error  - 0 = No error.
;                |1 = Error opening specified file
;                |2 = Unable to Split the file
; Author ........: Jonathan Bennett <jon at hiddensoft dot com>, Valik - Support Windows Unix and Mac line separator
; Modified.......: Jpm - fixed empty line at the end, Gary Fixed file contains only 1 line, guinness & DXRW4E - Add Optional flag, see $iFlag.
; Remarks .......: $aArray[0] will contain the number of records read into the array.
; Related .......: _FileWriteFromArray
; Link ..........:
; Example .......: Yes
; ===============================================================================================================================
Func _FileReadToArray($sFilePath, ByRef $aArray, $iFlag = 0)
    Local $ArrayCount, $RegExp = "(?:rn|n|r)([^rn]*)", $hFileOpen = FileOpen($sFilePath, $FO_READ)
    If $hFileOpen = -1 Then Return SetError(1, 0, 0)
    Local $sFileRead = FileRead($hFileOpen)
    FileClose($hFileOpen)

    ; Check Optional $iFlag
    If Not BitAND($iFlag, 1) Then $ArrayCount = "ArrayCount" & @LF
    If BitAND($iFlag, 2) Then $RegExp = "(?:rn|n|r)([^rn]+)"
    If BitAND($iFlag, 4) Then $RegExp = "s*(?:rn|n|r)([^rn]+)"
    If BitAND($iFlag, 8) Then $RegExp = StringReplace($RegExp, ")", ")h*", 1, 1)

    $aArray = StringRegExp(@LF & $ArrayCount & $sFileRead, $RegExp, 3)
    If @error Then
        If StringLen($sFileRead) Then
            Local $aReturn[2] = [1, $sFileRead]
            $aArray = $aReturn
        Else
            Return SetError(2, 0, 0)
        EndIf
    ElseIf $ArrayCount Then
        $aArray[0] = UBound($aArray) - 1
    EndIf
    Return 1
EndFunc   ;==>_FileReadToArray

tested with a INF File 42 MB approximately 1,000,000 line, Return in approximately 6802.4406 seconds, instead Default UDF that uses StringSplit Return in approximately 6620.1655 seconds, more or less StringRegExp and StringSplit have exactly the same speed

sorry again for my english

Ciao.

Edited October 7, 2012 by DXRW4E

[guinness]

DXRW4E,

Certainly alot more advanced than the UDF version. Thanks.

[jchd]

Not diving in the gory details of this thread, let me add a sidenote that some regexp fans/users could use some day.

At the start of a PCRE pattern, you can force your own line break character or combination, thusly:

(*CR) carriage return

(*LF) linefeed

(*CRLF) carriage return, followed by linefeed

(*ANYCRLF) any of the three above

Note that this last flag doesn't work with PCRE as compiled into AutoIt (would need a compile-time option)

(*ANY) all Unicode newline sequences

Current AutoIt PCRE defaults to the equivalent of (*CRLF).

It's also possible to change the line break sequence in the middle of a pattern.

Edited October 7, 2012 by jchd

[BrewManNH]

I'm not sure of how many people will actually have files with varying line endings unless they're doing something like this

copy filea.txt + fileb.txt filec.txt

That would be an extreme edge case I would think. Normally you'll have CRLF (Windows and numerous other file systems), CR (Normally only applies to older Apples or a few older computer systems), or LF (Unix/Linux/OSX, and a few smaller OSs). It would be a strange file indeed to have one that mixed them.

The way that _FileReadToArray works now seems fine to me. If there are LF characters, then use one method, if there are no LF characters, use the other. No need to over-complicate things. If someone has an issue with a strangely formatted text file, then I would suggest that they fix the file before using FRTA, and not have the overhead of having FRTA normalize line endings and then splitting the file. I'm sure on large files, reformatting LEs would be a considerable time waste for an insignificant gain on an insignificant number of files.

Edited October 7, 2012 by BrewManNH

[jchd]

That's my opinion too. I simply mentionned the PCRE line break modifiers as an aside, since it closely relates to this subject.