Re-write of _FileReadToArray()

[l3ill]

Thanks for the notes, that does make more sense now 4+2 = 6...   wish I hadn't wasted all that time testing with 4 

Yes there was a WS left from one of the test runs, which by the way works with 6 & with 2 but not with 4.

Thanks for all the testing, by the way.

Glad to help! I always learn something when I interact with you. Thanks for letting me

Edited January 10, 2014 by l3ill

[Melba23]

l3ill,

In this thread it was me who who was learning.

M23

[DXRW4E]

Hi Melba23, is impossible (you can not ever know how fields has a line ehhh) to work with fields in 2D array, the only way and that _FileReadToArray & _FileWriteFromArray check and use the Array access on expression , trivial example

If IsArray($aArray[1]) Then
    For $i = 1 To $aArray[0]
        For $y = 0 To UBound($aArray[$i]) - 1
            $sData &= ($aArray[$i])[$y] ;;& ","
        Next
        $sData &= @CRLF
    Next
Else
    For $i = 1 To $aArray[0]
        $sData &= $aArray[$i] & @CRLF
    Next
EndIf

so in the case of fields, the _FileReadToArray will have to return an array of array

;;example
Local $aArray[100] = [99]
    For $i = 0 To UBound($aArray[$i]) - 1
        $aArray[$i] = StringSplit("1,2,3,4,5,6,8,9,10", ",")
    Next
MsgBox(0, $aArray[0], ($aArray[1])[0] & @LF & ($aArray[1])[1])
;;ect ct ect

however so thanks to Array access on expression the user can always read them immediately

 

Ciao.

Edited January 10, 2014 by DXRW4E

[Melba23]

DXRW4E,

 

you can not ever know how fields has a line

Quite so - and this is why there is the 2 flag which allows the array to expand to match the greatest number of fields found in the file.

Create this file:

z
1,2,3,4,5
"a","b","c","d","e","f"
1,2,3,,4

and run this script:

$aRetArray = _FileReadToArray_New("2D_DiffSizes.txt", 2, ",") ; Note $iFlags set to 2 to allow array to expand
ConsoleWrite(@error & @CRLF)
_ArrayDisplay($aRetArray, "", Default, 8, 10)

The UDF produces this array:

Row        Col 0      Col 1      Col 2      Col 3      Col 4      Col 5      
[0]        z                                                                 
[1]        1          2          3          4          5                     
[2]        "a"        "b"        "c"        "d"        "e"        "f"        
[3]        1          2          3                     4

Note the array has expanded to fit the mximum number of fields in a line of the file - without having to specify that value when calling the function. Omitting the 2 flag will return an error if there are variable numbers of fields on different lines of teh file, allowing the user to check if a file which needs this precision is correctly structured.

I hope that answers your question.

M23

[DXRW4E]

Hi Melba23, I thinkbelieve is wrong and not practical because ve created 5 columns only because they only use one line, so if the file 1000 line 4 field for line and only one line will have 1000 (or 10000) fields, your array will be? $aArray[1000][1000] (996000 arrays that do not serve ehhhh), not to mention later which is very important to always know (is very important) the numer of fiels because a user can search a read only XXX fields, so in your array you never know when the actualreal number of fileds on line
 
examle see here http://msdn.microsoft.com/en-us/library/windows/hardware/ff547428(v=vs.85).aspx there a function only for fields count SetupGetFieldCount
 
Ciao.

Edited January 10, 2014 by DXRW4E

[Melba23]

DXRW4E,

I understand your point, but this is intended as a simple function to complement _FileWriteFromArray and is not designed to cater for every file type that a user may come across. So although I accept that a file such as you describe would cause problems, I must question the chances of coming across such a strange beast and needing to read it into a 2D array in real life rather than as a theoretical curiosity for discussion.

And if a user did try to do this without using the 2 flag, the function would return an error indicating that there were different numbers of fields within the file - which could prompt the user to load the file into a 1D array for further evaluation.

M23

[DXRW4E]

Hi Melba23, here's how I think it should be (written in the race, only a sample function, certainly can be done even better, it was just to give you an idea, as you can see 5 line resolves all )

expandcollapse popup

#include <Array.au3>
#include <FileConstants.au3>

FileDelete(@DesktopDir & '\Test.inf')
Local $sData,  $aArray
For $i = 1 to 20000
    $sData &= "Line - " & $i & ', "1,,, ",2,'',,,3'',4,5' & @CRLF
Next
For $i = 1 to 20000
    $sData &= "Line - " & $i & ", 1,2,3,4,5,6,7,8,9,10" & @CRLF
Next
FileWrite(@DesktopDir & '\Test.inf', $sData)

$aArray = _FileReadToArrayEx(@DesktopDir & '\Test.inf', ",")
_ArrayDisplay($aArray)
_ArrayDisplay(($aArray[1]))
_ArrayDisplay(($aArray[20001]))


; #FUNCTION# ====================================================================================================================
; Name...........: _FileReadToArrayEx
; Author ........: DXRW4E
; ===============================================================================================================================
Func _FileReadToArrayEx($sFilePath, $sDelim = "", $iFlag = 0)
    If $sDelim Then
        Local $hFileOpen = FileOpen($sFilePath, $FO_READ)
        If $hFileOpen = -1 Then Return SetError(1, 0, 0)
        $sData = StringRegExpReplace(FileRead($hFileOpen), '\r\n?', @LF)
        FileClose($hFileOpen)
        ;; '' do not need it added for compatibility with AU3 file, however $iFlag can also solve this
        ;;  so normal Pattern should be ("[^" & $sDelim & '\n"]*+(?:[^' & $sDelim & '\n"]|"[^"]*")*+\K' & $sDelim)
        $sData = StringRegExpReplace($sData, '[^' & $sDelim & '\n"'']*+(?:[^' & $sDelim & '\n"'']|''[^'']*''|"[^"]*")*+\K' & $sDelim, @CR)
        $sData = StringSplit($sData, @LF, 1)
        For $i = 1 To $sData[0]
            ;;however field count in first element is an option which will be in $iFlag
            ;;$sData[$i] = StringSplit($sData[$i], @CR, 3) ;;this is to use By default, in general 0 field is always the first field
            $sData[$i] = StringSplit($sData[$i], @CR, 1)  ;; fileds count in first element
        Next
        Return $sData
    Else
        ; normal mod ect ect
    EndIf
EndFunc    ;==>_FileReadToArrayEx

_FileReadToArray()

 

_FileReadToArrayEx() - also is three times faster

 
Ciao.

Edited January 11, 2014 by DXRW4E

[jchd]

AFAIK files with delimited structure rarely have varying number of fields. This is counterintuitive in the context of FileReadToArray, especially since AutoIt doesn't handle sparse arrays.

Anyway, missing fields should really be Null in the result array. Making them silently empty strings changes the semantic of an absent field.

In my (biased) view, this function should be kept simple and made to work with fixed number of fields. It is the responsability of the user to apply prior formatting of weird sources they may have to process. Else the real-world will force maintainer(s) to ask themselves how to deal with that:

     ACME ltd.

          Quaterly Sales Report 3Q-2013 (Million USD)

===============================================================

|  Region | Fishing accessories | iShit Products | Sunglasses |

|---------|---------------------|----------------|------------|

| Europe  |              158.46 |         118.20 |      45.13 |

| Africa  |               10.04 |          21.89 |   not sold |

| America |               45.66 |         999.99 |     777.77 |

| Asia    |               38.03 |           0.14 |      60.42 |

===============================================================

[Melba23]

DXRW4E,

I realised what you meant when I awoke this morning - sorry for not fully comprehending it last night. And thanks for the example which saved me having to write one. I like the idea a lot - thanks for suggesting it.

jchd,

 

this function should be kept simple and made to work with fixed number of fields

I agree - and that was why I set the default to be for a fixed number of fields. I will have a think about using the "array-in-array" idea for files with non-standard field numbers.

M23

[DXRW4E]

Hi Melba23, already posted here '?do=embed' frameborder='0' data-embedContent>> if you like you can use it, certainly can be done even better, but it already seems OKK

expandcollapse popup

#include <File.au3>
#include <Array.au3>
#include <MsgBoxConstants.au3>
#include "_FileReadToArrayEx.au3"
#include "_FileReadToArray_New.au3"

FileDelete(@DesktopDir & '\Test.inf')
Local $sData,  $aArray, $fTimerDiff
For $i = 1 to 50000
    $sData &= "Line - " & $i & ', "1  ,  ,  , ",2 ,'',,,3'',4 ,5      ' & @CRLF
Next
$sData &= @CRLF & @CRLF & "       " & @CRLF  & @CRLF & "    "  & @CRLF
For $i = 1 to 50000
    $sData &= "Line - " & $i & ", 1,2,3,4,5,6,7,8,9,10" & @CRLF
Next
FileWrite(@DesktopDir & '\Test.inf', $sData)

Local $aArray
$fTimerDiff = TimerInit()
_FileReadToArray(@DesktopDir & '\Test.inf', $aArray)
$fTimerDiff = TimerDiff($fTimerDiff)
ConsoleWrite("_FileReadToArray() : " & $fTimerDiff & @CRLF)

$fTimerDiff = TimerInit()
$aArray = _FileReadToArrayEx(@DesktopDir & '\Test.inf')
;$aArray = _FileReadToArrayEx(@DesktopDir & '\Test.inf', ",", 0 + 1 + 16 + 12)
$fTimerDiff = TimerDiff($fTimerDiff)
ConsoleWrite("_FileReadToArrayEx() : " & $fTimerDiff & @CRLF)

$fTimerDiff = TimerInit()
$aArray = _FileReadToArray_New(@DesktopDir & '\Test.inf', 2, ",")
$fTimerDiff = TimerDiff($fTimerDiff)
ConsoleWrite("_FileReadToArray_New() : " & $fTimerDiff & @CRLF)

$fTimerDiff = TimerInit()
$aArray = _FileReadToArrayEx(@DesktopDir & '\Test.inf', ",", $FRTA_ARRAYFIELD + 0 + $FRTA_STRIPALL + $FRTA_CHECKSINGEQUOTE)
;$aArray = _FileReadToArrayEx(@DesktopDir & '\Test.inf', ",", 0 + 1 + 16 + 12)
$fTimerDiff = TimerDiff($fTimerDiff)
ConsoleWrite("_FileReadToArrayEx() : " & $fTimerDiff & @CRLF)
_ArrayDisplay($aArray)
_ArrayDisplay(($aArray[1]))
 _ArrayDisplay(($aArray[20001]))

;~ >Running AU3Check (3.3.11.2)  from:C:\Program Files (x86)\AutoIt3\Beta
;~ +>13:22:39 AU3Check ended.rc:0
;~ >Running:(3.3.11.2):C:\Program Files (x86)\AutoIt3\Beta\autoit3.exe "C:\Users\DXRW4E\Desktop\_FileReadToArray_Test.au3"
;~ --> Press Ctrl+Alt+F5 to Restart or Ctrl+Break to Stop
;~ _FileReadToArray() : 393.66877891673
;~ _FileReadToArrayEx() : 317.098957171152
;~ _FileReadToArray_New() : 5494.90236191552
;~ _FileReadToArrayEx() : 2729.48573391968
;~ +>13:22:55 AutoIt3.exe ended.rc:0
;~ >Exit code: 0    Time: 15.927

Ciao.

Edited January 11, 2014 by DXRW4E

[Melba23]

Hi,

New Beta function and examples in the first post.

M23

[l3ill]

Not a surprise to see that America buys the most iShit products....

 

     ACME ltd.

          Quaterly Sales Report 3Q-2013 (Million USD)

===============================================================

|  Region | Fishing accessories | iShit Products | Sunglasses |

|---------|---------------------|----------------|------------|

| Europe  |              158.46 |         118.20 |      45.13 |

| Africa  |               10.04 |          21.89 |   not sold |

| America |               45.66 |         999.99 |     777.77 |

| Asia    |               38.03 |           0.14 |      60.42 |

===============================================================

Edited January 14, 2014 by l3ill