Improvement of included _FileListToArray function.

[Beege]

I agree. Very good example. Ive never seen it done that way.

[Wus]

If that algorithm is new to you then I highly suggest that you read up on graph-searching and tree-traversal algorithms as they come in handy all the time.

If you view the directories as nodes and the files as leaves then the following are all very applicable (and should be in every programmers toolbox anyway):

http://en.wikipedia.org/wiki/Depth-first_search

http://en.wikipedia.org/wiki/Breadth-first_search

http://en.wikipedia.org/wiki/Tree_traversal

Edited July 7, 2009 by Wus

[Spiff59]

I wasn't particularily comfortable with the recursive function calls added in this thread (borrowed from another FLTA version that's out there I believe). But I'd seen the practice used around the forums so much I assumed it was commonplace here. If I remember any of my schooling, it was probably ranked as about the second worst programming practice, with self-modifying code being the ultimate offense. Recursion always risked overflowing an actual system stack, but I suppsoe that's not nearly as much of an issue with memory-rich PC's and modern processors?

This sort of approach would definately make a more structured, by-the-rules, fullblown FLTA-Ex than the version in post 175.

I tinkered with Wus's routine a little. getting rid of the FileChangeDir commands and @WorkingDir macros has helped it out somewhat.

Func _FLTA_NonRecursiveRecursion($sPath, $bRecursive)
    Local $sRet = ""
    $sPath &= "|"
    While $sPath
        $sCurrPathLen = StringInStr($sPath, "|")
        $sCurrPath = StringLeft($sPath, $sCurrPathLen - 1) & "\"
        Local $search = FileFindFirstFile($sCurrPath & "*.*")
        While 1
            $file = FileFindNextFile($search)
            If @error Then ExitLoop
            If @extended Then; Folder (Beta 3.3.1.1)
                If $bRecursive Then $sPath &= $sCurrPath & $file & "|"
            Else; File
                $sRet &= "|" & $sCurrPath & $file
            EndIf
        WEnd
        $sPath = StringTrimLeft($sPath, $sCurrPathLen)
    WEnd
    FileClose($search)
    Return StringSplit(StringTrimLeft($sRet, 1), "|")
EndFunc;==>_FLTA_NonRecursiveRecursion

Edit: I guess there's no interest in fixing the production bug I mentioned at the top of this forum page?

Edited July 7, 2009 by Spiff59

[Valik]

Edit: I guess there's no interest in fixing the production bug I mentioned at the top of this forum page?

This isn't exactly the appropriate place to be reporting bugs now, is it?

[Beege]

Here it is written with the folderstack concept just shown and error handling. Also rewrote it not to return a stringsplit(). More like in the original filelisttoarray(). Its again even faster.

expandcollapse popup

Func _FileListToArrayEx100000($sPath, $sFilter = "*", $iFlag = 0, $bRecursive = False)
    Local $sFile, $hSearch, $sWorkingdir, $asFilelist[1], $aFolderStack[2]
    If Not FileExists($sPath) Then Return SetError(1, 1, "")
    If (StringInStr($sFilter, "\")) Or (StringInStr($sFilter, "/")) Or (StringInStr($sFilter, ":")) Or (StringInStr($sFilter, ">")) Or (StringInStr($sFilter, "<")) Or (StringInStr($sFilter, "|")) Or (StringStripWS($sFilter, 8) = "") Then Return SetError(2, 2, "")
    If Not ($iFlag = 0 Or $iFlag = 1 Or $iFlag = 2) Then Return SetError(3, 3, "")
    If (StringMid($sPath, StringLen($sPath), 1) = "\") Then $sPath = StringTrimRight($sPath, 1)
    $aFolderStack[0] = 1
    $aFolderStack[1] = $sPath
    While $aFolderStack[0] > 0
        $sWorkingdir = $aFolderStack[$aFolderStack[0]]
        $aFolderStack[0] -= 1
        $hSearch = FileFindFirstFile($sWorkingdir & '\' & $sFilter)
        While 1
            $sFile = FileFindNextFile($hSearch)
            If @error Then ExitLoop
            If Not @extended And $iFlag = 2 Then ContinueLoop
            If @extended Then
                If $bRecursive Then
                    $aFolderStack[0] += 1
                    If UBound($aFolderStack) <= $aFolderStack[0] Then ReDim $aFolderStack[UBound($aFolderStack) * 2]
                    $aFolderStack[$aFolderStack[0]] = $sWorkingdir & "\" & $sFile
                EndIf
                If $iFlag <> 1 Then
                    $asFilelist[0] += 1
                    If UBound($asFilelist) <= $asFilelist[0] Then ReDim $asFilelist[UBound($asFilelist) * 2]
                    $asFilelist[$asFilelist[0]] = $sWorkingdir & "\" & $sFile
                EndIf
            Else
                $asFilelist[0] += 1
                If UBound($asFilelist) <= $asFilelist[0] Then ReDim $asFilelist[UBound($asFilelist) * 2]
                $asFilelist[$asFilelist[0]] = $sWorkingdir & "\" & $sFile
            EndIf
        WEnd
        FileClose($hSearch)
    WEnd
    ReDim $asFilelist[$asFilelist[0] + 1]
    If UBound($asFilelist) = 1 Then Return SetError(4, 4, "")
    Return $asFilelist
EndFunc

Edited September 13, 2011 by Beege

[Spiff59]

This isn't exactly the appropriate place to be reporting bugs now, is it?

Ahem. No, it is not.

Thank you for the invitation (to report it through proper channels).

[Valik]

What does the comment at the end of this line mean?

If (StringMid($sPath, StringLen($sPath), 1) = "\") Then $sPath = StringTrimRight($sPath, 1); needed for Win98 for x:\  root dir

Is that need for Windows 98 or all versions? And if it's need for Windows 98... why is it in the code? AutoIt doesn't run on Windows 98.

[Beege]

I copied those first 5 lines from the original _filelisttoarray().

[Spiff59]

What does the comment at the end of this line mean?

If (StringMid($sPath, StringLen($sPath), 1) = "\") Then $sPath = StringTrimRight($sPath, 1); needed for Win98 for x:\  root dir

Is that need for Windows 98 or all versions? And if it's need for Windows 98... why is it in the code? AutoIt doesn't run on Windows 98.

That's staright from the production FLTA.

It strips off a trailing '/' from $sPath, then all future refernces to $sPath will append a '/' back on.

We'd switched to this for the edit:

If StringRight($sPath, 1) <> "\" Then $sPath &= "\"; ensure trailing slash

and then removed all the appends from the rest of the function.

Edited July 7, 2009 by Spiff59

[Valik]

My point was: Fix the code or fix the comment.

[SmOke_N]

Here it is written with the folderstack concept just shown and error handling. Also rewrote it not to return a stringsplit(). More like in the original filelisttoarray(). Its again even faster.

expandcollapse popup

Func _FileListToArrayEx100000($sPath, $sFilter = "*", $iFlag = 0, $bRecursive = False)
    Local $sFile, $hSearch, $sWorkingdir, $asFilelist[1], $aFolderStack[2]
    If Not FileExists($sPath) Then Return SetError(1, 1, "")
    If (StringInStr($sFilter, "\")) Or (StringInStr($sFilter, "/")) Or (StringInStr($sFilter, ":")) Or (StringInStr($sFilter, ">")) Or (StringInStr($sFilter, "<")) Or (StringInStr($sFilter, "|")) Or (StringStripWS($sFilter, 8) = "") Then Return SetError(2, 2, "")
    If Not ($iFlag = 0 Or $iFlag = 1 Or $iFlag = 2) Then Return SetError(3, 3, "")
    If (StringMid($sPath, StringLen($sPath), 1) = "\") Then $sPath = StringTrimRight($sPath, 1) ; needed for Win98 for x:\  root dir
    $aFolderStack[0] = 1
    $aFolderStack[1] = $sPath
    While $aFolderStack[0] > 0
        $sWorkingdir = $aFolderStack[$aFolderStack[0]]
        $aFolderStack[0] -= 1
        $hSearch = FileFindFirstFile($sWorkingdir & '\' & $sFilter)
        While 1
            $sFile = FileFindNextFile($hSearch)
            If @error Then ExitLoop
            If Not @extended And $iFlag = 2 Then ContinueLoop
            If @extended Then
                If $bRecursive Then
                    $aFolderStack[0] += 1
                    If UBound($aFolderStack) <= $aFolderStack[0] Then ReDim $aFolderStack[UBound($aFolderStack) * 2]
                    $aFolderStack[$aFolderStack[0]] = $sWorkingdir & "\" & $sFile
                EndIf
                If $iFlag <> 1 Then
                    $asFilelist[0] += 1
                    If UBound($asFilelist) <= $asFilelist[0] Then ReDim $asFilelist[UBound($asFilelist) * 2]
                    $asFilelist[$asFilelist[0]] = $sWorkingdir & "\" & $sFile
                EndIf
            Else
                $asFilelist[0] += 1
                If UBound($asFilelist) <= $asFilelist[0] Then ReDim $asFilelist[UBound($asFilelist) * 2]
                $asFilelist[$asFilelist[0]] = $sWorkingdir & "\" & $sFile
            EndIf
        WEnd
        FileClose($hSearch)
    WEnd
    ReDim $asFilelist[$asFilelist[0] + 1]
    If UBound($asFilelist) = 1 Then Return SetError(4, 4, "")
    Return $asFilelist
EndFunc

** Off topic, and strictly code related **

If (StringInStr($sFilter, "\")) Or (StringInStr($sFilter, "/")) Or (StringInStr($sFilter, ":")) Or (StringInStr($sFilter, ">")) Or (StringInStr($sFilter, "<")) Or (StringInStr($sFilter, "|")) Or (StringStripWS($sFilter, 8) = "") Then Return SetError(2, 2, "")

Would this make more sense?

If StringRegExp($sFilter, "[\\/ :> <\|]|(?s)\A\s*\z") Then Return SetError(2, 2, "")

If (StringMid($sPath, StringLen($sPath), 1) = "\") Then $sPath = StringTrimRight($sPath, 1) ; needed for Win98 for x:\  root dir

Since you're using two string functions here, and a 3rd if it's found, would this make more sense?

$sPath = StringRegExpReplace($sPath, "[\\]+\z", "")

[Beege]

That would make way more sense.

expandcollapse popup

Func _FileListToArrayEx100000($sPath, $sFilter = "*", $iFlag = 0, $bRecursive = False)
    Local $sFile, $hSearch, $sWorkingdir, $asFilelist[1], $aFolderStack[2]
    If Not FileExists($sPath) Then Return SetError(1, 1, "")
    If StringRegExp($sFilter, "[\\/ :> <\|]|(?s)\A\s*\z") Then Return SetError(2, 2, "")
    If Not ($iFlag = 0 Or $iFlag = 1 Or $iFlag = 2) Then Return SetError(3, 3, "")
    $sPath = StringRegExpReplace($sPath, "[\\]+\z", "")
    $aFolderStack[0] = 1
    $aFolderStack[1] = $sPath
    While $aFolderStack[0] > 0
        $sWorkingdir = $aFolderStack[$aFolderStack[0]]
        $aFolderStack[0] -= 1
        $hSearch = FileFindFirstFile($sWorkingdir & '\' & $sFilter)
        While 1
            $sFile = FileFindNextFile($hSearch)
            If @error Then ExitLoop
            If Not @extended And $iFlag = 2 Then ContinueLoop
            If @extended Then
                If $bRecursive Then
                    $aFolderStack[0] += 1
                    If UBound($aFolderStack) <= $aFolderStack[0] Then ReDim $aFolderStack[UBound($aFolderStack) * 2]
                    $aFolderStack[$aFolderStack[0]] = $sWorkingdir & "\" & $sFile
                EndIf
                If $iFlag <> 1 Then
                    $asFilelist[0] += 1
                    If UBound($asFilelist) <= $asFilelist[0] Then ReDim $asFilelist[UBound($asFilelist) * 2]
                    $asFilelist[$asFilelist[0]] = $sWorkingdir & "\" & $sFile
                EndIf
            Else
                $asFilelist[0] += 1
                If UBound($asFilelist) <= $asFilelist[0] Then ReDim $asFilelist[UBound($asFilelist) * 2]
                $asFilelist[$asFilelist[0]] = $sWorkingdir & "\" & $sFile
            EndIf
        WEnd
        FileClose($hSearch)
    WEnd
    ReDim $asFilelist[$asFilelist[0] + 1]
    If UBound($asFilelist) = 1 Then Return SetError(4, 4, "")
    Return $asFilelist
EndFunc

[Spiff59]

I like this one (of course)

expandcollapse popup

Func _FLTA_NonRecursiveRecursion($sPath, $sFilter, $iFlag, $bRecursive)
    Local $sRet = ""
    If Not FileExists($sPath) Then Return SetError(1, 1, "")
    $sPath = StringRegExpReplace($sPath, "[\\/]+\z", "")
    If StringRegExp($sFilter, "[\\/:><\|]|(?s)\A\s*\z") Then Return SetError(2, 2, "")
    If Not ($iFlag = 0 Or $iFlag = 1 Or $iFlag = 2) Then Return SetError(3, 3, "")

    $sPath &= "|"
    While $sPath
        $sCurrPathLen = StringInStr($sPath, "|")
        $sCurrPath = StringLeft($sPath, $sCurrPathLen - 1) & "\"
        Local $search = FileFindFirstFile($sCurrPath & $sfilter)
        Switch $iFlag
            Case 0
                While 1; Files and Folders
                    $file = FileFindNextFile($search)
                    If @error Then ExitLoop
                    If $bRecursive And @extended Then $sPath &= $sCurrPath & $file & "|"
                    $sRet &= $sCurrPath & $file & "|"
                WEnd
            Case 1
                While 1; Files only
                    $file = FileFindNextFile($search)
                    If @error Then ExitLoop
                    If @extended Then
                        If $bRecursive Then $sPath &= $sCurrPath & $file & "|"
                    Else
                        $sRet &= $sCurrPath & $file & "|"
                    EndIf
                WEnd
            Case 2
                While 1; Folders only
                    $file = FileFindNextFile($search)
                    If @error Then ExitLoop
                    If @extended Then
                        If $bRecursive Then $sPath &= $sCurrPath & $file & "|"
                        $sRet &= $sCurrPath & $file & "|"
                    EndIf
                WEnd
        EndSwitch
        $sPath = StringTrimLeft($sPath, $sCurrPathLen)
    WEnd
    FileClose($search)
    Return StringSplit(StringTrimRight($sRet, 1), "|")
EndFunc ;==>_FLTA_NonRecursiveRecursion

It is 46 lines, compared to 39, but a tad faster.

Edit: It could be about 30 lines, but I think the while loops, that can be iterated 1000's of times, should be optimized for speed.

fixed filter edit

Edited July 7, 2009 by Spiff59

[Beege]

Are you sure its tad faster cause im not getting those results?

[SmOke_N]

I like this one (of course)

If StringRight($sPath, 1) <> "\" Then $sPath &= "\"

If StringRegExp($sFilter, "[\\/:<>|]") Or (Not StringStripWS($sFilter, 8)) Then Return SetError(2, 2, "")

I'm curious on your logic in only using part of the solutions I provided?

1. "[\\/ <\|]|(?s)\A\s*\z"

Should cover both non-standard characters, and your StringStripWS All

2. $sPath = StringRegExpReplace($sPath, "[\\]+\z", "") (should actually be: $sPath = StringRegExpReplace($sPath, "[\\/]+\z", "")

Covers if there are more than 1 forward or back slash at the end of the path.

To each their own I suppose.

[Spiff59]

I'm curious on your logic in only using part of the solutions I provided?

I just reused the edits I've had from a prior version.

I like the first one. Appending a slash to the end of the seocnd edit would work for me too.

I was just busy sticking the $iFlag parameter in and hadn't looked over your post.

[Spiff59]

Are you sure its tad faster cause im not getting those results?

I don't know.

I tested with this:

CODE

expandcollapse popup

include<array.au3>
#include<file.au3>

$Path = @SystemDir
;$Path = "C:\Program Files\Autoit3"
$Filter = "*"
$Flag = 1; 0 = files/folder, 1 = files only, 2 = folders only
$repeat = 25

$timer = TimerInit()
For $j = 1 to $repeat
    $x = _FLTA_NonRecursiveRecursion($Path, $Filter, $Flag, True)
Next
$t1 = TimerDiff ($timer)
_ArrayDisplay($x)

$timer = TimerInit()
For $j = 1 to $repeat
    $x = _FileListToArrayEx100000($Path, $Filter, $Flag, True)
Next
$t2 = TimerDiff ($timer)
_ArrayDisplay($x)

MsgBox (0, "", "$iFlag = " & $Flag & @CRLF &"Ver1:  " & Int($t1/10)/100 & @CRLF & "Ver2:  " & Int($t2/10)/100)

;===============================================================================
Func _FLTA_NonRecursiveRecursion($sPath, $sFilter, $iFlag, $bRecursive)
    Local $sRet = ""
    If Not FileExists($sPath) Then Return SetError(1, 1, "")
    If StringRight($sPath, 1) <> "\" Then $sPath &= "\"
    If StringRegExp($sFilter, "[\\/:<>|]") Or (Not StringStripWS($sFilter, 8)) Then Return SetError(2, 2, "")
    If Not ($iFlag = 0 Or $iFlag = 1 Or $iFlag = 2) Then Return SetError(3, 3, "")

    $sPath &= "|"
    While $sPath
        $sCurrPathLen = StringInStr($sPath, "|")
        $sCurrPath = StringLeft($sPath, $sCurrPathLen - 1) & "\"
        Local $search = FileFindFirstFile($sCurrPath & $sfilter)
        Switch $iFlag
            Case 0
                While 1; Files and Folders
                    $file = FileFindNextFile($search)
                    If @error Then ExitLoop
                    If $bRecursive And @extended Then $sPath &= $sCurrPath & $file & "|"
                    $sRet &= $sCurrPath & $file & "|"
                WEnd
            Case 1
                While 1; Files only
                    $file = FileFindNextFile($search)
                    If @error Then ExitLoop
                    If @extended Then
                        If $bRecursive Then $sPath &= $sCurrPath & $file & "|"
                    Else
                        $sRet &= $sCurrPath & $file & "|"
                    EndIf
                WEnd
            Case 2
                While 1; Folders only
                    $file = FileFindNextFile($search)
                    If @error Then ExitLoop
                    If @extended Then
                        If $bRecursive Then $sPath &= $sCurrPath & $file & "|"
                        $sRet &= $sCurrPath & $file & "|"
                    EndIf
                WEnd
        EndSwitch
        $sPath = StringTrimLeft($sPath, $sCurrPathLen)
    WEnd
    FileClose($search)
    Return StringSplit(StringTrimRight($sRet, 1), "|")
EndFunc  ;==>_FLTA_NonRecursiveRecursion

;===============================================================================
Func _FileListToArrayEx100000($sPath, $sFilter = "*", $iFlag = 0, $bRecursive = False)
    Local $sFile, $hSearch, $sWorkingdir, $asFilelist[1], $aFolderStack[2]
    If Not FileExists($sPath) Then Return SetError(1, 1, "")
    If (StringInStr($sFilter, "\")) Or (StringInStr($sFilter, "/")) Or (StringInStr($sFilter, ":")) Or (StringInStr($sFilter, ">")) Or (StringInStr($sFilter, "<")) Or (StringInStr($sFilter, "|")) Or (StringStripWS($sFilter, 8) = "") Then Return SetError(2, 2, "")
    If Not ($iFlag = 0 Or $iFlag = 1 Or $iFlag = 2) Then Return SetError(3, 3, "")
    If (StringMid($sPath, StringLen($sPath), 1) = "\") Then $sPath = StringTrimRight($sPath, 1); needed for Win98 for x:\  root dir
    $aFolderStack[0] = 1
    $aFolderStack[1] = $sPath
    While $aFolderStack[0] > 0
        $sWorkingdir = $aFolderStack[$aFolderStack[0]]
        $aFolderStack[0] -= 1
        $hSearch = FileFindFirstFile($sWorkingdir & '\' & $sFilter)
        While 1
            $sFile = FileFindNextFile($hSearch)
            If @error Then ExitLoop
            If Not @extended And $iFlag = 2 Then ContinueLoop
            If @extended Then
                If $bRecursive Then
                    $aFolderStack[0] += 1
                    If UBound($aFolderStack) <= $aFolderStack[0] Then ReDim $aFolderStack[UBound($aFolderStack) * 2]
                    $aFolderStack[$aFolderStack[0]] = $sWorkingdir & "\" & $sFile
                EndIf
                If $iFlag <> 1 Then
                    $asFilelist[0] += 1
                    If UBound($asFilelist) <= $asFilelist[0] Then ReDim $asFilelist[UBound($asFilelist) * 2]
                    $asFilelist[$asFilelist[0]] = $sWorkingdir & "\" & $sFile
                EndIf
            Else
                $asFilelist[0] += 1
                If UBound($asFilelist) <= $asFilelist[0] Then ReDim $asFilelist[UBound($asFilelist) * 2]
                $asFilelist[$asFilelist[0]] = $sWorkingdir & "\" & $sFile
            EndIf
        WEnd
        FileClose($hSearch)
    WEnd
    ReDim $asFilelist[$asFilelist[0] + 1]
    If UBound($asFilelist) = 1 Then Return SetError(4, 4, "")
    Return $asFilelist
EndFunc

[SmOke_N]

I don't know.

I tested with this:

My results:

$iFlag = 1
Ver1:  7.24
Ver2:  5.37

[Danny35d]

I don't know.

I tested with this:

Now change the $Filter value to *.au3 and you get the wrong result. Not files found for both scripts.

[Spiff59]

Now change the $Filter value to *.au3 and you get the wrong result. Not files found for both scripts.

The slash edit from the old version pasted-in wasn't appropriate any more and was resulting in $sPath being terminated with "'//". I swapped in Smoke_N's improved $sPath edit (as well as the $sFilter edit) and all is well . I'll update post 233.

My results:

CODE$iFlag = 1

Ver1: 7.24

Ver2: 5.37

That brings up something that I'm sure has been discussed: How flawed benchmarking via the system clock is.

I wish it were possible to actually accumulate total clock ticks used by a process, or somehow get a truer test of a functions overall CPU usage. I've noticed in testing this and other routines, that for some reason certain combinations of parameters and iterations can produce numbers that are in total opposition to the results returned by 99% of the benchmarks using other parms/iterations. Sometimes I'll get negative results from the same sort of timing loop which leaves me scratching my head. I just tend to assume the results that I get nine times out of ten are the more accurate. I'm consistantly getting test results for this script opposite of the one you posted.

Edit: The Filter is still broken, in both scripts. As Wus had stated, at this point it is just a proof of concept, not intended to be a finished product. I just wanted to offer an alternative route than all the Ubounds and Redims. The filter processing has to be a completely different animal with recursion than the production FLTA. Applying the filter to the initial FileFindFirstFile excludes directories that may contain matches. So you'd (unfortunately) have to have a second loop with a FileFindFirstFile and no filter, just to grab directories (like in post 175), or apply the filter test to each file returned by FileFindNextFile (ouch!). The only other alternative that comes to mind right now is to modify FileFindFirstFile to allow two separate filters. One for files, and one for folders. (I didn't just say that!)

Edited July 7, 2009 by Spiff59