Improvement of included _FileListToArray function.

Tlem · June 17, 2009

This is a modified version by Zedna and me to gain a few milliseconds and to return the full path of the files/folders (thanks Zedna).

Func _FileListToArray($sPath, $sFilter = "*", $iFlag = 0)
    Local $hSearch, $sFile, $sFileList, $asFileList[1]
    If Not FileExists($sPath) Then Return SetError(1, 1, "")
    If StringRegexp($sFilter, "[\\/:<>|]") Or (Not StringStripWS($sFilter, 8)) Then Return SetError(2, 2, "")
    If Not ($iFlag = 0 Or $iFlag = 1 Or $iFlag = 2 Or $iFlag = 4 Or $iFlag = 5 Or $iFlag = 6) Then Return SetError(3, 3, "")
    If (StringMid($sPath, StringLen($sPath), 1) = "\") Then $sPath = StringTrimRight($sPath, 1); needed for Win98 for x:\  root dir
    $hSearch = FileFindFirstFile($sPath & "\" & $sFilter)
    If $hSearch = -1 Then Return SetError(4, 4, "")
    While 1
        $sFile = FileFindNextFile($hSearch)
        If @error Then
            SetError(0)
            ExitLoop
        EndIf
        If $iFlag = 1 And StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") <> 0 Then ContinueLoop
        If $iFlag = 2 And StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") = 0 Then ContinueLoop
        If $iFlag > 3 Then $sFile = $sPath & "\" & $sFile
        $sFileList &= $sFile & "|"
    WEnd
    FileClose($hSearch)
    $asFileList = StringSplit(StringTrimRight($sFileList, 1), "|")
    Return $asFileList
EndFunc ;==>_FileListToArray

The gain of time isn't really significant (about 1,2 seconds for 260000 files) but if you use this function a lot, perhaps it can help you to gain few seconds.

Here is some tests results that I have done on my hardware.

Each test was doing 5 times (in a loop) and the result is the average of five values.

Here is the short description of the material that in served for the tests :

PC1 = Laptop IBM R60 C2D T5500 1.66 Ghz / 2.5Go Ram - Windows XP Pro SP3

PC2 = PC no name Quad Core Q6700 overcloked to 3Ghz / 4Go Ram - Windows Vista Pro SP1

Tests results :

PC1 test on 260000 files (stored on local drive) :

Original _FileListToArray = 5350 ms

Modified _FileListToArray = 4150 ms

PC1 test on 100000 files (stored on local drive) :

Original _FileListToArray = 2391 ms / 4075 ms (on battery)

Modified _FileListToArray = 1834 ms / 3150 ms (on battery)

PC1 test on 50000 files (stored on local drive) :

Original _FileListToArray = 1197 ms / 2002 ms (on battery)

Modified _FileListToArray = 924 ms / 1551 ms (on battery)

PC1 test on 50000 files (stored on NAS connected at 1Gb) :

Original _FileListToArray = 24578 ms

Modified _FileListToArray = 23843 ms

PC2 test on 100000 files (stored on local drive) :

Original _FileListToArray = 1307 ms

Modified _FileListToArray = 1091 ms

PC2 test on 50000 files (stored on local drive) :

Original _FileListToArray = 681 ms

Modified _FileListToArray = 575 ms

PC2 test on 50000 files (stored on NAS connected at 1Gb) :

Original _FileListToArray = 24635 ms

Modified _FileListToArray = 23447 ms

Edit :

There is a more great improvement on the last function, but it's for the newer version of AutoIt (Work time is divided by 4/5 for file search or folder search only).

I do an UDF with the too version but keep in mind that if you use only AutoIt > 3.3.1.0 then you can simplify to keep only the new version (that use @Extented from FileFindNextFile()).

; New version of FileListToArray (27/06/2009)
; #FUNCTION# ====================================================================================================


================
; Name...........: _FileListToArray
; Description ...: Lists files and\or folders in a specified path (Similar to using Dir with the /B Switch)
; Syntax.........: _FileListToArray($sPath[, $sFilter = "*"[, $iFlag = 0]])
; Parameters ....: $sPath   - Path to generate filelist for.
;                 $sFilter - Optional the filter to use, default is *. Search the Autoit3 helpfile for the word "WildCards" For details.
;                 $iFlag   - Optional: specifies whether to return files folders or both
;                 |$iFlag=0(Default) Return both files and folders
;                 |$iFlag=1 Return files only
;                 |$iFlag=2 Return Folders only
;                 |$iFlag=4 Return Folder/File names also with full path
; Return values .: @Error - 1 = Path not found or invalid
;                 |2 = Invalid $sFilter
;                 |3 = Invalid $iFlag
;                 |4 = No File(s) Found
; Author ........: SolidSnake <MetalGX91 at GMail dot com>
; Modified.......:
; Remarks .......: The array returned is one-dimensional and is made up as follows:
;                               $array[0] = Number of Files\Folders returned
;                               $array[1] = 1st File\Folder
;                               $array[2] = 2nd File\Folder
;                               $array[3] = 3rd File\Folder
;                               $array[n] = nth File\Folder
; Related .......:
; Link ..........;
; Example .......; Yes
; ====================================================================================================


===========================
;Special Thanks to Helge and Layer for help with the $iFlag update
;===============================================================================
Func _FileListToArray($sPath, $sFilter = "*", $iFlag = 0)
    If @AutoItVersion > "3.3.1.0" Then
        _FileListToArrayNew($sPath, $sFilter, $iFlag)
    Else
        _FileListToArrayOld($sPath, $sFilter, $iFlag)
    EndIf
EndFunc ;==>_FileListToArray

; For versions of AutoIt > 3.3.1.0 (Use of @extented in FileFindNextFile()
Func _FileListToArrayNew($sPath, $sFilter = "*", $iFlag = 0)
    Local $hSearch, $sFile, $sFileList, $sDelim = "|"

    If Not FileExists($sPath) Then Return SetError(1, 1, "")
    If StringRegExp($sFilter, "[\\/:<>|]") Or (Not StringStripWS($sFilter, 8)) Then Return SetError(2, 2, "")
    If StringRight($sPath, 1) <> "\" Then $sPath &= "\"

    If $iFlag > 3 Then
        $sDelim &= $sPath
        $iFlag -= 4
    EndIf

    $hSearch = FileFindFirstFile($sPath & $sFilter)
    If $hSearch = -1 Then
        Return SetError(4, 4, "")
    EndIf

    Switch $iFlag
        Case 0; Files and Folders
            While 1
                $sFile = FileFindNextFile($hSearch)
                If @error Then
                    ExitLoop
                EndIf
                $sFileList &= $sDelim & $sFile
            WEnd
        Case 1; Files Only
            While 1
                $sFile = FileFindNextFile($hSearch)
                If @error Then
                    ExitLoop
                EndIf
        ; bypass folder
                If @extended Then ContinueLoop; This version for new beta
                $sFileList &= $sDelim & $sFile
            WEnd
        Case 2; Folders Only
            While 1
                $sFile = FileFindNextFile($hSearch)
                If @error Then
                    ExitLoop
                EndIf
        ; bypass file
                If @extended = 0 Then ContinueLoop; This version for new beta
                $sFileList &= $sDelim & $sFile
            WEnd
        Case Else
            Return SetError(3, 3, "")
    EndSwitch

    FileClose($hSearch)
    Return StringSplit(StringTrimLeft($sFileList, 1), "|")
EndFunc ;==>_FileListToArrayNew

; For versions of AutoIt < 3.3.1.1
Func _FileListToArrayOld($sPath, $sFilter = "*", $iFlag = 0)
    Local $hSearch, $sFile, $sFileList, $sDelim = "|"

    If Not FileExists($sPath) Then Return SetError(1, 1, "")
    If StringRegExp($sFilter, "[\\/:<>|]") Or (Not StringStripWS($sFilter, 8)) Then Return SetError(2, 2, "")
    If StringRight($sPath, 1) <> "\" Then $sPath &= "\"

    If $iFlag > 3 Then
        $sDelim &= $sPath
        $iFlag -= 4
    EndIf

    $hSearch = FileFindFirstFile($sPath & $sFilter)
    If $hSearch = -1 Then
        Return SetError(4, 4, "")
    EndIf

    Switch $iFlag
        Case 0; Files and Folders
            While 1
                $sFile = FileFindNextFile($hSearch)
                If @error Then ExitLoop
                $sFileList &= $sDelim & $sFile
            WEnd
        Case 1; Files Only
            While 1
                $sFile = FileFindNextFile($hSearch)
                If @error Then ExitLoop
        ; bypass folder
                If StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") <> 0 Then ContinueLoop
                $sFileList &= $sDelim & $sFile
            WEnd
        Case 2; Folders Only
            While 1
                $sFile = FileFindNextFile($hSearch)
                If @error Then ExitLoop
        ; bypass file
                If StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") = 0 Then ContinueLoop
                $sFileList &= $sDelim & $sFile
            WEnd
        Case Else
            Return SetError(3, 3, "")
    EndSwitch

    FileClose($hSearch)
    Return StringSplit(StringTrimLeft($sFileList, 1), "|")
EndFunc ;==>_FileListToArrayOld

New Edit :

A new version with recursive search, multiple search, exclusion of files/directory is in progress (see the last posts).

Thanks to BaKaMu.

Edited June 28, 2009 by Tlem

jpm · June 18, 2009

Good job old chaps(toi & Zedna)

You could have given the total execution before on the 260000 files so we can know what is the percentage gain.

1,2 sec on ???

jvanegmond · June 18, 2009

Related: http://www.autoitscript.com/forum/index.php?showtopic=96776

jpm · June 18, 2009

Manadar said:
Related: http://www.autoitscript.com/forum/index.php?showtopic=96776

Not sure the 1,2 sec is really described.

Your link should go in the first post if relevant

jvanegmond · June 18, 2009

It's related to the entire thread, not specifically what you asked in your post. Just doing this for future reference. : )

Tlem · June 18, 2009

Adding tests results on my first post.

jpm · June 18, 2009

Tlem said:
Adding tests results on my first post.

That what I really want to know

edit just missing CPU speed for PC1 and PC2 I can't believe Vista is faster than XP ...

Edited June 18, 2009 by jpm

Spiff59 · June 18, 2009

If you're after speed, I would think you need only execute the BitAND and a portion of the string statement once, like below. Adding a Switch statement eliminates tons of compares. Between 520,000 and 780,000 less compares in your 260,000 file example, depending on the value of $iFlag.

#include<file.au3>
#include<array.au3>
$iflag = 0

; Beta Version -------------------------------
$timer = TimerInit()
For $j = 1 to 100
    $x = _FileListToArray(@SystemDir,"*",$iflag)
Next
$timer1 = TimerDiff ($timer)

; ----------------------
$timer = TimerInit()
For $j = 1 to 100
    $x = _FileListToArray3(@SystemDir,"*",$iflag)
Next
$timer2 = TimerDiff ($timer) 

; ----------------------
$timer = TimerInit()
For $j = 1 to 100
    $x = _FileListToArray4(@SystemDir,"*",$iflag)
Next
$timer3 = TimerDiff ($timer) 

MsgBox (0, "", $timer1 & @CRLF & $timer2 & @CRLF & $timer3)
;_ArrayDisplay($x)

; ------------------------------------------------
Func _FileListToArray3($sPath, $sFilter = "*", $iFlag = 0)
    Local $hSearch, $sFile, $sFileList, $asFileList[1]
    If Not FileExists($sPath) Then Return SetError(1, 1, "")
    If (StringInStr($sFilter, "\")) Or (StringInStr($sFilter, "/")) Or (StringInStr($sFilter, ":")) Or (StringInStr($sFilter, ">")) Or (StringInStr($sFilter, "<")) Or (StringInStr($sFilter, "|")) Or (StringStripWS($sFilter, 8) = "") Then Return SetError(2, 2, "")
    If Not ($iFlag = 0 Or $iFlag = 1 Or $iFlag = 2 Or $iFlag = 4) Then Return SetError(3, 3, "")
    If (StringMid($sPath, StringLen($sPath), 1) = "\") Then $sPath = StringTrimRight($sPath, 1) ; needed for Win98 for x:\  root dir
    $hSearch = FileFindFirstFile($sPath & "\" & $sFilter)
    If $hSearch = -1 Then Return SetError(4, 4, "")
    While 1
        $sFile = FileFindNextFile($hSearch)
        If @error Then
            SetError(0)
            ExitLoop
        EndIf
        If $iFlag = 1 And StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") <> 0 Then ContinueLoop
        If $iFlag = 2 And StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") = 0 Then ContinueLoop
        If BitAND($iFlag, 4) Then $sFile = $sPath & "\" & $sFile
        $sFileList &= $sFile & "|"
    WEnd
    FileClose($hSearch)
    $asFileList = StringSplit(StringTrimRight($sFileList, 1), "|")
    Return $asFileList
EndFunc   ;==>_FileListToArray

; ------------------------------------------------
Func _FileListToArray4($sPath, $sFilter = "*", $iFlag = 0)
    Local $hSearch, $sFile, $sFileList, $sDelim = "|"
    If Not FileExists($sPath) Then Return SetError(1, 1, "")
    If StringRegexp($sFilter, "[\\/:<>|]") Or (Not StringStripWS($sFilter, 8)) Then Return SetError(2, 2, "") 
    If (StringMid($sPath, StringLen($sPath), 1) = "\") Then $sPath = StringTrimRight($sPath, 1) ; needed for Win98 for x:\  root dir
    $hSearch = FileFindFirstFile($sPath & "\" & $sFilter)
    If $hSearch = -1 Then Return SetError(4, 4, "")
    If $iFlag > 3 Then 
        $sDelim &= $sPath & "\" 
        $iflag -= 4
    EndIf
    Switch $iFlag
        Case 0; Files and Folders
            While 1
                $sFile = FileFindNextFile($hSearch)
                If @error Then ExitLoop
                $sFileList &= $sDelim & $sFile
            WEnd
        Case 1; Files Only
            While 1
                $sFile = FileFindNextFile($hSearch)
                If @error Then ExitLoop
                If @extended Then ContinueLoop ; bypass folder
                $sFileList &= $sDelim & $sFile
            WEnd
        Case 2; Folders Only
            While 1
                $sFile = FileFindNextFile($hSearch)
                If @error Then ExitLoop
                If @extended = 0 Then ContinueLoop  ; bypass file
                $sFileList &= $sDelim & $sFile
            WEnd
        Case Else
            Return SetError(3, 3, "")
    EndSwitch
    FileClose($hSearch)
    Return StringSplit(StringTrimLeft($sFileList, 1), "|")
EndFunc   ;==>_FileListToArray

Edit: Oops, I should have read the referenced thread first <sheepish grin>

Edit2: Swapped out the exisitng BitAND statement for a simple "> 3" test. Same result, but easier to read.

Edit3: Removed unnecessary $asFileList[] variable.

Edited June 19, 2009 by Spiff59

Tlem · June 19, 2009

Sorry guy, I have modified my first post because I had forgotten something important.

Shame on me.

In this function the flag 4 where added to have the full path of files/directory. So when we use this function flag can be 4 (for files and directory), 4 + 1 (5 for files only) or 4 + 2 (6 for directory only) and not only 4...

So I made the modification on the function.

Just for information, I do not use any more the BitAnd condition to test $iFlag, because it makes lose approximately 40ms for 50000 files.

Because of the first test : If Not ($iFlag = 0 Or $iFlag = 1 Or $iFlag = 2 Or $iFlag = 4 Or $iFlag = 5 Or $iFlag = 6) Then Return SetError(3, 3, "")

the value can be only 0, 1, 2, 4, 5, 6 so if the value is > 3 then that signify that we want full path.

For jpm, I have added the speed of my hardware on test result.

Spiff59 · June 19, 2009

Tlem said:
Sorry guy, I have modified my first post because I had forgotten something important.
Shame on me.
In this function the flag 4 where added to have the full path of files/directory. So when we use this function flag can be 4 (for files and directory), 4 + 1 (5 for files only) or 4 + 2 (6 for directory only) and not only 4...
So I made the modification on the function.
Just for information, I do not use any more the BitAnd condition to test $iFlag, because it makes lose approximately 40ms for 50000 files.
Because of the first test : If Not ($iFlag = 0 Or $iFlag = 1 Or $iFlag = 2 Or $iFlag = 4 Or $iFlag = 5 Or $iFlag = 6) Then Return SetError(3, 3, "")
the value can be only 0, 1, 2, 4, 5, 6 so if the value is > 3 then that signify that we want full path.
For jpm, I have added the speed of my hardware on test result.

The example I'd posted works exactly the same as the first post. $iFlag must be between 0 and 6 or it returns @error=3. If it is >3 then the full path is returned. The only difference it that it is much faster than the other examples.

Tlem · June 19, 2009

Sorry, I haven't seen your example _FileListToArray4.

For the StringRegexp, it was what I want to do.

I have tested with StringRegexp and we won 10ms.

Not a so big improvement, but as long as to make things, so much to make them well.

I have modified the code of the first post to take into account this modification.

Edited June 19, 2009 by Tlem

trancexx · June 19, 2009

Tlem said:
Sorry, I haven't seen your example _FileListToArray4.
For the StringRegexp, it was what I want to do.

I have tested with StringRegexp and we won 10ms.
Not a so big improvement, but as long as to make things, so much to make them well.

I have modified the code of the first post to take into account this modification.

You are almost missing the point of Spiff59's modifications.

His function is by far the fastest in my testings.

And if he would replace:

If ... Then...

to:

If... Then
    ...
EndIf

it should be even faster.

Tlem · June 19, 2009

According to all suggestions, this is a new version :

Func _FileListToArray($sPath, $sFilter = "*", $iFlag = 0)
    Local $hSearch, $sFile, $sFileList, $asFileList[1]
    If Not FileExists($sPath) Then
        Return SetError(1, 1, "")
    EndIf
    If StringRegExp($sFilter, "[\\/:<>|]") Or (Not StringStripWS($sFilter, 8)) Then
        Return SetError(2, 2, "")
    EndIf
    If Not ($iFlag = 0 Or $iFlag = 1 Or $iFlag = 2 Or $iFlag = 4 Or $iFlag = 5 Or $iFlag = 6) Then
        Return SetError(3, 3, "")
    EndIf
    If (StringMid($sPath, StringLen($sPath), 1) = "\") Then
        $sPath = StringTrimRight($sPath, 1); needed for Win98 for x:\  root dir
    EndIf
    $hSearch = FileFindFirstFile($sPath & "\" & $sFilter)
    If $hSearch = -1 Then
        Return SetError(4, 4, "")
    EndIf

    While 1
        $sFile = FileFindNextFile($hSearch)
        If @error Then
            SetError(0)
            ExitLoop
        EndIf
        Switch $iFlag
            Case 1
                If StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") <> 0 Then
                    ContinueLoop
                EndIf

            Case 2
                If StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") = 0 Then
                    ContinueLoop
                EndIf

            Case 4
                $sFile = $sPath & "\" & $sFile

            Case 5
                If StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") <> 0 Then
                    ContinueLoop
                EndIf
                $sFile = $sPath & "\" & $sFile

            Case 6
                If StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") = 0 Then
                    ContinueLoop
                EndIf
                $sFile = $sPath & "\" & $sFile
        EndSwitch
        $sFileList &= $sFile & "|"
    WEnd
    FileClose($hSearch)
    $asFileList = StringSplit(StringTrimRight($sFileList, 1), "|")
    Return $asFileList
EndFunc  ;==>_FileListToArray

This one seems to be more faster than the previous version.

On my test of 100000 files this is the result :

Original _FileListToArray = 2335ms

Modified _FileListToArray = 1591ms

I did not know that :

If... Then
    ...
EndIf

is faster than

If... Then

.

This evening, I would go to bed less stupid.

Thank you very much trancexx.

Edited June 19, 2009 by Tlem

Spiff59 · June 19, 2009

Since the vaue of $iFlag does not change during a call to the routine, I don't see the point in testing it's value potentially hundreds of thousands of times. It ought to be outside the while loop, in my opinion. I believe that yeilds the fastest results.

trancexx said:
And if he would replace:
CODEIf ... Then...to:
CODEIf... Then
...
EndIfit should be even faster.

That is surprising. One would think behind the scenes it would generate the same code. I'll see where I can make those changes... Edited June 19, 2009 by Spiff59

jpm · June 19, 2009

@Tlem you miss the point that using @extended do the same as StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") in a much faster way

Quote
NbFiles $iflag Version Tlem Spiff59
281100 0 3.3.0.0 2845 19.09% 1944 44.73%
281100 0 3.3.1.1 2907 20.12% 1980 45.58%
281100 4 3.3.0.0 4284 0.00% 2172 49.31%
281100 4 3.3.1.1 4394 0.00% 2209 49.72%

the percentage are calculated against Std UDF, for $iflag=4 the percentage is against "Tlem" version

Zedna · June 19, 2009

trancexx said:
And if he would replace:
If ... Then...
to:
If... Then
    ...
EndIf
it should be even faster.

Are you sure about this?

I haven't heard about it and I see no reason what such behaviour.

Zedna · June 19, 2009

jpm said:
@Tlem you miss the point that using @extended do the same as StringInStr(FileGetAttrib($sPath & "\" & $sFile), "D") in a much faster way

But this is only in latest Beta version (not in release 3.3).

Tlem · June 19, 2009

Effectively, the Spiff59 function is quicker than my last proposition, but like Zedna said, we must have the last version.

I'll take a look tomorrow, because now it's party time.

Edited June 19, 2009 by Tlem

KaFu · June 19, 2009

Zedna said:
Are you sure about this?

I've read somewhere that switch case is faster then if then...

Melba23 · June 19, 2009

KaFu,

Looks like Switch is a little slower - on my machine at least:

$fred = 0

$begin = TimerInit()
For $i = 1 To 1000000
    If $fred = 1 Then
        ConsoleWrite("fred" & @CRLF)
    EndIf
Next
ConsoleWrite(Timerdiff($begin) & @CRLF)

$begin = TimerInit()
For $i = 1 To 1000000

    If $fred = 1 Then ConsoleWrite("fred" & @CRLF)

Next
ConsoleWrite(Timerdiff($begin) & @CRLF)

$begin = TimerInit()
For $i = 1 To 1000000
    Switch $fred
        Case 1
            ConsoleWrite("fred" & @CRLF)
    EndSwitch
Next
ConsoleWrite(Timerdiff($begin) & @CRLF)

; Results:
; 1156.37022303114
; 890.749103587188
; 992.363624427127

M23

Sign In

Improvement of included _FileListToArray function.

Recommended Posts

Tlem

Top Posters In This Topic

Top Posters In This Topic

jpm

jvanegmond

jpm

jvanegmond

Tlem

jpm

Spiff59

Tlem

Spiff59

Tlem

trancexx

Tlem

Spiff59

jpm

Zedna

Zedna

Tlem

KaFu

Melba23

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members

Browse

AutoIt Resources

Release

Beta