Improvement of included _FileListToArray function.

[BaKaMu]

I'll acceed to the philosophy of allowing bad parameters to result in a no-return from the routine, but...

The $bRecursive line above is needed. Two different posters in this thread have already posted examples where they have used "1" for True, or "0" for False. It doesn't seem right to not accept those values for a boolean.

I agree there is action needed. I will see

Does putting "Local" in front of it cause the variable to be destroyed and recreated each loop? Is it more efficient to define it once at the top?

My tests produces no time difference between declaration on top and then assign in code or declaration and assign in code.

I doesn't matter.

Actually i spend a lot more time in posting then coding. ;-)

I will take a timeout now!

[Spiff59]

This would be one method of cleaning up the semi-colon problem"

StringReplace(StringStripWS(StringReplace($s, ";", @CR), 7), @CR, ";")

Any "dummy" control character could be used in place of @CR.

I'm trying to work out a proper StringRegExpReplace to see if it would be faster.

But I'm no SRER expert, so it could take a while...

Anyone?

Edit: The example above isn't going to work. It will pull extra spaces out of pathnames that may be necessary if searching for a folder oddly named something like "New <lots of spaces> Folder"

Edit2: This mod of the existing edit, that removes spaces next to the semi-colon, will also handle removing adjacent semi-colons:

$s = StringRegExpReplace($s, "(\s*;\s*)+", ";")

Just need to work the leading and trailing semi-colon removal into it.

Edit3: This is my best effort, a two-SRER solution that would replace the existing "spaces" edit:

$s = StringRegExpReplace(StringRegExpReplace($s, "(\s*;\s*)+", ";"), "\A;|;\z", "")

Edited July 3, 2009 by Spiff59

[BaKaMu]

This would be one method of cleaning up the semi-colon problem"

StringReplace(StringStripWS(StringReplace($s, ";", @CR), 7), @CR, ";")

Any "dummy" control character could be used in place of @CR.

I'm trying to work out a proper StringRegExpReplace to see if it would be faster.

But I'm no SRER expert, so it could take a while...

Anyone?

Thanks, i'll try it.

But now ... work break ...

[BaKaMu]

Edit3: This is my best effort, a two-SRER solution that would replace the existing "spaces" edit:

$s = StringRegExpReplace(StringRegExpReplace($s, "(\s*;\s*)+", ";"), "\A;|;\z", "")

Well done, you have been a little bit faster than i (same idea, but my attempt was If StringRight/Left ...)

Your code is more elegant (for path and filter)

Again back to ... Posting break ... :-) :-)

[Spiff59]

Well done, you have been a little bit faster than i (same idea, but my attempt was If StringRight/Left ...)

Your code is more elegant (for path and filter)

Again back to ... Posting break ... :-) :-)

What is cool is that:

$s = StringRegExpReplace(StringRegExpReplace($s, "(\s*;\s*)+", ";"), "\A;|;\z", "")

is almost the same speed as the existing:

$s = StringStripWS(StringRegExpReplace($s, "\s*;\s*", ";"), 3)

but it a better edit.

You mention path and filter... the same edit also exists for $sExclude.

Here's my test script if you want to play with it:

CODE

Local $s = "  ;;;C:\Windows;;;;  C:\RECYCLER;;C:\Temp     Folder;   ", $s1, $s2
$Repeat = 10000

;----------------------------------------------
$t1 = TimerInit();  old edit
For $x = 1 To $Repeat
    $s1 = StringStripWS(StringRegExpReplace($s, "\s*;\s*", ";"), 3)
Next
$t1 = TimerDiff($t1)

;----------------------------------------------
$t2 = TimerInit(); new edit
For $x = 1 To $Repeat
    $s2 = StringRegExpReplace(StringRegExpReplace($s, "(\s*;\s*)+", ";"), "\A;|;\z", "")
Next
$t2 = TimerDiff($t2)

;----------------------------------------------
$l = ">" & $s & "<" & @CRLF
$l1 = ">" & $s1 & "<" & @CRLF
$l2 = ">" & $s2 & "<" & @CRLF
$l4 = @CRLF & "Old time: " & Int($t1/10)/100 & "    New time: " & Int($t2/10)/100 & @CRLF
MsgBox(1,">IN<, >OLD<, >NEW<", $l & $l1 & $l2 & $l4)

Edit: I got the day off as a holiday, so have plenty of time to test... Edited July 3, 2009 by Spiff59

[BaKaMu]

Edit3: This is my best effort, a two-SRER solution that would replace the existing "spaces" edit:

$s = StringRegExpReplace(StringRegExpReplace($s, "(\s*;\s*)+", ";"), "\A;|;\z", "")

OK, your code works pretty well, but i found a new problem, that has to be discussed

If the path is given as "" or with your codeline for example ";;;;" is stripped to ""

and filter for example is "*" then FilterFileFindFirstFile/NextFile returns all files/folders of Root-Dir.

(It's again MS-Windows that is inconsequentially)

Therefore I would suggest:

After

$sPath = StringRegExpReplace(StringRegExpReplace($sPath, "(\s*;\s*)+", ";"), "\A;|;\z", "")

$sFilter = StringRegExpReplace(StringRegExpReplace($sFilter, "(\s*;\s*)+", ";"), "\A;|;\z", "")

we should test on "" (path and filter in the firstcall part) and exit function if so, so no harm can appear.

What do you think about this recommendation?

Edit1: maybe exit function with error 1 or 2 ??

Edit2: or recommendation:

If $sPath = "" Then Return SetError(1, 1, "")

If $sFilter = "" Then Return SetError(2, 2, "")

Edited July 3, 2009 by BaKaMu

[BaKaMu]

What is cool is that:

$s = StringRegExpReplace(StringRegExpReplace($s, "(\s*;\s*)+", ";"), "\A;|;\z", "")

is almost the same speed as the existing:

$s = StringStripWS(StringRegExpReplace($s, "\s*;\s*", ";"), 3)

but it a better edit.

Yes, cool, but time is not a problem here, cause this is only called once (in firstcall part)

Nevertheless good solution

You mention path and filter... the same edit also exists for $sExclude.

Yes of course, i will implement this.

Edit: I got the day off as a holiday, so have plenty of time to test...

Nothing better to do on holiday? :-))

[Spiff59]

Yes, cool, but time is not a problem here, cause this is only called once (in firstcall part)

Nevertheless good solution

Yes of course, i will implement this.

Nothing better to do on holiday? :-))

How about?

If $sWorkPath= "/*firstcall*/" Then
    If Not $sPath Then Return SetError(4, 4, "")

I mean, if you asked for it to search no paths, then it's still a "not find" situation?

I'm still feeling a little rough from gong out last night, and we are planning on going to a day-early fireworks show tinight.

We'll need a new comment. Those cryptic SRE's and SRER's need good ones. Maybe?

;strip leading/trailing spaces and semi-colons, all adjacent semi-colons, and spaces surrounding semi-colons

PS: I'm still voting for the $bRecursive handling

Edited July 3, 2009 by Spiff59

[BaKaMu]

How about?

If $sWorkPath= "/*firstcall*/" Then
    If Not $sPath Then Return SetError(4, 4, "")

See post #166, error 1 and 2 is compatible to _FileListToArray

...we are planning on going to a day-early fireworks show tinight.

Very enviable, i love fireworks!

We'll need a new comment. Those cryptic SRE's and SRER's need good ones. Maybe?

;strip leading/trailing spaces and semi-colons, all adjacent semi-colons, and spaces surrounding semi-colons

Sounds good.

[Spiff59]

Well, I'm still voting for the $bRecursive edit that allows 0 and 1, but otherwise, unless testing reveals more problems, I'm about ready to say we should tell jpm to "Come and get it!"

Which error number for the blank path doesn't much matter to me, I think a blank path also qualifies as an error 4 as well though.

Edit: Well, let's see what we got first I guess, then think about getting jpm. Since you're the unofficial keeper of the latest version... bring it forth! (when you get the time)

Edit2: And you're right, the "If Not $sPath" test goes after the path edit in case SRER strips ";;;" down to nothing.

Edited July 3, 2009 by Spiff59

[BaKaMu]

Well, I'm still voting for the $bRecursive edit that allows 0 and 1, but otherwise, unless testing reveals more problems, I'm about ready to say we should tell jpm to "Come and get it!"

If have it done this pragmatical way:

If $bRecursive = -1 Or $bRecursive = Default Then $bRecursive = False

If $bRecursive = 1 Then $bRecursive = True

OK?

Edit1: There is a logical error, i work on it.

Edit2: Oh there is always the same error, i have forgotten that True = -1

So better:

If $bRecursive = Default Then $bRecursive = False

If $bRecursive = 1 Then $bRecursive = True

Edited July 3, 2009 by BaKaMu

[Spiff59]

Doesn't this function correctly?

If $bRecursive = "1" Then
      $bRecursive = True
    Else
      $bRecursive = False
    EndIf

If the input is 1, "1" or True it returns True, everything else goes to the default of False

[BaKaMu]

Doesn't this function correctly?

If $bRecursive = "1" Then
      $bRecursive = True
    Else
      $bRecursive = False
    EndIf

If the input is 1, "1" or True it returns True, everything else goes to the default of False

Yes it works, and could be shortened to

$bRecursive = ($bRecursive = "1") ; on line, :-)

I'll take over to function.

Edited July 3, 2009 by BaKaMu

[Spiff59]

I tested the script from post #136 and Flag 0 should return 5, Flag 1 should return 5 and Flag 2 should return 0 instead they return 15, 10 and 5.

Unlike the existing _FileListToArray(), you could get the results you want with this version by adding an exclude parameter

$aFileList = _FileListToArrayNT6(@ScriptDir & "\testing", "*.dat", 0, 1, True, "*.dat?")

Edited July 3, 2009 by Spiff59

[BaKaMu]

Edit: Well, let's see what we got first I guess, then think about getting jpm. Since you're the unofficial keeper of the latest version... bring it forth!

OK that's the last version of _FileListToArrayNT7 (better known as _FileListToArraySuperTurbo4000PlusProWithMaximumUltraPowerboost)

Maybe the final version, so please check extensively ;-)

(As usual, close to deadline there is a hectic atmosphere :-))

expandcollapse popup

; #FUNCTION# ===========================================================================================
; Name:             _FileListToArrayNT7
; Description:      Lists files and\or folders in specified path(s) (Similar to using Dir with the /B Switch)
;                    additional features: multi-path, multi-filter, multi-exclude-filter, path format options, recursive search
; Syntax:           _FileListToArrayNT7([$sPath = @ScriptDir, [$sFilter = "*", [$iSearchType, [$bRecursive = False, [$sExclude = "", [$iRetFormat = 1]]]]]])
; Parameter(s):     $sPath = optional: Search path(s), semicolon delimited (default: @ScriptDir)
;                            (Example: "C:\Tmp;D:\Temp")
;                   $sFilter = optional: Search filter(s), semicolon delimited . Wildcards allowed. (default: "*")
;                              (Example: "*.exe;*.txt")
;                   $iSearchType = Include in search: 0 = Files and Folder, 1 = Files Only, 2 = Folders Only
;                   $iPathType = Returned element format: 0 = file/folder name only, 1 = relative path, 2 = full path
;                   $bRecursive = optional: True: recursive search including all subdirectories
;                                           False (default): search only in specified folder
;                   $sExclude = optional: Exclude filter(s), semicolon delimited. Wildcards allowed.
;                               (Example: "Unins*" will remove all files/folders that begin with "Unins")
;                   $iRetFormat =  optional: return format
;                                  0 = one-dimensional array, 0-based
;                                  1 = one-dimensional array, 1-based (default)
;                                  2 = String ( "|" delimited)
;                   $sWorkPath =   *** internal use only (do not use) ***
; Requirement(s):   none
; Return Value(s):  on success: 1-based or 0-based array or string (dependent on $iRetFormat)
;                   If no path is found, @error and @extended are set to 1, returns empty string
;                   If no filter is found, @error and @extended are set to 2, returns empty string
;                   If no data is found, @error and @extended are set to 4, returns empty string
; Author(s):        Half the AutoIt Community
; ====================================================================================================
Func _FileListToArrayNT7($sPath = @ScriptDir, $sFilter = "*", $iSearchType = 0, $iPathType = 0, $bRecursive = False, $sExclude = "", $iRetFormat = 1, $sWorkPath = "(-:>firstcall<:-)")
  Local $hSearch, $iPCount, $iFCount, $sFile, $sFileList, $sTWorkPath

  If $sWorkPath == "(-:>firstcall<:-)" Then
    If $sPath = -1 Or $sPath = Default Then $sPath = @ScriptDir
    ;strip leading/trailing spaces and semi-colons, all adjacent semi-colons, and spaces surrounding semi-colons
    $sPath = StringRegExpReplace(StringRegExpReplace($sPath, "(\s*;\s*)+", ";"), "\A;|;\z", "")
    ;check that at least one path is set
    If $sPath = "" Then Return SetError(1, 1, "")
    ;-----
    If $sFilter = -1 Or $sFilter = Default Then $sFilter = "*"
    ;strip leading/trailing spaces and semi-colons, all adjacent semi-colons, and spaces surrounding semi-colons
    $sFilter = StringRegExpReplace(StringRegExpReplace($sFilter, "(\s*;\s*)+", ";"), "\A;|;\z", "")
    ;check that at least one filter is set
    If $sFilter = "" Then Return SetError(2, 2, "")
    ;-----
    If $iSearchType <> "1" and $iSearchType <> "2" Then $iSearchType = "0"
    ;-----
    If $iPathType <> "1" And $iPathType <> "2" Then $iPathType = "0"
    ;-----
    $bRecursive = ($bRecursive = "1")
    ;-----
    If $sExclude = -1 Or $sExclude = Default Then $sExclude = ""
    If $sExclude Then
      ;prepare $sExclude
      ;strip leading/trailing spaces and semi-colons, all adjacent semi-colons, and spaces surrounding semi-colons
      $sExclude = StringRegExpReplace(StringRegExpReplace($sExclude, "(\s*;\s*)+", ";"), "\A;|;\z", "")
      ;convert $sExclude to fit StringRegExp (not perfect but useable)
      $sExclude = StringRegExpReplace($sExclude, '([\Q\.+[^]$(){}=!\E])', '\\$1')
      $sExclude = StringReplace($sExclude, "?", ".")
      $sExclude = StringReplace($sExclude, "*", ".*?")
      $sExclude = "(?i)\A" & StringReplace($sExclude, ";", "|") & "\z"
    EndIf
    ;-----
    If $iRetFormat <> "0" And $iRetFormat <> "2" Then $iRetFormat = "1"

    $sWorkPath = ""
  EndIf

  Local $aPath = StringSplit($sPath, ';', 1) ;paths array
  Local $aFilter = StringSplit($sFilter, ';', 1) ;filters array

  If $sExclude Then

    For $iPCount = 1 To $aPath[0] ;Path loop
      Local $sDelim = "|" ;reset $sDelim

      If StringRight($aPath[$iPCount], 1) <> "\" Then $aPath[$iPCount] &= "\" ;ensure trailing slash
      If $iPathType = 2 Then $sDelim &= $aPath[$iPCount] ;return full-path

      For $iFCount = 1 To $aFilter[0] ;filter loop
        If StringRegExp($aFilter[$iFCount], "[\\/:<>|]") Then ContinueLoop ;bypass filters with invalid chars
        $hSearch = FileFindFirstFile($aPath[$iPCount] & $aFilter[$iFCount])
        If @error Then ContinueLoop
        Switch $iSearchType
          Case 1 ;files Only
            While True
              $sFile = FileFindNextFile($hSearch)
              If @error Then ExitLoop
              If @extended Then ContinueLoop ;bypass folder
              ;check for exclude files
              If StringRegExp($sFile, $sExclude) Then ContinueLoop
              $sFileList &= $sDelim & $sWorkPath & $sFile
            WEnd
          Case 2 ;folders Only
            While True
              $sFile = FileFindNextFile($hSearch)
              If @error Then ExitLoop
              If @extended Then ;bypass file
                ;check for exclude folder
                If StringRegExp($sFile, $sExclude) Then ContinueLoop
                $sFileList &= $sDelim & $sWorkPath & $sFile
              EndIf
            WEnd
          Case Else ;files and folders
            While True
              $sFile = FileFindNextFile($hSearch)
              If @error Then ExitLoop
              ;check for exclude files/folder
              If StringRegExp($sFile, $sExclude) Then ContinueLoop
              $sFileList &= $sDelim & $sWorkPath & $sFile
            WEnd
        EndSwitch
        FileClose($hSearch)
      Next ;$iFCount - next filter

      ;---------------

      ;optional do a recursive search
      If $bRecursive Then
        $hSearch = FileFindFirstFile($aPath[$iPCount] & "*.*")
        If Not @error Then
          While True
            $sFile = FileFindNextFile($hSearch)
            If @error Then ExitLoop
            If @extended Then ;bypass file
              ;check for exclude folder
              If StringRegExp($sFile, $sExclude) Then ContinueLoop
              ;call recursive search
              If $iPathType = 1 Then $sTWorkPath = $sWorkPath & $sFile & "\"
              $sFileList &= _FileListToArrayNT7($aPath[$iPCount] & $sFile & "\", $sFilter, $iSearchType, $iPathType, $bRecursive, $sExclude, 2, $sTWorkPath)
            EndIf
          WEnd
          FileClose($hSearch)
        EndIf
      EndIf

    Next ;$iPCount - next path

  Else ;If Not $sExclude

    For $iPCount = 1 To $aPath[0] ;path loop
      Local $sDelim = "|" ;reset $sDelim

      If StringRight($aPath[$iPCount], 1) <> "\" Then $aPath[$iPCount] &= "\" ;ensure trailing slash
      If $iPathType = 2 Then $sDelim &= $aPath[$iPCount] ;return full-path

      For $iFCount = 1 To $aFilter[0] ;filter loop
        If StringRegExp($aFilter[$iFCount], "[\\/:<>|]") Then ContinueLoop ;bypass filters with invalid chars
        $hSearch = FileFindFirstFile($aPath[$iPCount] & $aFilter[$iFCount])
        If @error Then ContinueLoop
        Switch $iSearchType
          Case 1 ;files Only
            While True
              $sFile = FileFindNextFile($hSearch)
              If @error Then ExitLoop
              If @extended Then ContinueLoop ;bypass folder
              $sFileList &= $sDelim & $sWorkPath & $sFile
            WEnd
          Case 2 ;folders Only
            While True
              $sFile = FileFindNextFile($hSearch)
              If @error Then ExitLoop
              If @extended Then ;bypass file
                $sFileList &= $sDelim & $sWorkPath & $sFile
              EndIf
            WEnd
          Case Else ;files and folders
            While True
              $sFile = FileFindNextFile($hSearch)
              If @error Then ExitLoop
              $sFileList &= $sDelim & $sWorkPath & $sFile
            WEnd
        EndSwitch
        FileClose($hSearch)
      Next ;$iFCount - next filter

      ;---------------

      ;optional do a recursive search
      If $bRecursive Then
        $hSearch = FileFindFirstFile($aPath[$iPCount] & "*.*")
        If Not @error Then
          While True
            $sFile = FileFindNextFile($hSearch)
            If @error Then ExitLoop
            If @extended Then ;bypass file
              ;call recursive search
              If $iPathType = 1 Then $sTWorkPath = $sWorkPath & $sFile & "\"
              $sFileList &= _FileListToArrayNT7($aPath[$iPCount] & $sFile & "\", $sFilter, $iSearchType, $iPathType, $bRecursive, $sExclude, 2, $sTWorkPath)
            EndIf
          WEnd
          FileClose($hSearch)
        EndIf
      EndIf

    Next ;$iPCount - next path

  EndIf ;If $sExclude

  ;---------------

  ;set according return value
  If $sFileList Then
    Switch $iRetFormat
      Case 2 ;return a delimited string
        Return $sFileList
      Case 0 ;return a 0-based array
        Return StringSplit(StringTrimLeft($sFileList, 1), "|", 2)
      Case Else ;return a 1-based array
        Return StringSplit(StringTrimLeft($sFileList, 1), "|", 1)
    EndSwitch
  Else
    Return SetError(4, 4, "")
  EndIf

EndFunc   ;==>_FileListToArrayNT7

Edit1: little polish in parameter checking

Edit2: header adjustment

Edit3: header clarification

Edit4: SetError correction (according to jpm, thanks for final decision)

Edit5: Back to SetError(x, x, "")

Edited July 6, 2009 by BaKaMu

[Spiff59]

I'll continue testing, but at this point I can't see anything more to do. Test xersion NT7 has my blessing.

Ok, I guess this at some point will need to change to not be talking about the old version:

; Description:    full compatible _FileListToArray replacement  (with greater performance and additional features)
;                  additional: multi-path, multi-filter, multi-exclude-filter, path format options, recursive search

But I'm done! I like. Post #175 is the bomb.

Twice as fast as the 3.3.0.0 version, backward compatible, and WAY more flexible.

Edited July 4, 2009 by Spiff59

[BaKaMu]

Ok, I guees this at some point will need to change to not be talking about the old version:

; Description:    full compatible _FileListToArray replacement  (with greater performance and additional features)
;                  additional: multi-path, multi-filter, multi-exclude-filter, path format options, recursive search

Done, edited post #175

The function name will be changed at last to whatever jpm wants.

I will continue testing tomorrow.

[Beege]

The recursive search isn't working for me.

$list = _FileListToArrayNT7(@ProgramFilesDir, '*', 0, 0, True)
_ArrayDisplay($list)

Am I entering something wrong?

[Spiff59]

The recursive search isn't working for me.

$list = _FileListToArrayNT7(@ProgramFilesDir, '*', 0, 0, True)
_ArrayDisplay($list)

Am I entering something wrong?

This version takes advantage of a beta change to FileFindNextFile.

For the time being, you must have also installed the beta 3.3.1.1 version of autoit, and then run in beta mode, such as launching from Scite via Alt-F5.

[Beege]

This version takes advantage of a beta change to FileFindNextFile.

For the time being, you must have also installed the beta 3.3.1.1 version of autoit, and then run in beta mode, such as launching from Scite via Alt-F5.

Thanks. Got it now. This is alot faster!