Recursive _FileListToArray()

Spiff59 · April 6, 2012

I thought I'd post a recursive _FileListToArray() version that's approaching it's third birthday.

It's one of the latter versions buried in Tlem's 263-post thread from June 2009:

I don't claim authorship. Tlem started the ball rolling, contributing all along the way, many other members and about every MVP, MOD and even a couple DEV's had input in that thread. The function also draws from prior recursive _FileListToArray() versions.

I do think it is still the most straight-forward version floating around, having all the most useful bells and whistles, and functions both efficiently and properly. It hasn't the "depth" or "sort" options available in the newer popular version by Melba23, but it offers a very concise single-function alternative.

After nearly 3 years, I thought that something ought to be dug out of the obscurity of that massive thread (into which a lot of effort was invested by many) and placed where it's easier to find.

This is identical to post 252 from the 2009 thread except the the function name is changed, the parameters are reordered to make it backward compatible with the production _FileListToArray, and the SRER escaping special characters, that used to have the comment " ; what other characters? ", is now complete.

#include <Array.au3>
#include <File.au3>

$timer = TimerInit()
$aArray = _FileListToArrayPlus(@WindowsDir, "s*.???.*", 1, "", "*.exe.*", 1, True)
$timer = Round(TimerDiff($timer) / 1000, 2) & ' sec'
_ArrayDisplay($aArray, $timer)
Exit

; #FUNCTION# =====================================================================================================================
; _FileListToArrayPlus($sPath, $sInclude = "*", $iFlag = 0, $sExcludeFolder = "", $sExclude = "", $iPathType = 0, $bRecursive = False)
; Name...........: _FileListToArrayPlus
; Parameters ....: $sPath: Folder to search
;    $sInclude: String to match on (wildcards allowed, multiples delimited by ;)
;    $iFlag: Returned data type. 0 = Files and folders (default), 1 = Files only, 2 = Folders only
;    $sExcludeFolder: List of folders to exclude from search (wildcards allowed, multiples delimited by ;)
;    $sExclude: List of filenames to exclude from search (wildcards allowed, multiples delimited by ;)
;    $iPathType: Returned data format. 0 = Filename only (default), 1 = Path relative to $sPath, 2 = Full path/filename
;    $bRecursive: False = Search $sPath folder only (default), True = Search $sPath and all subfolders
; Author ........: Half the AutoIt community (Forum thread #96952)
;===================================================================================================================================
Func _FileListToArrayPlus($sPath, $sInclude = "", $iFlag = 0, $sExcludeFolder = "", $sExclude = "", $iPathType = 0, $bRecursive = False)
Local $sRet = "", $sReturnFormat = ""
$sPath = StringRegExpReplace($sPath, "[/]+z", "") & "" ; ensure single trailing slash
If Not FileExists($sPath) Then Return SetError(1, 1, "")
; Edit include files list
If $sInclude = "*" Then $sInclude = ""
If $sInclude Then
     If StringRegExp($sInclude, "[/:><|]|(?s)As*z") Then Return SetError(2, 2, "") ; invalid characters test
     $sInclude = StringRegExpReplace(StringRegExpReplace($sInclude, "(s*;s*)+", ";"), "A;|;z", "") ; Strip unwanted whitespace
     $sInclude = StringRegExpReplace($sInclude, "[][$.+^{}()]", "$0"); Ignore special characters
     $sInclude = StringReplace(StringReplace(StringReplace($sInclude, "?", "."), "*", ".*?"), ";", "$|") ; Convert ? to ., * to .*?, and ; to |
     $sInclude = "(?i)A(" & $sInclude & "$)"; case-insensitive, match from first char, terminate strings
EndIf
; Edit exclude folders list
If $sExcludeFolder Then
     $sExcludeFolder = StringRegExpReplace(StringRegExpReplace($sExcludeFolder, "(s*;s*)+", ";"), "A;|;z", "") ; Strip unwanted whitespace
     $sExcludeFolder = StringRegExpReplace($sExcludeFolder, "[][$.+^{}()]", "$0"); Ignore special characters
     $sExcludeFolder = StringReplace(StringReplace(StringReplace($sExcludeFolder, "?", "."), "*", ".*?"), ";", "$|") ; Convert ?=. *=.*? ;=|
     $sExcludeFolder = "(?i)A(?!" & $sExcludeFolder & "$)"; case-insensitive, match from first char, terminate strings
EndIf
; Edit exclude files list
If $sExclude Then
     $sExclude = StringRegExpReplace(StringRegExpReplace($sExclude, "(s*;s*)+", ";"), "A;|;z", "") ; Strip unwanted whitespace
     $sExclude = StringRegExpReplace($sExclude, "[][$.+^{}()]", "$0"); Ignore special characters
     $sExclude = StringReplace(StringReplace(StringReplace($sExclude, "?", "."), "*", ".*?"), ";", "$|") ; Convert ?=. *=.*? ;=|
     $sExclude = "(?i)A(?!" & $sExclude & "$)"; case-insensitive, match from first char, terminate strings
EndIf
;   MsgBox(1,"Masks","File include: " & $sInclude & @CRLF & "File exclude: " & $sExclude & @CRLF & "Dir exclude : " & $sExcludeFolder)

    If Not ($iFlag = 0 Or $iFlag = 1 Or $iFlag = 2) Then Return SetError(3, 3, "")

    Local $sOrigPathLen = StringLen($sPath), $aQueue[64] = [1,$sPath], $iQMax = 63 ; tuned?
    While $aQueue[0]
        $WorkFolder = $aQueue[$aQueue[0]]
        $aQueue[0] -= 1
        $search = FileFindFirstFile($WorkFolder & "*")
        If @error Then ContinueLoop
        Switch $iPathType
            Case 1 ; relative path
                $sReturnFormat = StringTrimLeft($WorkFolder, $sOrigPathLen)
            Case 2 ; full path
                $sReturnFormat = $WorkFolder
        EndSwitch
        While 1
            $file = FileFindNextFile($search)
            If @error Then ExitLoop
            If @extended Then ; Folder
                If $sExcludeFolder And Not StringRegExp($file, $sExcludeFolder) Then ContinueLoop
                If $bRecursive Then
                    If $aQueue[0] = $iQMax Then
                        $iQMax += 128 ; tuned?
                        ReDim $aQueue[$iQMax + 1]
                    EndIf
                    $aQueue[0] += 1
                    $aQueue[$aQueue[0]] = $WorkFolder & $file & ""
                EndIf
                If $iFlag = 1 Then ContinueLoop
                $sRet &= $sReturnFormat & $file & "|"
            Else ; File
                If $iFlag = 2 Then ContinueLoop
                If $sInclude And Not StringRegExp($file, $sInclude) Then ContinueLoop
                If $sExclude And Not StringRegExp($file, $sExclude) Then ContinueLoop
                $sRet &= $sReturnFormat & $file & "|"
            EndIf
        WEnd
        FileClose($search)
    WEnd
    If Not $sRet Then Return SetError(4, 4, "")
    Return StringSplit(StringTrimRight($sRet, 1), "|")
EndFunc

Edit: Put the mysteriously disappearing "" back in. Thanks, agerrika & AZJIO

Edit2: Correct all the missing backslashes.

Edited October 4, 2012 by Spiff59

Belini · April 6, 2012

How to show all folders and subfolders listed in alphabetical order?

$aArray = _FileListToArrayPlus(@WindowsDir, "s*.???.*", 2, "", "", 1, True)

Edited April 6, 2012 by Belini

knightz93 · April 11, 2012

nice function, thanks, you solve my problem..

Tlem · April 11, 2012

Just some words :

Can be that you should take back the same name of variable from the original function for the file(s) name ($sFilter).

And, personnaly, I think that some parameters are more important than other. I would indeed see the function as this :

_FileListToArrayPlus($sPath, $sFilter = "*", $iFlag = 0[, $bRecursive = False[, $iPathType = 0[, $sExcludeFile = ""[, $sExcludeFolder = ""]]]])

Notes and explanations :

$sExcludeFolder and $sExcludeFile are really optionnal by default. Their use are totally conditioned by a specific use. So they must be on the end of the function.

$iPathType : Should not have so many flag. Path relative and Full path should widely be enough. What the interest of only filename for a recursive search ??? What about if you have the same filename on hundred folders.

If you search on the root path, you should always have only the filename ... ^^

$bRecursive : The interest of this function is the recursive search, so this parameter should be after the last "official" parameter of the original function : $iFlag.

To be complete, you should add too in the description of the function, the error code and returns.

I prefer personally the simplicity of DXRW4E function, because it corresponds more to the basic philosophy of AutoIt (except his last modification about the last flag). but now, the file and folder exclusion is a not insignificant bit extra (for some people).

Regrettably, it is necessary to put limits, because otherwise, we meet fast with a heap of similar functions (you just have to make a search on this forum). All are quite interesting because there are specific, but by too complex for a simple use.

If Dev have to add a recurcive search function, It should be the simplest possible :

_FileListToArray($sPath[, $sFilter = "*"[, $iFlag = 0[, $iRecurse = 0]]])

The result should be Path relative, because if you want full path, you just have to add $sPath.

Now, for specific use, if you want only the filename or made file or folder exeption, a simple treatment should do the work.

But, this is only my point of view.

AZJIO · April 12, 2012

Tlem

The result should be Path relative, because if you want full path, you just have to add $sPath.

uneconomical shorten the path, then again to combine. Speed will be slower. In most cases, the full path is required.

If Dev have to add a recurcive search function

It would be a good idea to have at least one search function including subdirectories. Very often asked.

Tlem · April 13, 2012

Hello, all.

Because this function is presented to be a recursive files/folders FileListToArray, I really think that the recuse flag should be just after the file type.

In accordance with my precedent post and for be in the same philosophy than included AutoIt functions, I have changed some vars name and finally, I have added a description like others AutoIt functions. I hope that you like it!

; #FUNCTION# ====================================================================================================================
; Name...........: _FileListToArrayPlus
; Description ...: Lists files andor folders in a specified path (Similar to using Dir with the /B Switch)
; Syntax.........: _FileListToArrayPlus($sPath[, $sFilter = "*"[, $iFlag = 0 [, $bRecursive = False[, $sExcludeFile = ""[, $sExcludeFolder = "" [, $iPathType = 1]]]]]])
; Parameters ....: $sPath   - Path to generate filelist for.
;                 $sFilter  - Optional the filter to use, default is *. (Multiple filter groups such as "*.png;*.jpg;*.bmp") Search the Autoit3 helpfile for the word "WildCards" For details.
;                 $iFlag    - Optional: specifies whether to return files folders or both
;                   |$iFlag=0 (Default) Return both files and folders
;                   |$iFlag=1 Return files only
;                   |$iFlag=2 Return Folders only
;      $sExcludeFile   - Optional: filter for file(s) exclusion. Multiple filters must be separated by ";"
;      $sExcludeFolder - Optional: filter for folder(s) exclusion. Multiple filters must be separated by ";"
;                 $bRecursive   - Optional: Use this flag to specify if you want to search in sub-directories too.
;                   |$bRecursive=False (default) Do not search in sub-directories
;                   |$bRecursive=True Search in sub-directories
;                 $iPathType  - Optional: specifies displayed path of results
;                   |$iPathType = 0 - File/folder name only
;                   |$iPathType = 1 - Relative to initial path (Default)
;                   |$iPathType = 2 - Full path included
; Return values .: @Error
;         |1 = Path not found or invalid
;                   |2 = Invalid $sFilter
;                   |3 = Invalid $iFlag
;                   |4 = No File(s)/Folder(s) Found
; Author ........: Half the AutoIt community (Forum thread #96952)
; Modified.......:
; Remarks .......: The array returned is one-dimensional and is made up as follows:
;                              $array[0] = Number of FilesFolders returned
;                              $array[1] = 1st FileFolder
;                              $array[2] = 2nd FileFolder
;                              $array[3] = 3rd FileFolder
;                              $array[n] = nth FileFolder
; Related .......:
; Link ..........: http://www.autoitscript.com/forum/topic/139336-recursive-filelisttoarray/
; Example .......: Yes
; Note ..........: Special Thanks to the AutoIt community
; ===============================================================================================================================
Func _FileListToArrayPlus($sPath, $sFilter = "", $iFlag = 0, $bRecursive = False, $sExcludeFolder = "", $sExcludeFile = "", $iPathType = 1)
Local $sRet = "", $sReturnFormat = ""
$sPath = StringRegExpReplace($sPath, "[/]+z", "") & "" ; ensure single trailing slash
If Not FileExists($sPath) Then Return SetError(1, 1, "")
; Edit include files list
If $sFilter = "*" Then $sFilter = ""
If $sFilter Then
  If StringRegExp($sFilter, "[/:><|]|(?s)As*z") Then Return SetError(2, 2, "") ; invalid characters test
  $sFilter = StringRegExpReplace(StringRegExpReplace($sFilter, "(s*;s*)+", ";"), "A;|;z", "") ; Strip unwanted whitespace
  $sFilter = StringRegExpReplace($sFilter, "[][$.+^{}()]", "$0"); Ignore special characters
  $sFilter = StringReplace(StringReplace(StringReplace($sFilter, "?", "."), "*", ".*?"), ";", "$|") ; Convert ? to ., * to .*?, and ; to |
  $sFilter = "(?i)A(" & $sFilter & "$)"; case-insensitive, match from first char, terminate strings
EndIf
; Edit exclude folders list
If $sExcludeFolder Then
  $sExcludeFolder = StringRegExpReplace(StringRegExpReplace($sExcludeFolder, "(s*;s*)+", ";"), "A;|;z", "") ; Strip unwanted whitespace
  $sExcludeFolder = StringRegExpReplace($sExcludeFolder, "[][$.+^{}()]", "$0"); Ignore special characters
  $sExcludeFolder = StringReplace(StringReplace(StringReplace($sExcludeFolder, "?", "."), "*", ".*?"), ";", "$|") ; Convert ?=.  *=.*?  ;=|
  $sExcludeFolder = "(?i)A(?!" & $sExcludeFolder & "$)"; case-insensitive, match from first char, terminate strings
EndIf
; Edit exclude files list
If $sExcludeFile Then
  $sExcludeFile = StringRegExpReplace(StringRegExpReplace($sExcludeFile, "(s*;s*)+", ";"), "A;|;z", "") ; Strip unwanted whitespace
  $sExcludeFile = StringRegExpReplace($sExcludeFile, "[][$.+^{}()]", "$0"); Ignore special characters
  $sExcludeFile = StringReplace(StringReplace(StringReplace($sExcludeFile, "?", "."), "*", ".*?"), ";", "$|") ; Convert ?=.  *=.*?  ;=|
  $sExcludeFile = "(?i)A(?!" & $sExcludeFile & "$)"; case-insensitive, match from first char, terminate strings
EndIf
If Not ($iFlag = 0 Or $iFlag = 1 Or $iFlag = 2) Then Return SetError(3, 3, "")
Local $sOrigPathLen = StringLen($sPath), $aQueue[64] = [1, $sPath], $iQMax = 63 ; tuned?
While $aQueue[0]
  $WorkFolder = $aQueue[$aQueue[0]]
  $aQueue[0] -= 1
  $search = FileFindFirstFile($WorkFolder & "*")
  If @error Then ContinueLoop
  Switch $iPathType
   Case 1 ; relative path
    $sReturnFormat = StringTrimLeft($WorkFolder, $sOrigPathLen)
   Case 2 ; full path
    $sReturnFormat = $WorkFolder
  EndSwitch
  While 1
   $file = FileFindNextFile($search)
   If @error Then ExitLoop
   If @extended Then ; Folder
    If $sExcludeFolder And Not StringRegExp($file, $sExcludeFolder) Then ContinueLoop
    If $bRecursive Then
     If $aQueue[0] = $iQMax Then
      $iQMax += 128 ; tuned?
      ReDim $aQueue[$iQMax + 1]
     EndIf
     $aQueue[0] += 1
     $aQueue[$aQueue[0]] = $WorkFolder & $file & ""
    EndIf
    If $iFlag = 1 Then ContinueLoop
    $sRet &= $sReturnFormat & $file & "|"
   Else ; File
    If $iFlag = 2 Then ContinueLoop
    If $sFilter And Not StringRegExp($file, $sFilter) Then ContinueLoop
    If $sExcludeFile And Not StringRegExp($file, $sExcludeFile) Then ContinueLoop
    $sRet &= $sReturnFormat & $file & "|"
   EndIf
  WEnd
  FileClose($search)
WEnd
If Not $sRet Then Return SetError(4, 4, "")
Return StringSplit(StringTrimRight($sRet, 1), "|")
EndFunc   ;==>_FileListToArrayPlus

It has not a lot of importance, but the way which are listed files/ folders is rather disturbing !

Edited April 13, 2012 by Tlem

agerrika · September 30, 2012

Hello,

Agree with the last changes made by Tlem

I found a mistake using it. Missing in the following line

$aQueue[$aQueue[0]] = $WorkFolder & $file & ""
$aQueue[$aQueue[0]] = $WorkFolder & $file & ""

Edited September 30, 2012 by agerrika

AZJIO · October 1, 2012

This is a problem forum. Symbol disappears, not only in this. When the example was uploaded it was without errors.

guinness · October 1, 2012

Just out of curiosity is this a feature request to update _FileListToArray?

Melba23 · October 1, 2012

guinness,

I think AZJIO is referring to the nasty habit the forum software has of removing "" characters from uploaded code from time to time.

M23

guinness · October 1, 2012

Ah, I meant Spiff59's post.

Spiff59 · October 1, 2012

Just out of curiosity is this a feature request to update _FileListToArray?

That was the initial intention of the 2009 thread with 260+ posts.

We had one Dev fully in favor of the idea, but later in the thread another Dev indicated he had no interest in replacing a 27-line function with one of 64 lines (even though the additional functionality was one constantly requested). The thread did end up creating a bugtrack entry, one that reduced the production _FileListToArray() down to it's current 18 lines.

guinness · October 1, 2012

I saw your addition/report and have to say what a clever approach, especially >> If ($iFlag + @extended = 2) Then ContinueLoop, most would have used a If..Else. Just nice!

Spiff59 · October 1, 2012

Just nice!

Thank you, Sir

It was convenient that the required logic allowed for such trickery.

I, personally, still think a _FileListToArray() that processed subfolders would be a worthwhile addition to the production UDF. Whether this version, or this version with some changes, or a completely different version doesn't really matter (although I don't think anyone has beaten this version in a speed test to date).

The inclusion of a production _FileListToArray() with folder recursion would, in perpetuity, reduce the number of threads in the general help forum by at least a couple a month.

AZJIO · October 3, 2012

Why is it not working?

Returns the folder "System32"

$aArray = _FileListToArrayPlus(@SystemDir)

Spiff59 · October 3, 2012

Why is it not working?
Returns the folder "System32"
$aArray = _FileListToArrayPlus(@SystemDir)

When I patched the code because the forum oddly removes backslash characters, I only corrected one line and neglected to fix this one:

$sPath = StringRegExpReplace($sPath, "[/]+z", "") & "" ; ensure single trailing slash

which should be

$sPath = StringRegExpReplace($sPath, "[/]+z", "") & "" ; ensure single trailing slash

fixed now

guinness · October 3, 2012

My only gripe is that it doesn't support the Default keyword.

AZJIO · October 3, 2012

$aArray = _FileListToArrayPlus(@WindowsDir, "s*.???.*", 1, "", "*.exe.*", 1, True)

Returns 400 files. Do not match the mask. For example TASKMAN.EXE. The name must contain two dots.

Spiff59 · October 4, 2012

$aArray = _FileListToArrayPlus(@WindowsDir, "s*.???.*", 1, "", "*.exe.*", 1, True)
Returns 400 files. Do not match the mask. For example TASKMAN.EXE. The name must contain two dots.

Geez, I had thought the forum glitch had only converted "" into "".

It did more than that! All the backslashes are missing from the regular expressions.

I'll fix it when I get to work in the morning.

Thanks.

Edit: I replaced all the SRER statements.

Edited October 4, 2012 by Spiff59

AZJIO · October 4, 2012

check the line 38, 45

$0

Recursive _FileListToArray()

Recommended Posts

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members