FileListToArray recursive with max depth

phonk · November 30, 2010

Hello there.

I played arround with many FileListToArray scripts but none of them does it job as I want to.

I have a structure like this:

\\server\share\folder\123\456\20101130\*.pdf

\\server\share\folder\123\456\20101129\*.pdf

...

\\server\share\folder\123\456\Blah1.csv

\\server\share\folder\123\456\Blah2.csv

...

My problem is, that I want to find only the *.csv files. The daily dirs with many *.pdf inside should be ignored.

So I have to use a script which I can tell the maximum recursive search depth.

The scripts I used searches the whole path and I have to filter for the csv's. This searches takes a long time because of the mass of pdf's.

Can somebody help me ?

Melba23 · November 30, 2010

phonk,

My version of a recursive filesearch allows you to set masks for the files to find so you will only get the *.csv files you want returned. You cannot set a level of recursion, but it is pretty quick.

If you want to give it a try, you can find it here. Please ask if you have problems setting the correct parameters. :graduated:

M23

phonk · December 1, 2010

phonk,

My version of a recursive filesearch allows you to set masks for the files to find so you will only get the *.csv files you want returned. You cannot set a level of recursion, but it is pretty quick.

If you want to give it a try, you can find it here. Please ask if you have problems setting the correct parameters.

M23

Thank you.

I tried your script, but its not as fast as I need to. In my testcase I have ca. 3 % of the final data size. In this case the search takes about 5 minutes only to find 56 csv files.

For the normal operation the script will need over 1 hour to do this job.

So, an depth parameter is really needed :-(

Melba23 · December 1, 2010

phonk,

Must be a really big folder tree!

Try this minimalist version where I have added a depth limit. If you set the $iMaxLevel parameter to 0 it basically does not do a recursive search - for each additional 1 added to the parameter it searches another level down. I have hard coded it for *.csv files (becasue that is what you are searching for) but that is easy to change if needed.

#include <Array.au3> ; Just for demo

_DriveList("Your_Path", 1) ; This is searching to one level down

Func _DriveList($sInitialPath, $iMaxLevel = 0)

    Local $asFolderList[2][2] = [[1, 0], [0, 0]]
    Local $asFileList[1] = [0]

    ; Add trailing \ if required
    If StringRight($sInitialPath, 1) <> "\" Then $sInitialPath = $sInitialPath & "\"
    ; Store result
    $asFolderList[1][0] = $sInitialPath

    ; Search in listed folders
    While $asFolderList[0][0] > 0

        ; Check if we have exceeded the level limit
        If $asFolderList[$asFolderList[0][0]][1] <= $iMaxLevel Then

            ; Set current level
            $iCurrLevel = $asFolderList[$asFolderList[0][0]][1]

            ; Set path to search
            $sCurrentPath = $asFolderList[$asFolderList[0][0]][0]
            ; Reduce folder array count
            $asFolderList[0][0] -= 1

            ; Get Search handle
            $hSearch = FileFindFirstFile($sCurrentPath & "*.csv")

            ; If folder empty move to next in list
            If $hSearch = -1 Then ContinueLoop

            ; Search folder
            While 1
                $sName = FileFindNextFile($hSearch)
                ; Check for end of folder
                If @error Then ExitLoop
                ; Check for subfolder - @extended set in 3.3.1.1 +
                If @extended Then ; Add to folder list
                    ; Increase folder list count
                    $asFolderList[0][0] += 1
                    ; Double folder list size if too small (fewer ReDim needed)
                    If UBound($asFolderList) <= $asFolderList[0][0] Then ReDim $asFolderList[UBound($asFolderList) * 2][2]
                    ; Add folder name
                    $asFolderList[$asFolderList[0][0]][0] = $sCurrentPath & $sName & "\"
                    ; Add folder level
                    $asFolderList[$asFolderList[0][0]][1] = $iCurrLevel + 1
                Else ; Add to file list
                    ; Increase file list count
                    $asFileList[0] += 1
                    ; Double file list size if too small (fewer ReDim needed)
                    If UBound($asFileList) <= $asFileList[0] Then ReDim $asFileList[UBound($asFileList) * 2]
                    ; Add file name
                    $asFileList[$asFileList[0]] = $sCurrentPath & $sName
                EndIf
            WEnd

            ; Close current search
            FileClose($hSearch)

        Else

            ; Reduce folder array count
            $asFolderList[0][0] -= 1

        EndIf

    WEnd

    ; Remove any unused return list elements from last ReDim
    ReDim $asFileList[$asFileList[0] + 1]

    ; Display results
    _ArrayDisplay($asFileList) ; Just for demo - will take a long time if you have a lot of files

EndFunc   ;==>_DriveList

Let me know how it does - it is working fine for me. :graduated:

M23

phonk · December 1, 2010

Oh thanks, I will try it ...

phonk · December 1, 2010

Hmmm, it seems only to search the given path. It doesn't go into depth with the given parameter .. :-(

Melba23 · December 1, 2010

phonk,

Ooops! :graduated:

Trying to hard code it for just *.csv files, I ended up also only looking for *.csv subfolders - no wonder it would not go into the tree!

Try this version:

#include <Array.au3> ; Just for demo

_DriveList("Your_Path", 0) ; This is searching to one level down

Func _DriveList($sInitialPath, $iMaxLevel = 0)

    Local $asFolderList[2][2] = [[1, 0], [0, 0]]
    Local $asFileList[1] = [0]

    ; Add trailing \ if required
    If StringRight($sInitialPath, 1) <> "\" Then $sInitialPath = $sInitialPath & "\"
    ; Store result
    $asFolderList[1][0] = $sInitialPath

    ; Search in listed folders
    While $asFolderList[0][0] > 0

        ; Check if we have exceeded the level limit
        If $asFolderList[$asFolderList[0][0]][1] <= $iMaxLevel Then

            ; Set current level
            $iCurrLevel = $asFolderList[$asFolderList[0][0]][1]

            ; Set path to search
            $sCurrentPath = $asFolderList[$asFolderList[0][0]][0]
            ; Reduce folder array count
            $asFolderList[0][0] -= 1

            ; Get Search handle
            $hSearch = FileFindFirstFile($sCurrentPath & "*.*")

            ; If folder empty move to next in list
            If $hSearch = -1 Then ContinueLoop

            ; Search folder
            While 1
                $sName = FileFindNextFile($hSearch)
                ; Check for end of folder
                If @error Then ExitLoop
                ; Check for subfolder - @extended set in 3.3.1.1 +
                If @extended Then ; Add to folder list
                    ; Increase folder list count
                    $asFolderList[0][0] += 1
                    ; Double folder list size if too small (fewer ReDim needed)
                    If UBound($asFolderList) <= $asFolderList[0][0] Then ReDim $asFolderList[UBound($asFolderList) * 2][2]
                    ; Add folder name
                    $asFolderList[$asFolderList[0][0]][0] = $sCurrentPath & $sName & "\"
                    ; Add folder level
                    $asFolderList[$asFolderList[0][0]][1] = $iCurrLevel + 1

                Else ; Add to file list if it is *.csv file

                    If StringRight($sName, 4) = ".csv" Then
                        ; Increase file list count
                        $asFileList[0] += 1
                        ; Double file list size if too small (fewer ReDim needed)
                        If UBound($asFileList) <= $asFileList[0] Then ReDim $asFileList[UBound($asFileList) * 2]
                        ; Add file name
                        $asFileList[$asFileList[0]] = $sCurrentPath & $sName
                    EndIf
                EndIf
            WEnd

            ; Close current search
            FileClose($hSearch)

        Else

        ; Reduce folder array count
            $asFolderList[0][0] -= 1

        EndIf

    WEnd

    ; Remove any unused return list elements from last ReDim
    ReDim $asFileList[$asFileList[0] + 1]

    ; Display results
    _ArrayDisplay($asFileList) ; Just for demo - will take a long time if you have a lot of files

EndFunc   ;==>_DriveList

Sorry about that - I think this one might do the trick.

M23

phonk · December 1, 2010

Yes much better,

But I think the script searches now in the first depth for *.csv ?

Because I set the max Level to 4, but only the first subdirs under the init path are listed in my ConsoleWrite of $sCurrentPath

For understanding:

The Structure goes like this:

SOURCEPATH\001\001\*.csv

SOURCEPATH\002\001\*.csv

SOURCEPATH\002\002\*.csv

and so on.

Furthermore I had to put some "Local" into your Script:

Func _DriveList($sInitialPath, $iMaxLevel = 4)

    Local $asFolderList[2][2] = [[1, 0], [0, 0]]
    Local $asFileList[1] = [0]

    ; Add trailing \ if required
    If StringRight($sInitialPath, 1) <> "\" Then $sInitialPath = $sInitialPath & "\"
    ; Store result
    $asFolderList[1][0] = $sInitialPath

    ; Search in listed folders
    While $asFolderList[0][0] > 0

        ; Check if we have exceeded the level limit
        If $asFolderList[$asFolderList[0][0]][1] <= $iMaxLevel Then

            ; Set current level
            Local $iCurrLevel = $asFolderList[$asFolderList[0][0]][1]
            ConsoleWrite($iCurrLevel & @LF)
            ; Set path to search
            Local $sCurrentPath = $asFolderList[$asFolderList[0][0]][0]
            ConsoleWrite($sCurrentPath & @LF)
            ; Reduce folder array count
            $asFolderList[0][0] -= 1

            ; Get Search handle
            Local $hSearch = FileFindFirstFile($sCurrentPath & "*.*")

            ; If folder empty move to next in list
            If $hSearch = -1 Then ContinueLoop

            ; Search folder
            While 1
                Local $sName = FileFindNextFile($hSearch)
                ; Check for end of folder
                If @error Then ExitLoop
                ; Check for subfolder - @extended set in 3.3.1.1 +
                If @extended Then ; Add to folder list
                    ; Increase folder list count
                    $asFolderList[0][0] += 1
                    ; Double folder list size if too small (fewer ReDim needed)
                    If UBound($asFolderList) <= $asFolderList[0][0] Then ReDim $asFolderList[UBound($asFolderList) * 2][2]
                    ; Add folder name
                    $asFolderList[$asFolderList[0][0]][0] = $sCurrentPath & $sName & "\"
                    ; Add folder level
                    $asFolderList[$asFolderList[0][0]][1] = $iCurrLevel + 1

                Else ; Add to file list if it is *.csv file

                    If StringRight($sName, 4) = ".csv" Then
                        ; Increase file list count
                        $asFileList[0] += 1
                        ; Double file list size if too small (fewer ReDim needed)
                        If UBound($asFileList) <= $asFileList[0] Then ReDim $asFileList[UBound($asFileList) * 2]
                        ; Add file name
                        $asFileList[$asFileList[0]] = $sCurrentPath & $sName
                    EndIf
                EndIf
            WEnd

            ; Close current search
            FileClose($hSearch)

        Else

        ; Reduce folder array count
            $asFolderList[0][0] -= 1

        EndIf

    WEnd

    ; Remove any unused return list elements from last ReDim
    ReDim $asFileList[$asFileList[0] + 1]

    ; Display results
    _ArrayDisplay($asFileList) ; Just for demo - will take a long time if you have a lot of files

EndFunc   ;==>_DriveList

Edited December 1, 2010 by phonk

Melba23 · December 1, 2010

phonk,

It works for me - I can go down as far as I want into my local tree. Perhaps there is something different about networks? :graduated:

This script has added lines to tell you which folders it is searching and which it is ignoring - try gradually increasing the $iMaxLevel parameter and see if it searches where you think it should. As I said before, it is working fine for me:

#include <Array.au3> ; Just for demo

_DriveList("M:\Program\Au3 Scripts\Snippets", 1) ; This is searching to one level down

Func _DriveList($sInitialPath, $iMaxLevel = 0)

    Local $asFolderList[2][2] = [[1, 0], [0, 0]]
    Local $asFileList[1] = [0]

    ; Add trailing \ if required
    If StringRight($sInitialPath, 1) <> "\" Then $sInitialPath = $sInitialPath & "\"
    ; Store result
    $asFolderList[1][0] = $sInitialPath

    ; Search in listed folders
    While $asFolderList[0][0] > 0

        ; Check if we have exceeded the level limit
        If $asFolderList[$asFolderList[0][0]][1] <= $iMaxLevel Then

            ConsoleWrite("+ Searching: " & $asFolderList[$asFolderList[0][0]][0] & @CRLF)

            ; Set current level
            $iCurrLevel = $asFolderList[$asFolderList[0][0]][1]

            ; Set path to search
            $sCurrentPath = $asFolderList[$asFolderList[0][0]][0]
            ; Reduce folder array count
            $asFolderList[0][0] -= 1

            ; Get Search handle
            $hSearch = FileFindFirstFile($sCurrentPath & "*.*")

            ; If folder empty move to next in list
            If $hSearch = -1 Then ContinueLoop

            ; Search folder
            While 1
                $sName = FileFindNextFile($hSearch)
                ; Check for end of folder
                If @error Then ExitLoop
                ; Check for subfolder - @extended set in 3.3.1.1 +
                If @extended Then ; Add to folder list
                    ; Increase folder list count
                    $asFolderList[0][0] += 1
                    ; Double folder list size if too small (fewer ReDim needed)
                    If UBound($asFolderList) <= $asFolderList[0][0] Then ReDim $asFolderList[UBound($asFolderList) * 2][2]
                    ; Add folder name
                    $asFolderList[$asFolderList[0][0]][0] = $sCurrentPath & $sName & "\"
                    ; Add folder level
                    $asFolderList[$asFolderList[0][0]][1] = $iCurrLevel + 1

                Else ; Add to file list if it is *.csv file

                    If StringRight($sName, 4) = ".csv" Then
                        ; Increase file list count
                        $asFileList[0] += 1
                        ; Double file list size if too small (fewer ReDim needed)
                        If UBound($asFileList) <= $asFileList[0] Then ReDim $asFileList[UBound($asFileList) * 2]
                        ; Add file name
                        $asFileList[$asFileList[0]] = $sCurrentPath & $sName
                    EndIf
                EndIf
            WEnd

            ; Close current search
            FileClose($hSearch)

        Else

            ConsoleWrite("! Ignoring: " & $asFolderList[$asFolderList[0][0]][0] & @CRLF)

            ; Reduce folder array count
            $asFolderList[0][0] -= 1

        EndIf

    WEnd

    ; Remove any unused return list elements from last ReDim
    ReDim $asFileList[$asFileList[0] + 1]

    ; Display results
    _ArrayDisplay($asFileList) ; Just for demo - will take a long time if you have a lot of files

EndFunc   ;==>_DriveList

Furthermore I had to put some "Local" into your Script

If you are going to get that picky you can develop the solution yourself!

M23

phonk · December 1, 2010

Ah, this Output covers up an "Error between Headphones" from myself :-)

Your Script worked fine, but I played arround with the $iMaxLevel = 0 and forgot that I call the function with the parameter 1 which permenantly overwrites the $iMaxLevel = X

Thank your very much, now it works as fast as I can live with ...

UEZ · December 1, 2010

Would you be so kind and test my script, too?

#include <Array.au3>
Dim $result[1]

$path = "\\SERVER\PATH" ;enter here the network path

If StringRight($path, 1) <> "\" Then $path &= "\" ; adds a \ to the end of path if not available
$depth = StringReplace($path, "\", "\") ;depth needed for recursion level
$depth = @extended - 1
If StringLeft($path, 2) = "\\" Then $depth -= 2
$file = ".csv" ;file extension to search for

$result = File_Seeker($path, $file, $result, 1, 0, 1, 0, 5) ;searches up to recursion level 5 relative to starting point

If Not @error Then
    ;~ _ArraySort($result)
    _ArrayDisplay($result)
;~  Local $log
;~  For $i = 1 To UBound($result) - 1
;~      $log &= $result[$i] & @CRLF
;~  Next
;~  $file = FileOpen(@ScriptDir & "\Ergebnis.log", 1)
;~  FileWrite($file, $log)
;~  FileClose($file)
EndIf

Exit



;==================================================================================================
; Function Name:    File_Seeker($path, $file, $array)
; Description:          searchs for a file recursively on given drive
; Parameters:           $path: path where to search
;                               $file: filename to search for
;                               $array: to save the search results
; Optional:
;                               $ext: search for file extension, e.g. .mp3
;                               $exact: to search for exact file name (not case sensitive)
;                               $rec: to search recursively for files and folders
;                               $folder: include also folder names to search pattern
;                               $maxr: maximum of recursion depth, 0 = maxiumum
; Return Value(s):  an array with the full path of the files, $array[0] = amount of found files
; Version:                  v0.85 Build 2010-12-01 Beta
; Author(s):                UEZ
;==================================================================================================
Func File_Seeker($path, $file, ByRef $aResults, $ext = 0, $exact = 0, $rec = 1, $folder = 0, $maxr = 0)
    Local $extended, $d
    If Not IsInt($maxr) Then Return SetError(2)
    If $file <> "" And IsArray($aResults) And BitAND($ext, $exact) <> 1 Then
        If StringLeft($path, 2) = "\\" Or (StringRegExp(StringLeft($path, 1), "(?i:[A-Z])") And StringMid($path, 2, 1) = ":") Then
            If $maxr Then
                $d = StringReplace($path, "\", "\")
                $d = @extended - 1
                If StringLeft($path, 2) = "\\" Then $d -= 2
                $d -= $depth
                ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $d = ' & $d & ">=" & $maxr & ": " & $path & @crlf & '>Error code: ' & @error & @crlf) ;### Debug Console
                If $d > $maxr Then Return SetError(5)
            EndIf
            If StringRight($path, 1) <> "\" Then $path &= "\"
            Local $seek, $suffix = ""
            Local $search = FileFindFirstFile($path & "*")
            If $search = -1 Then
                Return SetError(3)
            EndIf
            While 1
                $seek = FileFindNextFile($search)
                $extended = @extended
                If @error Then ExitLoop
                If $extended And $rec Then File_Seeker($path & $seek & "\", $file, $aResults, $ext, $exact, $rec, $folder, $maxr)
                If $exact And Not $ext Then
                    If $seek = $file Then _ArrayAdd($aResults, $path & $seek)
                Else
                    If $ext Then
                        If StringRegExp($seek, "(?i:\" & $file & "+$\b)") And StringLeft($file, 1) = "." Then
                            If ($extended And $folder) Or Not $extended Then _ArrayAdd($aResults, $path & $seek)
                        EndIf
                    Else
                        If $file = "*" Then
                            If ($extended And $folder) Or Not $extended Then _ArrayAdd($aResults, $path & $seek)
                        Else
                            If StringInStr($file, ".") Then
                                If StringRegExp($seek, "(?i:\" & $file & ")+") Then
                                    If ($extended And $folder) Or Not $extended Then _ArrayAdd($aResults, $path & $seek)
                                EndIf
                            Else
                                If StringRegExp($seek, "(?i:" & $file & ")+") Then
                                    If ($extended And $folder) Or Not $extended Then _ArrayAdd($aResults, $path & $seek)
                                EndIf
                            EndIf
                        EndIf
                    EndIf
                EndIf
            WEnd
            FileClose($search)
            $aResults[0] = UBound($aResults) - 1
            Return SetError(0, 0, $aResults)
        Else
            Return SetError(4)
        EndIf
    Else
        Return SetError(1)
    EndIf
EndFunc

Thanks!

Br,

UEZ

removed html codes from au3 code. seems to be a problem with me especially Posted Image and made some small modifications!

Edited December 1, 2010 by UEZ

Melba23 · December 1, 2010

phonk,

Delighted you got it working. Looks like a draw on the "seat-keyboard" interface problems!

UEZ,

Your code works fine for me - once I realised that your level is +1 relative to mine. :graduated:

M23

UEZ · December 1, 2010

I defined it this way:

e.g. network path:

\\server1\software is level 0 whereas

\\server1\software\test1\test2 is level 2

Br,

UEZ

Melba23 · December 9, 2010

phonk,

If you are still interested, I have developed a new version of my full recursive search UDF with the added possibility to limit the search level as you required.

If you need to limit the search level, you merely set the $iRecur parameter to the negative of the level you want (i.e. -2 for 2 levels down). Why negative? That way I avoid script-breaking changes to the UDF - positive 1 already meant unlimited recursion. :shifty:

#include-once

;#AutoIt3Wrapper_au3check_parameters=-d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6

; #INDEX# =======================================================================================================================
; Title .........: _RecFileListToArray
; AutoIt Version : v3.3.1.1 or higher
; Language ......: English
; Description ...: Lists files and\or folders in specified path with optional recursion to defined level and result sorting
; Note ..........:
; Author(s) .....: Melba23
; Remarks .......: Modified Array.au3 functions - credit: Jos van der Zande, LazyCoder, Tylo, Ultima, SolidSnake and gcriaco
;                  SRE patterns - credit: various forum members and Spiff59 in particular
; ===============================================================================================================================

; #CURRENT# =====================================================================================================================
; _RecFileListToArray: Lists files and\or folders in a specified path with optional recursion to defined level and result sorting
; ===============================================================================================================================

; #INTERNAL_USE_ONLY#============================================================================================================
; _RFLTA_ListToMask ......; Convert include/exclude lists to SRE format
; _RFLTA_AddToList .......; Add element to list which is resized if necessary
; _RFLTA_AddFileLists ....; Add internal lists after resizing and optional sorting
; _RFLTA_ArraySearch .....; Search array for partial match
; _RFLTA_ArraySort .......; Wrapper for QuickSort function
; _RFLTA_QuickSort .......: Recursive array sort
; _RFLTA_ArrayConcatenate : Join 2 arrays
; _RFLTA_ArrayInsert .....: Insert element into array
; ===============================================================================================================================

; #FUNCTION# ====================================================================================================================
; Name...........: _RecFileListToArray
; Description ...: Lists files and\or folders in a specified path with optional recursion to defined level and result sorting
; Syntax.........: _RecFileListToArray($sPath[, $sInclude_List = "*"[, $iReturn = 0[, $iRecur = 0[, $iSort = 0[, $iReturnPath = 1[, $sExclude_List = ""]]]]]])
; Parameters ....: $sPath - Initial path used to generate filelist.  If path ends in \ then folders will be returned with an ending \
;                  $sInclude_List - Optional: filter for included results (default "*"). Multiple filters must be separated by ";"
;                  $iReturn - Optional: specifies whether to return files, folders or both
;                  |$iReturn = 0 (Default) Return both files and folders
;                  |$iReturn = 1 Return files only
;                  |$iReturn = 2 Return folders only
;                  $iRecur - Optional: specifies whether to search recursively in subfolders and to what level
;                  |$iRecur = 1 Search in all subfolders (unlimited recursion)
;                  |$iRecur = 0 (Default) Do not search in subfolders
;                  |$iRecur = Negative integer - Search in subfolders to specified depth
;                  $iSort - Optional: sort ordered in alphabetical and depth order
;                  |$iSort = 0 (Default) Not sorted
;                  |$iSort = 1 Sorted
;                  $iReturnPath - Optional: specifies displayed path of results
;                  |$iReturnPath = 0 File/folder name only
;                  |$iReturnPath = 1 (Default) Relative to initial path
;                  |$iReturnPath = 2 Full path included
;                  $sExclude_List - Optional: filter for excluded results (default ""). Multiple filters must be separated by ";"
; Requirement(s).: v3.3.1.1 or higher
; Return values .: Success: One-dimensional array made up as follows:
;                  |$array[0] = Number of Files\Folders returned
;                  |$array[1] = 1st File\Folder
;                  |$array[2] = 2nd File\Folder
;                  |...
;                  |$array[n] = nth File\Folder
;                   Failure: Null string and @error = 1 with @extended set as follows:
;                  |1 = Path not found or invalid
;                  |2 = Invalid $sInclude_List
;                  |3 = Invalid $iReturn
;                  |4 = Invalid $iRecur
;                  |5 = Invalid $iSort
;                  |6 = Invalid $iReturnPath
;                  |7 = Invalid $sExclude_List
;                  |8 = No files/folders found
; Author ........: Melba23
; Remarks .......: Compatible with existing _FileListToArray syntax
; Related .......:
; Link ..........;
; Example .......; Yes
; ===============================================================================================================================
Func _RecFileListToArray($sInitialPath, $sInclude_List = "*", $iReturn = 0, $iRecur = 0, $iSort = 0, $iReturnPath = 1, $sExclude_List = "")

    Local $asReturnList[100] = [0], $asFileMatchList[100] = [0], $asRootFileMatchList[100] = [0], $asFolderMatchList[100] = [0], $asFolderSearchList[100] = [1]
    Local $sFolderSlash = "", $iMaxLevel, $sInclude_List_Mask, $sExclude_List_Mask, $hSearch, $fFolder, $sRetPath = "", $sCurrentPath, $sName

    ; Check for valid path
    If Not FileExists($sInitialPath) Then Return SetError(1, 1, "")
    ; Check if folders should have trailing \ and ensure that initial path does have one
    If StringRight($sInitialPath, 1) = "\" Then
        $sFolderSlash = "\"
    Else
        $sInitialPath = $sInitialPath & "\"
    EndIf
    ; Add path to folder search list
    $asFolderSearchList[1] = $sInitialPath

    ; Check for valid recur value
    If $iRecur > 1 Or Not IsInt($iRecur) Then Return SetError(1, 4, "")
    ; If required, determine \ count for max recursive level setting
    If $iRecur < 0 Then
        StringRegExpReplace($sInitialPath, "\\", "")
        $iMaxLevel = @extended - $iRecur
    EndIf

    ; Create Include List mask
    If $sInclude_List = "*" Then
        $sInclude_List_Mask = ".+" ; Set mask to exclude base folder with NULL name
    Else
        If Not _RFLTA_ListToMask($sInclude_List_Mask, $sInclude_List) Then Return SetError(1, 2, "")
    EndIf

    ; Create Exclude List mask
    If $sExclude_List = "" Then
        $sExclude_List_Mask = ":" ; Set unmatchable mask
    Else
        If Not _RFLTA_ListToMask($sExclude_List_Mask, $sExclude_List) Then Return SetError(1, 7, "")
    EndIf

    ; Verify other parameters
    If Not ($iReturn = 0 Or $iReturn = 1 Or $iReturn = 2) Then Return SetError(1, 3, "")
    If Not ($iSort = 0 Or $iSort = 1) Then Return SetError(1, 5, "")
    If Not ($iReturnPath = 0 Or $iReturnPath = 1 Or $iReturnPath = 2) Then Return SetError(1, 6, "")

    ; Search within listed folders
    While $asFolderSearchList[0] > 0

        ; Set path to search
        $sCurrentPath = $asFolderSearchList[$asFolderSearchList[0]]
        ; Reduce folder search list count
        $asFolderSearchList[0] -= 1

        ; Determine return path to add to file/folder name
        Switch $iReturnPath
            ; Case 0 ; Name only
            ; Leave as ""
            Case 1 ;Relative to initial path
                $sRetPath = StringReplace($sCurrentPath, $sInitialPath, "")
            Case 2 ; Full path
                $sRetPath = $sCurrentPath
        EndSwitch

        ; Get search handle
        $hSearch = FileFindFirstFile($sCurrentPath & "*")
        ; If folder empty move to next in list
        If $hSearch = -1 Then ContinueLoop

        ; Search folder
        While 1
            $sName = FileFindNextFile($hSearch)
            ; Check for end of folder
            If @error Then ExitLoop
            ; Set subfolder flag - @extended set in 3.3.1.1 +
            $fFolder = @extended

            ; If folder then check whether to add to search list
            If $fFolder Then
                Switch $iRecur
                    Case 1 ; Always add
                        _RFLTA_AddToList($asFolderSearchList, $sCurrentPath & $sName & "\")
                    Case 0 ; Never add
                        ; Do nothing
                    Case Else ; Add if max level not exceeded
                        StringRegExpReplace($sCurrentPath, "\\", "")
                        If @extended < $iMaxLevel Then _RFLTA_AddToList($asFolderSearchList, $sCurrentPath & $sName & "\")
                EndSwitch
            EndIf

            ; Match name against Include/Exclude masks
            If StringRegExp($sName, $sInclude_List_Mask) And Not StringRegExp($sName, $sExclude_List_Mask) Then

                If $iSort Then ; Save in relevant folders for later sorting

                    If $fFolder Then
                        _RFLTA_AddToList($asFolderMatchList, $sRetPath & $sName & $sFolderSlash)
                    Else
                        ; Select required list for files
                        If $sCurrentPath = $sInitialPath Then
                            _RFLTA_AddToList($asRootFileMatchList, $sRetPath & $sName)
                        Else
                            _RFLTA_AddToList($asFileMatchList, $sRetPath & $sName)
                        EndIf
                    EndIf

                Else ; Save directly in return list

                    ; Check file/folder type against required return value
                    If $fFolder + $iReturn <> 2 Then
                        ; Add final "\" to folders if required
                        If $fFolder Then $sName &= $sFolderSlash
                        _RFLTA_AddToList($asReturnList, $sRetPath & $sName)
                    EndIf

                EndIf
            EndIf
        WEnd

        ; Close current search
        FileClose($hSearch)

    WEnd

    If $iSort Then

        ; Check if any file/folders have been added
        If $asRootFileMatchList[0] = 0 And $asFileMatchList[0] = 0 And $asFolderMatchList[0] = 0 Then Return SetError(1, 8, "")

        Switch $iReturn
            Case 2 ; Folders only
                ; Correctly size folder match list
                ReDim $asFolderMatchList[$asFolderMatchList[0] + 1]
                ; Copy size folder match array
                $asReturnList = $asFolderMatchList
                ; Simple sort list
                _RFLTA_ArraySort($asReturnList)
            Case 1 ; Files only
                If $iReturnPath = 0 Then ; names only so simple sort suffices
                    ; Combine file match lists
                    _RFLTA_AddFileLists($asReturnList, $asRootFileMatchList, $asFileMatchList)
                    ; Simple sort combined file list
                    _RFLTA_ArraySort($asReturnList)
                Else
                    ; Combine sorted file match lists
                    _RFLTA_AddFileLists($asReturnList, $asRootFileMatchList, $asFileMatchList, 1)
                EndIf
            Case 0 ; Both files and folders
                If $iReturnPath = 0 Then ; names only so simple sort suffices
                    ; Combine file match lists
                    _RFLTA_AddFileLists($asReturnList, $asRootFileMatchList, $asFileMatchList)
                    ; Set correct count for folder add
                    $asReturnList[0] += $asFolderMatchList[0]
                    ; Resize and add file match array
                    ReDim $asFolderMatchList[$asFolderMatchList[0] + 1]
                    _RFLTA_ArrayConcatenate($asReturnList, $asFolderMatchList)
                    ; Simple sort final list
                    _RFLTA_ArraySort($asReturnList)
                Else
                    ; Combine sorted file match lists
                    _RFLTA_AddFileLists($asReturnList, $asRootFileMatchList, $asFileMatchList, 1)
                    ; Add folder count
                    $asReturnList[0] += $asFolderMatchList[0]
                    ; Sort folder match list
                    ReDim $asFolderMatchList[$asFolderMatchList[0] + 1]
                    _RFLTA_ArraySort($asFolderMatchList)

                    ; Now add folders in correct place
                    Local $iLastIndex = $asReturnList[0]
                    For $i = $asFolderMatchList[0] To 1 Step -1
                        ; Find first filename containing folder name
                        Local $iIndex = _RFLTA_ArraySearch($asReturnList, $asFolderMatchList[$i])
                        If $iIndex = -1 Then
                            ; Empty folder so insert immediately above previous
                            _RFLTA_ArrayInsert($asReturnList, $iLastIndex, $asFolderMatchList[$i])
                        Else
                            ; Insert folder at correct point above files
                            _RFLTA_ArrayInsert($asReturnList, $iIndex, $asFolderMatchList[$i])
                            $iLastIndex = $iIndex
                        EndIf
                    Next
                EndIf
        EndSwitch

    Else ; No sort

        ; Check if any file/folders have been added
        If $asReturnList[0] = 0 Then Return SetError(1, 8, "")

        ; Remove any unused return list elements from last ReDim
        ReDim $asReturnList[$asReturnList[0] + 1]

    EndIf

    Return $asReturnList

EndFunc   ;==>_RecFileListToArray

; #INTERNAL_USE_ONLY#============================================================================================================
; Name...........: _RFLTA_ListToMask
; Description ...: Convert include/exclude lists to SRE format
; Syntax ........: _RFLTA_ListToMask(ByRef $sMask, $sList)
; Parameters ....: $asMask - Include/Exclude mask to create
;                  $asList - Include/Exclude list to convert
; Return values .: Success: 1
;                  Failure: 0
; Author ........: SRE patterns developed from those posted by various forum members and Spiff59 in particular
; Remarks .......: This function is used internally by _RecFileListToArray
; ===============================================================================================================================
Func _RFLTA_ListToMask(ByRef $sMask, $sList)

    ; Check for invalid characters within list
    If StringRegExp($sList, "\\|/|:|\<|\>|\|") Then Return 0
    ; Strip WS and insert | for ;
    $sList = StringReplace(StringStripWS(StringRegExpReplace($sList, "\s*;\s*", ";"), 3), ";", "|")
    ; Convert to SRE pattern
    $sList = StringReplace(StringReplace(StringRegExpReplace($sList, "(\^|\$|\.)", "\\$1"), "?", "."), "*", ".*?")  ; Add prefix and suffix
    $sMask =  "(?i)^(" & $sList & ")\z"
    Return 1

EndFunc   ;==>_RFLTA_ListToMask

; #INTERNAL_USE_ONLY#============================================================================================================
; Name...........: _RFLTA_AddToList
; Description ...: Add element to list which is resized if necessary
; Syntax ........: _RFLTA_AddToList(ByRef $asList, $sValue)
; Parameters ....: $asList - List to be added to
;                  $sValue - Value to add
; Return values .: None - array modified ByRef
; Author ........: Melba23
; Remarks .......: This function is used internally by _RecFileListToArray
; ===============================================================================================================================
Func _RFLTA_AddToList(ByRef $asList, $sValue)

    ; Increase list count
    $asList[0] += 1
    ; Double list size if too small (fewer ReDim needed)
    If UBound($asList) <= $asList[0] Then ReDim $asList[UBound($asList) * 2]
    ; Add value
    $asList[$asList[0]] = $sValue

EndFunc   ;==>_RFLTA_AddToList

; #INTERNAL_USE_ONLY#============================================================================================================
; Name...........: _RFLTA_AddFileLists
; Description ...: Add internal lists after resizing and optional sorting
; Syntax ........: _RFLTA_AddFileLists(ByRef $asTarget, $asSource_1, $asSource_2[, $iSort = 0])
; Parameters ....: $asReturnList - Base list
;                  $asRootFileMatchList - First list to add
;                  $asFileMatchList - Second list to add
;                  $iSort - (Optional) Whether to sort lists before adding
;                  |$iSort = 0 (Default) No sort
;                  |$iSort = 1 Sort in descending alphabetical order
; Return values .: None - array modified ByRef
; Author ........: Melba23
; Remarks .......: This function is used internally by _RecFileListToArray
; ===============================================================================================================================
Func _RFLTA_AddFileLists(ByRef $asTarget, $asSource_1, $asSource_2, $iSort = 0)

    ; Correctly size root file match array
    ReDim $asSource_1[$asSource_1[0] + 1]
    ; Simple sort root file match array if required
    If $iSort = 1 Then _RFLTA_ArraySort($asSource_1)
    ; Copy root file match array
    $asTarget = $asSource_1
    ; Add file match count
    $asTarget[0] += $asSource_2[0]
    ; Correctly size file match array
    ReDim $asSource_2[$asSource_2[0] + 1]
    ; Simple sort file match array if required
    If $iSort = 1 Then _RFLTA_ArraySort($asSource_2)
    ; Add file match array
    _RFLTA_ArrayConcatenate($asTarget, $asSource_2)

EndFunc   ;==>_RFLTA_AddFileLists

; #INTERNAL_USE_ONLY#============================================================================================================
; Name...........: _RFLTA_ArraySearch
; Description ...: Search array downwards for partial match
; Syntax ........: _RFLTA_ArraySearch(Const ByRef $avArray, $vValue)
; Parameters ....: $avArray - Array to search
;                  $vValue - PValue to search for
; Return values .: Success: Index of array in which element was found
;                  Failure: returns -1
; Author ........: SolidSnake, gcriaco, Ultima
; Modified.......: Melba23
; Remarks .......: This function is used internally by _RecFileListToArray
; ===============================================================================================================================
Func _RFLTA_ArraySearch(Const ByRef $avArray, $vValue)

    For $i = 1 To UBound($avArray) - 1
        If StringInStr($avArray[$i], $vValue) > 0 Then Return $i
    Next
    Return -1

EndFunc   ;==>_RFLTA_ArraySearch

; #INTERNAL_USE_ONLY#============================================================================================================
; Name...........: _RFLTA_ArraySort
; Description ...: Wrapper for QuickSort function
; Syntax ........: _RFLTA_ArraySort(ByRef $avArray)
; Parameters ....: $avArray - Array to sort
;                  $pNew_WindowProc - Pointer to new WindowProc
; Return values .: None - array modified ByRef
; Author ........: Jos van der Zande, LazyCoder, Tylo, Ultima
; Modified.......: Melba23
; Remarks .......: This function is used internally by _RecFileListToArray
; ===============================================================================================================================
Func _RFLTA_ArraySort(ByRef $avArray)

    Local $iStart = 1, $iEnd = UBound($avArray) - 1
    _RFLTA_QuickSort($avArray, $iStart, $iEnd)

EndFunc   ;==>_RFLTA_ArraySort

; #INTERNAL_USE_ONLY#============================================================================================================
; Name...........: _RFLTA_QuickSort
; Description ...: Recursive array sort
; Syntax ........: _RFLTA_QuickSort(ByRef $avArray, ByRef $iStart, ByRef $iEnd)
; Parameters ....: $avArray - Array to sort in descending alphabetical order
;                  $iStart - Start index
;                  $iEnd - End index
; Return values .: None - array modified ByRef
; Author ........: Jos van der Zande, LazyCoder, Tylo, Ultima
; Modified.......: Melba23
; Remarks .......: This function is used internally by _RFLTA_ArraySort
; ===============================================================================================================================
Func _RFLTA_QuickSort(ByRef $avArray, ByRef $iStart, ByRef $iEnd)

    Local $vTmp
    If ($iEnd - $iStart) < 15 Then
        Local $i, $j, $vCur
        For $i = $iStart + 1 To $iEnd
            $vTmp = $avArray[$i]
            If IsNumber($vTmp) Then
                For $j = $i - 1 To $iStart Step -1
                    $vCur = $avArray[$j]
                    If ($vTmp >= $vCur And IsNumber($vCur)) Or (Not IsNumber($vCur) And StringCompare($vTmp, $vCur) >= 0) Then ExitLoop
                    $avArray[$j + 1] = $vCur
                Next
            Else
                For $j = $i - 1 To $iStart Step -1
                    If (StringCompare($vTmp, $avArray[$j]) >= 0) Then ExitLoop
                    $avArray[$j + 1] = $avArray[$j]
                Next
            EndIf
            $avArray[$j + 1] = $vTmp
        Next
        Return
    EndIf
    Local $L = $iStart, $R = $iEnd, $vPivot = $avArray[Int(($iStart + $iEnd) / 2)], $fNum = IsNumber($vPivot)
    Do
        If $fNum Then
            While ($avArray[$L] < $vPivot And IsNumber($avArray[$L])) Or (Not IsNumber($avArray[$L]) And StringCompare($avArray[$L], $vPivot) < 0)
                $L += 1
            WEnd
            While ($avArray[$R] > $vPivot And IsNumber($avArray[$R])) Or (Not IsNumber($avArray[$R]) And StringCompare($avArray[$R], $vPivot) > 0)
                $R -= 1
            WEnd
        Else
            While (StringCompare($avArray[$L], $vPivot) < 0)
                $L += 1
            WEnd
            While (StringCompare($avArray[$R], $vPivot) > 0)
                $R -= 1
            WEnd
        EndIf
        If $L <= $R Then
            $vTmp = $avArray[$L]
            $avArray[$L] = $avArray[$R]
            $avArray[$R] = $vTmp
            $L += 1
            $R -= 1
        EndIf
    Until $L > $R
    _RFLTA_QuickSort($avArray, $iStart, $R)
    _RFLTA_QuickSort($avArray, $L, $iEnd)

EndFunc   ;==>_RFLTA_QuickSort

; #INTERNAL_USE_ONLY#============================================================================================================
; Name...........: _RFLTA_ArrayConcatenate
; Description ...: Joins 2 arrays
; Syntax ........: _RFLTA_ArrayConcatenate(ByRef $avArrayTarget, Const ByRef $avArraySource)
; Parameters ....: $avArrayTarget - Base array
;                  $avArraySource - Array to add from element 1 onwards
; Return values .: None - array modified ByRef
; Author ........: Ultima
; Modified.......: Melba23
; Remarks .......: This function is used internally by _RecFileListToArray
; ===============================================================================================================================
Func _RFLTA_ArrayConcatenate(ByRef $avArrayTarget, Const ByRef $avArraySource)

    Local $iUBoundTarget = UBound($avArrayTarget) - 1, $iUBoundSource = UBound($avArraySource)
    ReDim $avArrayTarget[$iUBoundTarget + $iUBoundSource]
    For $i = 1 To $iUBoundSource - 1
        $avArrayTarget[$iUBoundTarget + $i] = $avArraySource[$i]
    Next

EndFunc   ;==>_RFLTA_ArrayConcatenate

; #INTERNAL_USE_ONLY#============================================================================================================
; Name...........: _RFLTA_ArrayInsert
; Description ...: Insert element into array
; Syntax ........: _RFLTA_ArrayInsert(ByRef $avArray, $iElement, $vValue = "")
; Parameters ....: $avArray - Array to modify
;                  $iElement - Index position for insertion
;                  $vValue - Value to insert
; Return values .: None - array modified ByRef
; Author ........: Jos van der Zande, Ultima
; Modified.......: Melba23
; Remarks .......: This function is used internally by _RecFileListToArray
; ===============================================================================================================================
Func _RFLTA_ArrayInsert(ByRef $avArray, $iElement, $vValue = "")

    Local $iUBound = UBound($avArray) + 1
    ReDim $avArray[$iUBound]
    For $i = $iUBound - 1 To $iElement + 1 Step -1
        $avArray[$i] = $avArray[$i - 1]
    Next
    $avArray[$iElement] = $vValue

EndFunc   ;==>_RFLTA_ArrayInsert

Any comments from others also welcomed, of course! :nuke:

M23

Edit: Small, but important, change to SRE code - they really make my brain hurt at times!

Edited December 9, 2010 by Melba23

Melba23 · December 10, 2010

Deathbringer,

Thank you for having taken the time to comment and test.

Is there any way to make it sort faster?

If you knew the time I have spent getting that sort algorithm to go as fast as it does now! :nuke:

Seriously, the array functions in AutoIt are not the fastest - as you can see I rewrote the existing UDF code to streamline it as much as possible by removing nearly all error-checking (for example checking if the parameters are actually arrays) and redundant code (such as multiple dimension checking). I also wanted a very particular sort result where, if files and folders are listed with paths, the files follow their folder - a straight alphabetic sort does not give you that necessarily. That drove up the time a little but you can, of course, turn the sorting off if you wish.

less than a tenth of the size of your UDF

Given the size of compiled AutoIt scripts I am not losing any sleep over that. :shifty:

The speed comparisons are interesting. I did an awful lot of these using various search and sort algorithms while developing the UDF and found that it is the very first run where the code takes time - something which I imagine would affect any code as the FileFindFirst/NextFile calls have to read the disk directly. The results were pretty stable (and much faster) once the path data was in the cache. I never tested it against a raw DOS search - I must confess I am quite pleased at the results you posted.

Thanks again for the comments.

M23

Melba23 · December 10, 2010

Deathbringer,

If you run into any problems with the "trimming", please do not hesitate to let me know - I would be delighted to help.

M23

Melba23 · December 11, 2010

Deathbringer,

I assumed that if I reversed the order by which $asFolderSearchList was processed, then the directories would be in order without the need to be sorted

Not a valid assumption I am afraid. You can get the files/folder returned in non-sorted order (in my experience on 98/XP/Vista it seems to depend mainly on the order they were created on the disk) - they are not necessarily be in alphabetic order in all cases.

I do not want to rain on your parade, but that is why I found the various sorts were necessary. Now it might well be that your specific folder/file structure is such that you do get them in the order you require, but I would be careful making the assumption that it will be the case for all searches.

M23

Melba23 · December 11, 2010

Deathbringer,

Certainly my experience has been that files are not necessarily returned in alphabetical order - but it is your function!

As to recursion level, you unwind the recursion each time you come back up so I forsee no problems there - but if you are concerned you can check it with this slightly modified script to show how deep you are going. AutoIt has a recursion limit of 5100 - I get nowhere near it even on very deep trees:

#AutoIt3Wrapper_Au3Check_Parameters=-q -d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6
#include <Array.au3>
#NoTrayIcon

Global $sCurrentPath = @ScriptDir & "\"
Global $asFolderSearchList[100] = [1, $sCurrentPath]

_Test($sCurrentPath)

Func _Test($sInitialPath)
    Local $iTimer = TimerInit()
    Local $hSearch = FileFindFirstFile($sInitialPath & "*"), $sName
    While 1
        $sName = FileFindNextFile($hSearch)
        If @error Then ExitLoop
        If Not @extended Then ContinueLoop
        _RFLTA_AddToList($asFolderSearchList, $sInitialPath & $sName)
        _GetSubDirs($sInitialPath & $sName, 1)
    WEnd
    FileClose($hSearch)
    ReDim $asFolderSearchList[$asFolderSearchList[0] + 1]
    ConsoleWrite("! Total - " & TimerDiff($iTimer) & @CRLF)
    _ArrayDisplay($asFolderSearchList)
EndFunc   ;==>_Test

Func _GetSubDirs($sPath_Sub, $iCurrLevel)
    ConsoleWrite("Searching Level " & $iCurrLevel & @CRLF)
    Local $hSearch_Sub = FileFindFirstFile($sPath_Sub & "\*"), $sName_Sub
    While 1
        $sName_Sub = FileFindNextFile($hSearch_Sub)
        If @error Then ExitLoop
        If Not @extended Then ContinueLoop
        _RFLTA_AddToList($asFolderSearchList, $sPath_Sub & "\" & $sName_Sub)
        _GetSubDirs($sPath_Sub & "\" & $sName_Sub, $iCurrLevel + 1)
    WEnd
    FileClose($hSearch_Sub)
EndFunc   ;==>_GetSubDirs

Func _RFLTA_AddToList(ByRef $asList, $sValue)
    $asList[0] += 1
    If UBound($asList) <= $asList[0] Then ReDim $asList[UBound($asList) * 2]
    $asList[$asList[0]] = $sValue
EndFunc   ;==>_RFLTA_AddToList

M23

Edit:

Found this on MSDN - looks like you should be OK on NTFS. Perhaps it was my Win98/FAT experience that coloured my impressions:

"The order in which this function returns the file names is dependent on the file system type. With the NTFS file system and CDFS file systems, the names are usually returned in alphabetical order. With FAT file systems, the names are usually returned in the order the files were written to the disk, which may or may not be in alphabetical order. However, as stated previously, these behaviors are not guaranteed."

Edited December 11, 2010 by Melba23

Melba23 · December 11, 2010

Deathbringer,

Glad you have got it running as you wish.

M23

michaelslamet · March 5, 2011

Deathbringer,
Glad you have got it running as you wish.
M23

Hi M23,

Can use show me example code using this UDF? I would like to search folder ABC from $source_folder. If found then copy entire folder to $dest_folder. If not found that find any files which name contain string ABC in it and copy to $dest_folder

Thanks for your help and your time

FileListToArray recursive with max depth

Recommended Posts

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members