List files in folders and subfolders quickly.

[Belini]

Use a function that lists 39,709 files in 44 seconds in folders and subfolders, is there another function that lists faster?

#include <String.au3>
#include <Array.au3>
global $listing, $timer
$directory = FileSelectFolder("Browse for Folder.", "")
$timer = TimerInit()
list($directory, 0)
time()
Func list($path = "", $counter = 0)
$counter = 0
$path &= '\'
Local $list_files = '', $file, $demand_file = FileFindFirstFile($path & '*')
If $demand_file = -1 Then Return ''
While 1
  $file = FileFindNextFile($demand_file)
  If @error Then ExitLoop
  If @extended Then
   If $counter >= 10 Then ContinueLoop
   list($path & $file, $counter + 1)
  Else
            $listing &= $path & $file & "|"
  EndIf
WEnd
FileClose($demand_file)
EndFunc
func time()
local $total_time = TimerDiff($timer)
local $minutes = Int(Mod($total_time, 3600000) / 60000)
local $seconds = Int(Mod(Mod($total_time, 3600000), 60000) / 1000)
local $min_seg = StringFormat("%02d:%02d", $minutes, $seconds)
MsgBox(4096,'Total time', $min_seg)
$listing = StringTrimRight($listing, 1)
$listing = _StringExplode($listing, "|", 0)
_ArraySort($listing)
_ArrayDisplay($listing, "files found")
endfunc

Print:

[BrewManNH]

I'll direct you to that did a speed test comparing various FileListToArray functions floating around.

[Belini]

@ BrewManNH put this function in the test and it was faster.

-> -> List_files returned 42264 items in 0.42 seconds.

-> FileListToArrayRecursive returned 42264 items in 0.92 seconds.

-> FileListToArray3 returned 42264 items in 1.84 seconds.

-> RecFileListToArray returned 42264 items in 1.81 seconds.

-> FileListToArrayEx returned 42264 items in 0.88 seconds.

-> FileListToArrayPlus returned 42264 items in 0.75 seconds.

-> FileListTreeToArray returned 42264 items in 2.66 seconds.

Speed Test.rar

[AZJIO]

Belini

Func list($path = "", $counter = 0)

$counter = 0

If modify my function

Func __FO_FileSearchAll(ByRef $sFileList, $sPath, $iDepth, $iCurD = 0)
    Local $sFile, $s = FileFindFirstFile($sPath & '*')
    If $s = -1 Then Return
    While 1
        $sFile = FileFindNextFile($s)
        If @error Then ExitLoop
        If @extended Then
            If $iCurD >= $iDepth Then ContinueLoop
            __FO_FileSearchAll($sFileList, $sPath & $sFile & '', $iDepth, $iCurD + 1)
        Else
            $sFileList &= $sPath & $sFile & @CRLF
        EndIf
    WEnd
    FileClose($s)
EndFunc   ;==>__FO_FileSearchAll

you get this

Func __FO_FileSearchAll(ByRef $sFileList, $sPath)
    Local $sFile, $s = FileFindFirstFile($sPath & '*')
    If $s = -1 Then Return
    While 1
        $sFile = FileFindNextFile($s)
        If @error Then ExitLoop
        If @extended Then
            __FO_FileSearchAll($sFileList, $sPath & $sFile & '')
        Else
            $sFileList &= $sPath & $sFile & @CRLF
        EndIf
    WEnd
    FileClose($s)
EndFunc   ;==>__FO_FileSearchAll

Edited August 27, 2012 by AZJIO

[Belini]

@AZJIO auditioned using the ByRef and speed had a slight decline.

-> -> Dos dir command returned 120888 items in 83.72 seconds.

-> FileListToArrayRecursive returned 122225 items in 3.43 seconds.

-> FileListToArray3 returned 122225 items in 19.68 seconds.

-> RecFileListToArray returned 122225 items in 6.41 seconds.

-> FileListToArrayEx returned 122225 items in 3.44 seconds.

-> FileListToArrayPlus returned 122225 items in 2.99 seconds.

-> FileListTreeToArray returned 122225 items in 11.92 seconds.

-> _List_FilesEx (normal) returned 122225 items in 2.16 seconds.

-> _List_FilesEx (ByRef) returned 122225 items in 2.27 seconds.

_List_FilesEx.au3

expandcollapse popup

#include <RecFileListToArray.au3>
#include <FileListToArray3.au3>
#include <File.au3>
#include <Array.au3>
#include <FileListTreeToArray.au3>
#include <FileListToArrayRecursive.au3>
#include <FileListToArrayEx.au3>
#include <FileListToArrayPlus.au3>
#include <_List_FilesEx.au3>; Belini

FileDelete(@ScriptDir & "Speed Test.ini")

$Location = "C:"
;$Location = FileSelectFolder("Procurando pasta.", "")

ConsoleWrite("+>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++" & @CRLF)
ConsoleWrite("-> OS:" & @OSVersion  & " " & @OSServicePack & ", OS Arch:" & @OSArch  & ", Language:" & @OSLang & ", AutoIt: " & @AutoItVersion & @CRLF & @CRLF)
;~ _ResultWrite("Function Name","Item Count","Duration (s)")
;~ ConsoleWrite("-> |=============================================================|" & @CRLF)

;~ Dos Dir command Start
Local $DosFileList, $DosFileList_File = @ScriptDir & 'FileList.txt'
If FileExists($DosFileList_File) Then FileDelete($DosFileList_File)
$DosTimer = TimerInit()
RunWait(@ComSpec & " /c " & 'dir "' & $Location & '" /B /S > "' & $DosFileList_File & '"',@ScriptDir,@SW_HIDE)
_FileReadToArray($DosFileList_File,$DosFileList)
$DosTimer = Round(TimerDiff($DosTimer)/1000,2)
_ResultWrite_ForumFriendly("Dos dir command", UBound($DosFileList)-1, $DosTimer)
If FileExists($DosFileList_File) Then FileDelete($DosFileList_File)
;~ Dos Dir command End

;~ FileListToArrayRecursive Start
$FileListToArrayRecursive_Timer = TimerInit()
$FileListToArrayRecursive_List = _FileListToArrayRecursive($Location,"*",4)
$FileListToArrayRecursive_Timer =  Round(TimerDiff($FileListToArrayRecursive_Timer)/1000,2)
_ResultWrite_ForumFriendly("FileListToArrayRecursive", UBound($FileListToArrayRecursive_List )-1,$FileListToArrayRecursive_Timer)
;~ FileListToArrayRecursive End

;~ _FileListToArray3 Start
$FileListtoArray_Timer = TimerInit()
$FileListtoArray_List = _FileListToArray3($Location,"*", 0, 1, 1)
$FileListtoArray_Timer = Round(TimerDiff($FileListtoArray_Timer)/1000,2)
_ResultWrite_ForumFriendly("FileListToArray3", UBound($FileListtoArray_List)-1, $FileListtoArray_Timer)
;~ _FileListToArray3 End

;~ _RecFileListToArray Start
$RecFileListToArray_Timer = TimerInit()
$RecFileListToArray_List = _RecFileListToArray($Location, "*",0,1,0,2)
$RecFileListToArray_Timer =  Round(TimerDiff($RecFileListToArray_Timer)/1000,2)
_ResultWrite_ForumFriendly("RecFileListToArray",UBound($RecFileListToArray_List)-1, $RecFileListToArray_Timer)
;~ _RecFileListToArray EndFunc

;~ _FileListToArrayEx Start
$FileListToArrayEx_Timer = TimerInit()
$FileListToArrayEx_List = _FileListToArrayEx($Location,"*",0,2)
$FileListToArrayEx_Timer =  Round(TimerDiff($FileListToArrayEx_Timer)/1000,2)
_ResultWrite_ForumFriendly("FileListToArrayEx",UBound($FileListToArrayEx_List)-1,$FileListToArrayEx_Timer)
;~ _FileListToArrayEx End

;~ _FileListToArrayPlus Start
$FileListToArrayPlus_Timer = TimerInit()
$FileListToArrayPlus_List = _FileListToArrayPlus($Location,"*",0,0,"",2,True)
$FileListToArrayPlus_Timer =  Round(TimerDiff($FileListToArrayPlus_Timer)/1000,2)
_ResultWrite_ForumFriendly("FileListToArrayPlus",UBound($FileListToArrayPlus_List)-1,$FileListToArrayPlus_Timer)
;~ _FileListToArrayPlus End

;~ _FileListTreeToArray Start
$FileListTreeToArray_Timer = TimerInit()
$FileListTreeToArray_List = _FileListTreeToArray($Location)
$FileListTreeToArray_Timer =  Round(TimerDiff($FileListTreeToArray_Timer)/1000,2)
_ResultWrite_ForumFriendly("FileListTreeToArray",UBound($FileListTreeToArray_List)-1,$FileListTreeToArray_Timer)
;~ _FileListTreeToArray End

; =============================================================================================================== new test
;~ _List_FilesEx (normal) Start
$List_files_Timer = TimerInit()
$List_files_List = list1($Location)
$List_files_Timer =  Round(TimerDiff($List_files_Timer)/1000,2)
$List_files_List = _StringExplode($listing1, "|", 0)
_ResultWrite_ForumFriendly("_List_FilesEx (normal)", UBound($List_files_List)-1,$List_files_Timer)
;~ _List_FilesEx (normal) End

;~ _List_FilesEx (ByRef) Start
$List_files_Timer = TimerInit()
$List_files_List = list2($listing2, $Location)
$List_files_Timer =  Round(TimerDiff($List_files_Timer)/1000,2)
$List_files_List = _StringExplode($listing2, "|", 0)
_ResultWrite_ForumFriendly("_List_FilesEx (ByRef)", UBound($List_files_List)-1,$List_files_Timer)
;~ _List_FilesEx (ByRef) End
; =============================================================================================================== new test

ConsoleWrite("+>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++" & @CRLF)

Func _ResultWrite_ForumFriendly($sFuncName, $sItemCount, $sTime)
    ConsoleWrite("-> " & $sFuncName & " returned " & $sItemCount & " items in " & $sTime & " seconds." & @CRLF)
    FileWriteLine(@ScriptDir & "Speed Test.ini", $sFuncName & " returned " & $sItemCount & " items in " & $sTime & " seconds." & @CRLF)
EndFunc

Func _ResultWrite($sFuncName, $sItemCount, $sTime)
    If StringLen($sFuncName) < 19 Then $sFuncName &= @TAB
    If StringLen($sItemCount) < 6 Then $sItemCount &= @TAB
    If StringLen($sTime) < 6 Then $sTime &= @TAB
    ConsoleWrite("-> | " & $sFuncName & @TAB & "| " & $sItemCount & @TAB & "| " & $sTime & @TAB & " |" & @CRLF)
    FileWriteLine(@ScriptDir & "Speed Test.ini", $sFuncName & " returned " & $sItemCount & " items in " & $sTime & " seconds." & @CRLF)
EndFunc

Note: Listed 122.225 files from C:

Link: http://www.mediafire.com/?y274s1311j1iao2

Edited August 28, 2012 by Belini

[AZJIO]

Belini

Test takes a few times to get the mid-statistics. ByRef theoretically should not be slow, because without it, the data is copied to a new variable, and it's too much action. It is better to pass a pointer using a single variable.

[Bowmore]

@Belini

I ran your script on my PC and got slightly different relative speeds although I have a lot more files on my C: drive. The biggest difference is the performance of DOS dir. I've run the tests a number of times and the results are fairly consistent. One thing I would mention is that it is to be expected that the function that have lots of options are always going to be slower than a stripped down version without all the options removed, which is why when when speed is a priority I always use a custom function tailored to just what is required for my script.

I've ordered the results slowest to fastest.

-> FileListToArray3 returned 258901 items in 22.4 seconds.

-> FileListTreeToArray returned 258901 items in 17.17 seconds.

-> RecFileListToArray returned 258901 items in 7.49 seconds.

-> Dos dir command returned 256545 items in 6.28 seconds.

-> FileListToArrayRecursive returned 258901 items in 5.22 seconds.

-> FileListToArrayEx returned 258901 items in 4.79 seconds.

-> FileListToArrayPlus returned 258901 items in 4.48 seconds.

-> _List_FilesEx (normal) returned 258901 items in 4.06 seconds.

-> _List_FilesEx (ByRef) returned 258901 items in 3.64 seconds.

As an after thought I tried it on my large data drive and ran into problems with 2 of the functions running out of memory.

-> FileListToArray3 returned 1370660 items in 26.37 seconds.

-> RecFileListToArray returned 1370660 items in 22.9 seconds.

-> Dos dir command returned 1370427 items in 14.58 seconds.

-> FileListToArrayRecursive returned 1370660 items in 12.63 seconds.

-> FileListToArrayEx returned 1370660 items in 11.59 seconds.

-> FileListToArrayPlus returned 1370660 items in 9.66 seconds.

-> _List_FilesEx (ByRef) returned 1370660 items in 6.17 seconds.

-> _List_FilesEx (normal) Fails to complete (Out of memmory). Recursion generates multiple copies of list

-> FileListTreeToArray Fails to complete (Out of memmory). String* functions need multiple copies of list

Edited August 29, 2012 by Bowmore

[AZJIO]

Why you're not testing my function and some of the alternate list?

Test in one setting is not correct. For example my function when "all files" uses a feature that ignores the comparison operation. And when using the mask difference will be much smaller.

Try calling the mask "*.gif|*.txt|*.htm"

Edited August 29, 2012 by AZJIO

[guinness]

Even though _FileListoArrayRec() now exists, I thought I would post this "simple" update of Belini's code minus the global variable as stated above.

expandcollapse popup

#include <WinAPIFiles.au3>

Global Enum Step *2 $GETFILES_NOT_DIRECTORY, $GETFILES_NOT_EXISTS ; GetFiles @error

Example()

Func Example()
    Local $sFileList = ''
    GetFiles($sFileList, @ScriptDir) ; This uses no global variable
    If @error Then
        ConsoleWrite("Error: Check the enumeration above that matches the error code = " & @error & @CRLF)
    Else
        $sFileList = StringReplace($sFileList, '|', @CRLF) ; Replace the pipe char for @CRLF
        ConsoleWrite($sFileList & @CRLF)
    EndIf
EndFunc   ;==>Example

; By guinness on 2015/03/15. Idea by Belini and every AutoIt user who has done file searching.
Func GetFiles(ByRef $sFileList, $sFilePath)
    Local Static $iCounter = 0

    $sFilePath = _WinAPI_PathAddBackslash($sFilePath) ; Add backslash
    If _WinAPI_PathIsDirectory($sFilePath) <> $FILE_ATTRIBUTE_DIRECTORY Then
        Return SetError($GETFILES_NOT_DIRECTORY, 0, '')
    EndIf

    Local $hFileFind = FileFindFirstFile($sFilePath & '*')
    If $hFileFind = -1 Then ; File not found
        Return SetError($GETFILES_NOT_EXISTS, 0, '')
    EndIf

    Local $sFileName = ''
    While True
        $sFileName = FileFindNextFile($hFileFind)
        If @error Then
            ExitLoop
        EndIf

        If @extended Then ; Is directory.
            $iCounter += 1 ; Used for recursion level
            GetFiles($sFileList, $sFilePath & $sFileName)
            $iCounter -= 1 ; Used for recursion level
        Else
            $sFileList &= $sFilePath & $sFileName & '|'
        EndIf
    WEnd
    FileClose($hFileFind)

    If $iCounter = 0 Then ; First recursion level, therefore strip pipe char
        $sFileList = StringTrimRight($sFileList, StringLen('|'))
    EndIf
EndFunc   ;==>GetFiles

Edit: Why isn't this in the Examples section? Very strange considering users are using the UDF by Belini in production code. Oh well, I did it for the right reasons this time around. Edited March 15, 2015 by guinness

[veda2010]

I am not aware of any coding but when i need any specific this, i just google it and tried to modify the code as per my requirement. 

My main requirement now is to pull the .jpg file names from multiple sub folders and put it in one txt file. I wrote a bat file and it's taking 1hour because of the huge data. So, i tired googling it and found this thread.  

Belini's code looks like it will be work for my requirement but Not sure how to tweak it . Can someone help with me this please. Thanks in advance.

Following is Belini's code

Quote

#include <String.au3>
#include <Array.au3>
global $listing, $timer
$directory = FileSelectFolder("Browse for Folder.", "")
$timer = TimerInit()
list($directory, 0)
time()
Func list($path = "", $counter = 0)
$counter = 0
$path &= '\'
Local $list_files = '', $file, $demand_file = FileFindFirstFile($path & '*')
If $demand_file = -1 Then Return ''
While 1
  $file = FileFindNextFile($demand_file)
  If @error Then ExitLoop
  If @extended Then
   If $counter >= 10 Then ContinueLoop
   list($path & $file, $counter + 1)
  Else
            $listing &= $path & $file & "|"
  EndIf
WEnd
FileClose($demand_file)
EndFunc
func time()
local $total_time = TimerDiff($timer)
local $minutes = Int(Mod($total_time, 3600000) / 60000)
local $seconds = Int(Mod(Mod($total_time, 3600000), 60000) / 1000)
local $min_seg = StringFormat("%02d:%02d", $minutes, $seconds)
MsgBox(4096,'Total time', $min_seg)
$listing = StringTrimRight($listing, 1)
$listing = _StringExplode($listing, "|", 0)
_ArraySort($listing)
_ArrayDisplay($listing, "files found")
endfunc

 

Edited June 30, 2016 by veda2010

[Belini]

You just use the File WriteLine:

expandcollapse popup

#include <String.au3>
#include <Array.au3>
Global $listing, $timer
$directory = FileSelectFolder("Browse for Folder.", "")
$timer = TimerInit()
list($directory, 0)
time()
Func list($path = "", $counter = 0)
    $counter = 0
    $path &= '\'
    Local $list_files = '', $file, $demand_file = FileFindFirstFile($path & '*')
    If $demand_file = -1 Then Return ''
    While 1
        $file = FileFindNextFile($demand_file)
        If @error Then ExitLoop
        If @extended Then
            If $counter >= 10 Then ContinueLoop
            list($path & $file, $counter + 1)
        Else
            $listing &= $path & $file & "|"
            FileWriteLine(@ScriptDir & "\listing.ini", $path & $file); Use this to list full path
            ;FileWriteLine(@ScriptDir & "\listing.ini", $file); Use this to list only the name
        EndIf
    WEnd
    FileClose($demand_file)
EndFunc   ;==>list
Func time()
    Local $total_time = TimerDiff($timer)
    Local $minutes = Int(Mod($total_time, 3600000) / 60000)
    Local $seconds = Int(Mod(Mod($total_time, 3600000), 60000) / 1000)
    Local $min_seg = StringFormat("%02d:%02d", $minutes, $seconds)
    MsgBox(4096, 'Total time', $min_seg)
    $listing = StringTrimRight($listing, 1)
    $listing = _StringExplode($listing, "|", 0)
    _ArraySort($listing)
    _ArrayDisplay($listing, "files found")
EndFunc   ;==>time

 

Edited June 30, 2016 by Belini

[veda2010]

Thanks Belini for the quick response. 

I was able to get the files details in txt file. 

 

Is it possible to extract only ".jpg" files?

[Belini]

Quote

I was able to get the files details in txt file.

What details do you need?

If you want to list only .jpg can use StringInStr() to filter!

expandcollapse popup

#include <String.au3>
#include <Array.au3>
Global $listing, $timer
$directory = FileSelectFolder("Browse for Folder.", "")
$timer = TimerInit()
list($directory, 0)
time()
Func list($path = "", $counter = 0)
    $counter = 0
    $path &= '\'
    Local $list_files = '', $file, $demand_file = FileFindFirstFile($path & '*')
    If $demand_file = -1 Then Return ''
    While 1
        $file = FileFindNextFile($demand_file)
        If @error Then ExitLoop
        If @extended Then
            If $counter >= 10 Then ContinueLoop
            list($path & $file, $counter + 1)
        Else
            If StringInStr($file, ".jpg") > 0 Then
                $listing &= $path & $file & "|"
                FileWriteLine(@ScriptDir & "\listing.ini", $path & $file); Use this to list full path
                ;FileWriteLine(@ScriptDir & "\listing.ini", $file); Use this to list only the name
            EndIf
        EndIf
    WEnd
    FileClose($demand_file)
EndFunc   ;==>list
Func time()
    Local $total_time = TimerDiff($timer)
    Local $minutes = Int(Mod($total_time, 3600000) / 60000)
    Local $seconds = Int(Mod(Mod($total_time, 3600000), 60000) / 1000)
    Local $min_seg = StringFormat("%02d:%02d", $minutes, $seconds)
    MsgBox(4096, 'Total time', $min_seg)
    $listing = StringTrimRight($listing, 1)
    $listing = _StringExplode($listing, "|", 0)
    _ArraySort($listing)
    _ArrayDisplay($listing, "files found")
EndFunc   ;==>time

 

Edited June 30, 2016 by Belini

[veda2010]

Thank you. 

That worked very well.

It took 16mins to get all the files names/path from all the sub folders.  Saved 45mins running time.

[Belini]

I was glad to get helped you!