FileListToArray comprehensive comparison

[Scriptonize]

Hi weaponx,

FYI, this all is a bit off topic.

I was using the first version from your example #1.

After reading your reply I tried the updated code.

That one doesn't count right (at least on my pc).

Situation tested:

Folder structure:

C:\Temp\ <-- two files in sub folder (Document 0-1.txt and Document 0-2.txt)

C:\Temp\fldr1\subfldr1-1\ <-- two files in sub folder (Document 1-1.txt and Document 1-1.txt)

C:\Temp\fldr1\subfldr1-2\ <-- two files in sub folder (Document 2-1.txt and Document 2-2.txt)

So the total amount of files = 6

When I run your first placed code #1, it returns and give me the amount of 6 found files, but in the array listbox there are only 5 files listed

When I run your second placed code #1, the result is 7 found files (which isn't right)

When I run your first placed code #1 (modified by me), it returns 6 found files and lists all 6 of them.

This is all no big deal, just a minor issue.

[weaponx]

I don't know what to to tell you. I copied and pasted the posted code into a new script just now, and ran it on a dummy folder with 17 pictures plus 22 pictures in a subfolder and it shows 39. Ubound (last msgbox item) shows 40 just like it should.

Don't forget this new version uses RegEx filters so if you want it to find everything put "." for the filter.

;Match all files

RecursiveFileSearch("C:\", ".")

;Match mp3s

RecursiveFileSearch("C:\", "(?i)\.(mp3)")

;Match mp3s + exes

RecursiveFileSearch("C:\", "(?i)\.(mp3|exe)")

Edited December 3, 2007 by weaponx

[Scriptonize]

I don't know what to to tell you. I copied and pasted the posted code into a new script just now, and ran it on a dummy folder with 17 pictures plus 22 pictures in a subfolder and it shows 39. Ubound (last msgbox item) shows 40 just like it should.

Don't forget this new version uses RegEx filters so if you want it to find everything put "." for the filter.

;Match all files

RecursiveFileSearch("C:\", ".")

;Match mp3s

RecursiveFileSearch("C:\", "(?i)\.(mp3)")

;Match mp3s + exes

RecursiveFileSearch("C:\", "(?i)\.(mp3|exe)")

Yep, that was the cause of the problem of my test run of sample #1 (version 2).

I used *.* for finding all files instead of "." (dot).

Thanks for clearing that up.

[weaponx]

Sorry about that. It's not completely ready for distribution, although since I am using RegEx as the file filter there is no need to have an exclude option since filenames can be negated in the expression. The only thing it's missing is the option to recurse.

[weaponx]

Updated my version: RecursiveFileSearch (#1)

-Added recursion option

-Cleaned up variable names (still don't follow UDF standards, but WeaponX standards)

I will be retesting all of the entries at some point from a cold boot to eliminate any speed / caching issues.

[tazlikesrobots]

Is there a way to make the it skip certain directories? I.E. skip c:\windows, c:\temp, etc.

Thanks

PS great script!

[weaponx]

I updated the code for my RecursiveFileSearch.

-Added new parameter for regex folder matching.

-Added new parameter for return type (Files, Folders, Both)

This WILL break older calls to the function due to parameters being added.

Here is an example which searches for *.mp3 and *.txt, excluding folders named "Photos" and "Movies", returns files only:

#include <array.au3>
#include "RecursiveFileSearch.au3"
   
   $Array = RecursiveFileSearch(@DesktopDir & "\Test", "(?i)\.(mp3|txt)","(?!(Photos|Movies)\b)\b.+", 1, true)
   
   _ArrayDisplay($Array)

EDIT: Removed autoit tags (damn smiley faces)

EDIT: Fixed regex skipping filenames containing parentheses

Edited February 22, 2008 by weaponx

[tazlikesrobots]

I updated the code for my RecursiveFileSearch.

-Added new parameter for regex folder matching.

-Added new parameter for return type (Files, Folders, Both)

This WILL break older calls to the function due to parameters being added.

Here is an example which searches for *.mp3 and *.txt, excluding folders named "Photos" and "Movies", returns files only:

#include <array.au3>

$Array = RecursiveFileSearch(@DesktopDir & "\Test", "(?i)\.(mp3|txt)","(?!(Photos|Movies)\ B) \b\w+", 1, true)

_ArrayDisplay($Array)

Thank you! for the update. You saved me lots of work

[randallc]

Hi,

This really looks the goods now; I presume

1. Allows filtering files and folders in the one command, and 1 run through.

2. Won't get any dupes? - so no need to remove them (is a fault of SmokN's btw when multiple file filters give overlapping results) - actually maybe this is still a problem for yours?

3. By "release" I guess you mean you will put the RegExp itself within the func, so the parameters are simpler?

4. Is there now any feature this does not perform?!

5. Testing; I am going to set up 2 VMWare instances to run the test scripts for timing; this will eliminate the need to reboot, I think.. I'll let you know. - I doubt there are any speed problems with yours, so it will be mainly a matter of proving that, and looking at any discrepancies to check whose script gets wrong results, if any!

Thanks, Randall

[PS not sure it is workin properly yet, though?

6. This still returns "Program Files" subfolders, and "Documents and Settings" folders?

$Array = RecursiveFileSearch2("c:", "(?i)\.","(?!(WINDOWS|MSOCache|backup|Documents and Settings|Program Files)\\b\w+", 2, true)

7. It is case sensitive as it stands, for folders?

8. Question of exact vs partial match?]

Edited February 21, 2008 by randallc

[weaponx]

I only tested on a small set of folders so there may be an issue I have missed. I know the exclusion regex has to be very specific to work correctly. Will return shortly.

[lemony]

hey weaponx, just wanted to say you did an excellent job with this thread

[smashly]

Hi WeaponX and thank you for your testing and code.

I've been using your function #1 for about month and found it to nice to use.

Some bugs and observations on function #1:

Using False for $RFSrecurse and 0 or 2 for $RFSFlag then folders are not returned.

Reason is there is no allowance for no recurse of folders in this block of code.

;If folder and recurse flag is set and regex matches
         If StringInStr(FileGetAttrib($RFSstartDir & $RFSnext), "D") Then
             
             If $RFSrecurse AND StringRegExp($RFSnext, $RFSFolderpattern, 0) Then
                 RecursiveFileSearch($RFSstartDir & $RFSnext, $RFSFilepattern, $RFSFolderpattern, $RFSFlag, $RFSrecurse, $RFSdepth + 1)
                 If $RFSFlag <> 1 Then
                ;Append folder name to array
                     $RFSarray[$RFSarray[0] + 1] = $RFSstartDir & $RFSnext
                     $RFSarray[0] += 1
                 EndIf
             EndIf
         ElseIf StringRegExp($RFSnext, $RFSFilepattern, 0) AND $RFSFlag <> 2 Then
        ;Append file name to array
             $RFSarray[$RFSarray[0] + 1] = $RFSstartDir & $RFSnext
             $RFSarray[0] += 1
         EndIf

Using True for $RFSrecurse and using the RFSFolderpattern makes the function not recurse.

(I didn't look to far into this as it's something I haven't needed yet.)

Not clearing the Global $RFSarray on exit of the function means the array is in memory twice.

This means the user needs to clear the Global array as well as the returned array when done if they

are keeping the script running after using the RFS function and want the memory cleared.

Example: Search my 40GB hd, it contains 26GB of data consisting of 52195 folders and files total.

When the array is returned by the function my memory usage is at 259MB.

I finish with the return and clear the returned array ($array = 0) and I have memory in use 224MB.

I then clear the Global $RFSarray array ($RFSarray = 0) and I have memory in use 209MB.

Clearing the Global array after the function has returned I'm about 10 ~ 15MB better off in my above situation.

I think maybe copy the Global array to a Local array and clear the Global array and return the Local array.

This would save the user from having to do it after the function has finished.

Near end of function maybe something like:

If $RFSdepth = 0 Then
         Redim $RFSarray[$RFSarray[0] + 1]
         Local $aRet = $RFSarray
         $RFSarray = 0       
         Return $aRet
     EndIf

Sorry to bring up an old post, but it wasn't until today that I actually sat down and looked at how your code worked.

I managed to customize your code to my needs and gained a little more speed all over from your original code.

Once again thank you for education in recursive file search.

Cheers

Edit: I meant to say #1 not #7

Edited May 27, 2008 by smashly

[BrettF]

Hi WeponX

I really like your function, helps a lot.

I am using the following:

#Include <Array.au3>
$filename = "D:\My Documents\"
If StringRight($filename, 1) = "\"  Then $filename = StringTrimRight($filename, 1)
$pattern = "\.(mp3|mp2|mp1|ogg|wav|aiff|xm|it|s3m|mod|mtm|umx|flac)" 
$searching = RecursiveFileSearch($filename, $pattern, ".", 1, True)
_ArrayDisplay($searching)

And reading your explanation, it should be case sensitive?

When searching for .xm files, I end up with .xml (and .itc) files as well... Is that meant to happen? Is there a way around it?

Thanks,

Brett

[weaponx]

Hi WeponX

I really like your function, helps a lot.

I am using the following:

#Include <Array.au3>
 $filename = "D:\My Documents\"
 If StringRight($filename, 1) = "\"  Then $filename = StringTrimRight($filename, 1)
 $pattern = "\.(mp3|mp2|mp1|ogg|wav|aiff|xm|it|s3m|mod|mtm|umx|flac)" 
 $searching = RecursiveFileSearch($filename, $pattern, ".", 1, True)
 _ArrayDisplay($searching)

And reading your explanation, it should be case sensitive?

When searching for .xm files, I end up with .xml (and .itc) files as well... Is that meant to happen? Is there a way around it?

Thanks,

Brett

Put a \z at the end of the pattern.

[BrettF]

Thank you! Works perfectly!

[weaponx]

Hi WeaponX and thank you for your testing and code.

I've been using your function #1 for about month and found it to nice to use.

Some bugs and observations on function #1:

Using False for $RFSrecurse and 0 or 2 for $RFSFlag then folders are not returned.

Reason is there is no allowance for no recurse of folders in this block of code.

Not clearing the Global $RFSarray on exit of the function means the array is in memory twice.

This is by no means the fastest code, but it is quite fast for how little code there is. The use of the global array avoids making the user declare an array before hand, making it a little simpler (also it can't be declared locally in a recursive function).

There are some quirks in the current example:

-Folders that don't match the folder pattern won't be recursed at all

-The regular expression only matches against the file/folder name, not the FULL path, this is generally okay but if you match the absolute path you can match files that have a specific parent folder

I guess its simple enough that people can tailor it as needed. I just hated having to include some of the other FileListToArray UDF's because they were more cumbersome.

[cypher175]

How can I use this Function in conjunction with what I'm trying to do here in my other thread..??

http://www.autoitscript.com/forum/index.php?showtopic=84226

[sshrum]

I too am on a quest for a more functional, faster FileSearch().

I like your recursivefilesearch function however, I hit a snag (sort of)...let's say you want to list all folders in a folder (no sub-subfolders; recurse=false).

You have a flag for disabling Recurse so I figured this should work.

$aDirs = _RecursiveFileSearch("o:\player\hotkeys", ".", ".", 2, False)

I set recurse to false and I get nothing back (empty array).

$aDirs = _RecursiveFileSearch("o:\player\hotkeys", ".", ".", 2, True, 1)

I set recurse to true and depth to 1; the function bombs out.

Is this a bug or am I passing the arguments wrong? does this go beyond what the function is designed to do?

TIA

Edited March 17, 2009 by sshrum

[sshrum]

I made a small add to you code to handle non-recursive folder only returns:

Func _RecursiveFileSearch($RFSstartDir, $RFSFilepattern = ".", $RFSFolderpattern = ".", $RFSFlag = 0, $RFSrecurse = True, $RFSdepth = 0)
    If StringRight($RFSstartDir, 1) <> "\" Then $RFSstartDir &= "\"
    If $RFSdepth = 0 Then
        $RFSfilecount = DirGetSize($RFSstartDir, 1)
        Global $RFSarray[$RFSfilecount[1] + $RFSfilecount[2] + 1]
    EndIf
    $RFSsearch = FileFindFirstFile($RFSstartDir & "*.*")
    If @error Then Return
    While 1
        $RFSnext = FileFindNextFile($RFSsearch)
        If @error Then ExitLoop
        If StringInStr(FileGetAttrib($RFSstartDir & $RFSnext), "D") Then
            If $RFSrecurse And StringRegExp($RFSnext, $RFSFolderpattern, 0) Then
                _FileSearch($RFSstartDir & $RFSnext, $RFSFilepattern, $RFSFolderpattern, $RFSFlag, $RFSrecurse, $RFSdepth + 1)
            EndIf
            If $RFSFlag <> 1 Then
                $RFSarray[$RFSarray[0] + 1] = $RFSstartDir & $RFSnext
                $RFSarray[0] += 1
            EndIf   
        ElseIf StringRegExp($RFSnext, $RFSFilepattern, 0) And $RFSFlag <> 2 Then
            $RFSarray[$RFSarray[0] + 1] = $RFSstartDir & $RFSnext
            $RFSarray[0] += 1
        EndIf
    WEnd
    FileClose($RFSsearch)
    If $RFSdepth = 0 Then
        ReDim $RFSarray[$RFSarray[0] + 1]
        Return $RFSarray
    EndIf
EndFunc

Edited March 17, 2009 by sshrum

[caozhiju]

Try it ,so fast!

#692828