RecFileListToArray - Deprecated

[MrCreatoR]

Melba23

It is very strange that sorting option make the function return much more longer than the search process itself.

I mean this:

_RecFileListToArray(@ProgramFilesDir, "*", 0, 1, 0, 2)

will return much faster then this:

_RecFileListToArray(@ProgramFilesDir, "*", 0, 1, 1, 2)

(sort option is 1)

[Melba23]

It is very strange that sorting option make the function return much more longer

I quite agree that sorting the results adds a fair time to the whole process - but why is it strange that this is the case?

If you do not sort the result you get the files in the order returned by FileFindNextFile which is not necessarily alphabetical order (although it usually is on NTFS systems). And because the process is iterative and not recursive, it actually explores the folders in reverse order. So as I wanted an aphabetical and folder-based list I wrote the algorithm to make sure that the results were sorted that way - which was pretty complex and took most of the development time.

As I pointed out earler in the thread, you would not have been impressed with the early versions - they were painfully slow on anything but the shortest paths. The development of FileFindNextFile setting @extended to indicate folders helped enormously as it removed the need to get and search the attributes, but the main bottleneck is the need to manipulate the arrays holding the returned files and folders - especially if you wish to display both as you are in the examples you post. As Valik pointed out not too long ago, arrays are not exactly AutoIt's strongest suit.

If you have any ideas for speeding the sort algorithm, while retaining the same output, I would be delighted to hear them. I used custom versions of the _Array_* functions with most of the errorchecking removed to increase their speed and tried to keep the ReDim calls to a minimum, but there are certainly things I missed or tricks I do not know.

M23

[MrCreatoR]

If you have any ideas for speeding the sort algorithm, while retaining the same output, I would be delighted to hear them.

First of all you should remove the ReDim in ArrayInsert (redim the array before the loop), it is a major speed limiter.

You also could use temp array to sort (build it during the search).

[Melba23]

MrCreatoR,

Good points - I will look into them. Thanks for the interest and input.

M23

[Bowmore]

@Melba23

As your array of files will is likely to be mostly already sorted a simple insertion sort, such as the one below, will usually be quicker than a quicksort.

Func _InsertionSort(ByRef $aArray, $iMinRow = 0, $iMaxRow = 0 )
    Local $iUB = UBound($aArray) - 1
    Local $tmp = ""
    Local $j = 0

    If $iMaxRow = 0 or $iMaxRow > $iUB Then
        $iMaxRow = $iUB
    EndIf

    If $iMinRow >= $iMaxRow Then
        Return SetError(1,0,0)
    EndIf

  for $i = $iMinRow to $iMaxRow
    $tmp = $aArray[$i];
        $j = $i - 1;
    While ($j >= 0) And ($aArray[$j] > $tmp)
      $aArray[$j + 1] = $aArray[$j]
      $j -= 1;
    WEnd
    $aArray[$j + 1] = $tmp;
    Next
EndFunc

Edit: I should have done more testing before posting. All though an insertion sort is quicker when the data is mostly already sorted in this case if sub folders are searched the resulting array is a long way from being sorted and the insertion sort is much slower than quicksort.

Edited March 8, 2011 by Bowmore

[Melba23]

Bowmore,

if sub folders are searched the resulting array is a long way from being sorted

Too true - and using the iterative method almost guarantees that the files are nowhere near sorted initially. That is why I found the sort algorithm the most complicated part of the UDF.

It gets even worse if you have files in the root folder because a simple insertion sort can stuff them in among the folder files rather than keeping them all together in a section as logic would suggest. Thus the code to keep them in a separate array to be concatenated later.

M23

[Melba23]

New Version - 14 Mar 2011

Changed - New sort algorithm for "return files and folders" search - very, very much faster.

Added - New sort option: 0 = No sort; 1 = Sort files alphabetically; 2 = Leave files in order returned FileFindNextFile - this is usually alphabetical for NTFS drives.

New UDF and zip in first post.

M23

[startrek]

It doesn't search recursively in subfolders if you specify search criteria??

Or am i too tired to understand

$aArray = _RecFileListToArray($sProgFiles & "\AutoIt3\", "*", 0, 1, 1) scans subfolders

$aArray = _RecFileListToArray($sProgFiles & "\AutoIt3\", "*.txt", 0, 1, 1) Do not scan subfolders

or

$aArray = _RecFileListToArray($sProgFiles & "\AutoIt3", "*", 1, 0, 1, 0, "*.exe") Do not scan subfolders

Edited March 21, 2011 by startrek

[Melba23]

startrek,

Welcome to the AutoIt forum.

The UDF will search recursively if you set the $iRecur parameter to anything other than 0. But the search parameters apply to both files and folders.

So in your 3 examples:

_RecFileListToArray($sProgFiles & "\AutoIt3\", "*", 0, 1, 1) will search in all subfolders and sort the resulting array of all folders and files - because the mask is "*" and everything matches.

_RecFileListToArray($sProgFiles & "\AutoIt3\", "*.txt", 0, 1, 1) will search in folders which match the mask for files which match the mask and sort the resulting array of all folders and files - but it will not search in subfolders that do not match the "*.txt" mask, which is probably the majority.

_RecFileListToArray($sProgFiles & "\AutoIt3", "*", 1, 0, 1, 0, "*.exe") will search in folders which match the mask for files which match the mask and are not excluded and sort the resulting array of files - here you should match all the folders and only exclude .exe files.

This behaviour was inherited from the _FileListToArray UDF included with the standard install. I hope the explanation is sufficient - please ask again if not. And you are not the first to get caught out by it either - both in the original or this more capable UDF .

By a strange coincidence I am currently working on a new version of the UDF which will allow you to search all folders for files which match the mask - thus removing the limitation you have come across. Check back in a few days - I am testing it hard at the moment.....in between decorating my bedroom, reoiling the garden furniture, playing golf, getting the lawnmower started.........

M23

Edit: Typnig!

Edited March 21, 2011 by Melba23

[startrek]

Thanks:)

i've actually been a member for a couple of years.. But i lost my account information in a harddrive crach along with all my scripts:( and can't seem to remember the email i used to register:S

ooh obviously

I didn't see that the variable for recursive was set to 0 in the (exclude .exe) example.. that was what got me confused:P

I remember modifying an UDF so it could search any folder.. But since I been playing golf and getting my lawn mower started to much lately;) and not enough coding i don't remember how:P

Looking forward to your updated UDF:)

[Melba23]

New Version - 31 Mar 2011

Changed - Filter behaviour when recursively searching for "files and folders" - addition of a further optional filter parameter.

This change has been implemented to retain compatibility with the existing _FileListToArray function while removing a limitation of that compatibility when returning both "files and folders".

Full explanation in first post.

Short version:

Searching non-recursively for "files and folders" with values in $sInclude_List (other than "*") or $sExclude_List (other than "") means that both files and folders that do not match the filters are ignored during the search.

Searching recursively for "files and folders" only applies the filters to the files - all folders are searched and returned. Folders can be excluded by using an additional filter parameter.

This change does not affect the behaviour of the UDF when returning only "files" or "folders" in either recursive or non-recursive searches.

New UDF, example and zip in first post.

M23

[guinness]

Brilliant Update It's personally not a biggy for me, but I can see some users asking "Why the Example fails in the .ZIP?" It's currently pointing to "RecFileListToArray_Mod.au3"... so I though I would let you know.

Edited March 31, 2011 by guinness

[Melba23]

guinness,

TVM - fixed.

Glad you like it - keep an eye out for the new version of ChooseFileFolder which is much faster with this new capability.

M23

[Spiff59]

New Version - 31 Mar 2011

Changed - Filter behaviour when recursively searching for "files and folders" - addition of a further optional filter parameter.

This change has been implemented to retain compatibility with the existing _FileListToArray function while removing a limitation of that compatibility when returning both "files and folders".

Full explanation in first post.

Short version:

Searching non-recursively for "files and folders" with values in $sInclude_List (other than "*") or $sExclude_List (other than "") means that both files and folders that do not match the filters are ignored during the search.

Searching recursively for "files and folders" only applies the filters to the files - all folders are searched and returned. Folders can be excluded by using an additional filter parameter.

This change does not affect the behaviour of the UDF when returning only "files" or "folders" in either recursive or non-recursive searches.

New UDF, example and zip in first post.

M23

We'd come to the same conclusion 2 years ago and implemented separate Exclude parms for folders and files:

[Melba23]

Spiff59,

Ah well, I got there in the end!

M23

[AdamUL]

@Melba23

I wanted to let you know that there is a problem with the update for this UDF. The 3-14 version returns an array without a problem, but the 3-31 version returns @error = 1 and @extended = 9.

Here is an example script showing the issue.

$sPathSearch = "C:"
$sFolderFilter = "of???arc"
$aArray = _RecFileListToArray($sPathSearch, $sFolderFilter, 2, 1, 0, 2)
    If @error Then
        ConsoleWrite("Directory Search Error: @error=" & @error & " @extended=" & @extended & @CRLF)
        Exit 1
    EndIf

I know that "of???arc" is a valid search string as used in FileFindFirstFile, and worked in the previous version of your UDF for the include list. For this example, the $aArray returned should have been 2 elements: [1, "C:\gwarch\ofl7carc"].

I have reverted back to using the previous version, 3-14, since it does not give me errors. I thought you would like to know about this issue.

Adam

[Melba23]

AdamUL,

Thank you for pointing out that problem.

When I changed the logic of the filters in the last update I missed the fact that it would have that effect on a recursive "folders only" search. I have now rejigged the filter logic to allow recursive searching for folders which should solve your problem - without, I hope, introducing any new ones!

The new version is in the first post.

M23

[nico18n]

@ Melba23

Very good. Just a useful function. I needed really. Good job.

[AdamUL]

Melba23,

I'm glad I could help. Thanks for updating the UDF so quickly. I wasn't exactly sure what change caused the problem. Hopefully, I don't run into any more problems. If I do, I will be sure to let you know.

Adam

[AZJIO]

necessary $sFilter = 'exe;msi;scr;dll;cpl;ax' or '*.exe;*.msi;*.scr;*.dll;*.cpl;*.ax'

Edited June 11, 2011 by AZJIO