_FileListToArray new proposal; comments?

[MadBoy]

hi,

** 2j version in post #1 fixes the multiple piped wildcards, by fixing the dupes removal; also quicker for that.

Randall

Thanks for this, gotta update to newest version

[GEOSoft]

GEOSoft: take a look at http://www.autoitscript.com/forum/index.php?showtopic=57433. I wrote it based on some code from general forum and it seems to work just fine

That works. The trick is to get all the "Fixed" drive types and then feed those drive letters to the function to determine if it s actually a USB drive. Virtual drives will also show as USB drives but IIRC they don't appear as "Fixed" drives. I could be wrong about that.

Win32_Drives will also return the data you need.

[shEiD]

I'm fairly new to AutoIt and coding overall.

Am I doing something wrong, because funtion returns dodgy array:

#include <Array.au3>
#include <_FileListToArrayNew2j.au3>
Dim $SourceDir
$SourceDir = "C:\test Source Dir"
$DirList = _FileListToArray3($SourceDir, "*", 2, 1)
_ArrayDisplay($DirList, "test")

#cs
While function description states, that returned array should be:
Note(s): The array returned is one-dimensional and is made up as follows:
$array[0] = Number of Files\Folders returned
$array[1] = 1st File\Folder
$array[2] = 2nd File\Folder
$array[3] = 3rd File\Folder
$array[n] = nth File\Folder
#ce

Returned array is:

I inserted image, as to show/ask - whats that at the ends of returned directories? And as you can see, $array[0] does not contain a number of directories found.

I can work around that, though I thought I'll share whats happening, maybe its a mistake in recent version? (I never tried older ones, so I'm not sure).

PS: Thanks a bunch for this function, its very handy with all its options.

[randallc]

I'm fairly new to AutoIt and coding overall.

Am I doing something wrong, because funtion returns dodgy array:

#include <Array.au3>
#include <_FileListToArrayNew2j.au3>
Dim $SourceDir
$SourceDir = "C:\test Source Dir"
$DirList = _FileListToArray3($SourceDir, "*", 2, 1)
_ArrayDisplay($DirList, "test")

#cs
While function description states, that returned array should be:
Note(s): The array returned is one-dimensional and is made up as follows:
$array[0] = Number of Files\Folders returned
$array[1] = 1st File\Folder
$array[2] = 2nd File\Folder
$array[3] = 3rd File\Folder
$array[n] = nth File\Folder
#ce

Returned array is:

I inserted image, as to show/ask - whats that at the ends of returned directories? And as you can see, $array[0] does not contain a number of directories found.

I can work around that, though I thought I'll share whats happening, maybe its a mistake in recent version? (I never tried older ones, so I'm not sure).

PS: Thanks a bunch for this function, its very handy with all its options.

Hi,

I think you have an old version of AutoIt? - the ArrayDisplay perhaps not working properly

current v is 3.2.8.1 or beta, 3.2.9.14

Best, Randall

[shEiD]

I have 3.2.8.1 version installed - just started using AutoIt 3 days ago

And ArrayDisplay works fine, as I used it with "normal" FileListToArray, provided in <File.au3>, which returned array as it was supposed to.

Then I found your function, which had recursion - eureka!

Should I use beta? I usually don't use betas.

[randallc]

I have 3.2.8.1 version installed - just started using AutoIt 3 days ago

And ArrayDisplay works fine, as I used it with "normal" FileListToArray, provided in <File.au3>, which returned array as it was supposed to.

Then I found your function, which had recursion - eureka!

Should I use beta? I usually don't use betas.

Hi, No 3.2.8.1 should be fine, and I don't get that extra whitespace displayed; i'll have to check my function, stringlength, chr(last char) etc to see what i'm picking up.

what windows version are you using?

Best, randall

[shEiD]

Hi, No 3.2.8.1 should be fine, and I don't get that extra whitespace displayed; i'll have to check my function, stringlength, chr(last char) etc to see what i'm picking up.

what windows version are you using?

Best, randall

I'm using WinXP Tablet PC Edition 2005 SP2.

[randallc]

I'm using WinXP Tablet PC Edition 2005 SP2.

OK,

Can you please run this and look at the console output (are you using Scite?); I get no discrepancy

; FileListToArraytestShort2j.au3
#include<_FileListToArrayNew2j.au3>; _FileListToArray3($sPath, $sFilter = "*", $iFlag = 0, $iRecurse = 0, $iBaseDir = 1, $sExclude = "")
local $ar_Array = _FileListToArray3 ("c:", "*", 2, 0, 0,"Windows|Backup")
;,$c= 
_ArrayDisplay($ar_Array, "Folder  List, excluding "&"Windows,Backup")
for $i=0 to UBound($ar_Array)-1
    ConsoleWrite("StringLen($ar_Array[$i])="&StringLen($ar_Array[$i])&@TAB&"$ar_Array[$i]="&$ar_Array[$i]&@LF)
Next

Best, Randall

[shEiD]

OK,

Can you please run this and look at the console output (are you using Scite?); I get no discrepancy

Best, Randall

Thanks for your answer, yes, i'm using Scite. I just made a couple of tests with the code you provided (thanks you again, this time for showing me how handy a console can be ) Ok, to the point, after some testing I figured out:

1. if I use recursive mode I get that 1 extra character at the end:

command used:

; _FileListToArray3($sPath, $sFilter = "*", $iFlag = 0, $iRecurse = 0, $iBaseDir = 1, $sExclude = "")
local $ar_Array = _FileListToArray3 ("C:\test Source Dir", "*", 2, 1, 0)

console readout (StringLen shows 1 character too much):

StringLen($ar_Array[$i])=1  $ar_Array[$i]=6
StringLen($ar_Array[$i])=10 $ar_Array[$i]=test Dir1
StringLen($ar_Array[$i])=10 $ar_Array[$i]=test Dir2
StringLen($ar_Array[$i])=22 $ar_Array[$i]=test Dir2\test subDir
StringLen($ar_Array[$i])=10 $ar_Array[$i]=test Dir3
StringLen($ar_Array[$i])=23 $ar_Array[$i]=test Dir3\test subDir1
StringLen($ar_Array[$i])=23 $ar_Array[$i]=test Dir3\test subDir2

2. if I dont use BaseDir or use it as 1, I dont get a number of directories found in ar_Array[0]:

commands used:

; _FileListToArray3($sPath, $sFilter = "*", $iFlag = 0, $iRecurse = 0, $iBaseDir = 1, $sExclude = "")
local $ar_Array = _FileListToArray3 ("C:\test Source Dir", "*", 2, 1, 1)

or:

local $ar_Array = _FileListToArray3 ("C:\test Source Dir", "*", 2, 1)


console readout:
StringLen($ar_Array[$i])=29 $ar_Array[$i]=C:\test Source Dir\test Dir1
StringLen($ar_Array[$i])=29 $ar_Array[$i]=C:\test Source Dir\test Dir2
StringLen($ar_Array[$i])=41 $ar_Array[$i]=C:\test Source Dir\test Dir2\test subDir
StringLen($ar_Array[$i])=29 $ar_Array[$i]=C:\test Source Dir\test Dir3
StringLen($ar_Array[$i])=42 $ar_Array[$i]=C:\test Source Dir\test Dir3\test subDir1
StringLen($ar_Array[$i])=42 $ar_Array[$i]=C:\test Source Dir\test Dir3\test subDir2

Sadly, I would need it to work like that: recursion with base directory and number of found directories in array[0]

[randallc]

OK,

1. Fixed in 2k attached.

2. Fix manually after; "_ArrayInsert( $avArray,0,ubound($avArray))" until i fix it.

Best, randall

If $iRecurse And ($iFlag = 2) And $sFilter <> "*"  And $sFilter <> "*.*" and not StringInStr($sFilter,"**") Then ; filte
;...............
ElseIf $iRecurse And ($iFlag = 2) Then
$sFileString = StringStripCR($sFileString1)

Thanks for your help with this..

Best, Randall

Edited November 25, 2007 by randallc

[shEiD]

Thank You. Awesome function and support

[randallc]

Thanks,

"** 2m version in post #1 fixes extra @cr in Folder recurse list, and array[0] accurate after multiple wildcards"

Best, randall

[weaponx]

I just wanted to share some thoughts after testing this out. I want to warn you that this may sound harsh, but from a professional standpoint I think a function inteded for mass distribution needs to be very polished.

1. Its fast, a few seconds faster than recursing the FileSystemObject with 2000+ files

2. The code is not commented well (less helpful if you are hoping for integration into autoit)

3. You have array.au3 nested in the udf (Bad coding practice, unless it's documented where it's needed)

4. It's not very clear which version i'm supposed to download (i.e. If the latest version has bugfixes from a prior version, why leave the old version posted?)

5. The zip contains 8 files so i'm making the assumption that since I downloaded _FileListToArrayNew2m.zip, the UDF is _FileListToArrayNew2m.au3, but what are all these other files? I see there are some usage examples in there like FileListToArraytestShortm.au3

6. There are 7 different functions in _FileListToArrayNew2m.au3 with almost identical descriptions (lists all files and folders in a specified path), why? I prefer to see very concise functions, and Sm0ke_N's version contains only one function, although also not commented well: http://www.autoitscript.com/forum/index.ph...t=0&start=0

7. The description post (first post) really needs to highlight why this is different from other versions of this function, using on ordered list of features / drawbacks and maybe comparisons to other versions

Edited November 26, 2007 by weaponx

[randallc]

I just wanted to share some thoughts after testing this out. I want to warn you that this may sound harsh, but from a professional standpoint I think a function inteded for mass distribution needs to be very polished.

1. Its fast, a few seconds faster than recursing the FileSystemObject with 2000+ files

OK -

it doesn't feel harsh at all; mostly, any attention is gratefully accepted after all this time...

I guess, when I started, I thought someone more a programmer might take on the task as it seemed so basic (when you look at the functionality of DOS); naive again! -who wants to work?..

2. The code is not commented well (less helpful if you are hoping for integration into autoit)

I''l try more comments, thanks

3. You have array.au3 nested in the udf (Bad coding practice, unless it's documented where it's needed)

True; _ArrayToString() and _ArrayInsert(); should I make these inside the UDF, and remove reliance on #include, or just document it and leave the #include?

4. It's not very clear which version i'm supposed to download (i.e. If the latest version has bugfixes from a prior version, why leave the old version posted?)

OK, removed old one (although I am following Jon's practice with the beta; I am scared people will want to quickly revert to the last one if I make a new bug!)

5. The zip contains 8 files so i'm making the assumption that since I downloaded _FileListToArrayNew2m.zip, the UDF is _FileListToArrayNew2m.au3, but what are all these other files? I see there are some usage examples in there like FileListToArraytestShortm.au3

True, all are examples of usage, plus the corrected _ArrayDisplay..

6. There are 7 different functions in _FileListToArrayNew2m.au3 with almost identical descriptions (lists all files and folders in a specified path), why? I prefer to see very concise functions, and Sm0ke_N's version contains only one function, although also not commented well: http://www.autoitscript.com/forum/index.ph...t=0&start=0

I thought the descriptions were OK; I'll have to check them again..

** I think this is the only way to make a readable set of (often) recursive functions which optimise speed depending on the nature of the call;; ie folders/ recursive or whatever.. if I put the "if" commands inside the loops, it all slows down...**

You're right of course, I'll have to fix it.. [correct here, now, i think..] - although they are only for use locally, and never called outside the UDF, so no-one needs to know?

_FileListToArrayFolders; lists all folders in a specified path, +/- recursive

_FileListToArrayFiles; ;fast, with files only, not recursed

_FileListToArrayRecAll; lists all files and folders in a specified path, recursive

_FileListToArrayRecFiles; lists all files in a specified path, recursive

_FileListToArrayBrief2; fastest, with files and folders, not recursed, with $baseDir

_FileListToArrayBrief1; not $iRecurse and And $iBaseDir ;fast, with files and folders, not recursed

7. The description post (first post) really needs to highlight why this is different from other versions of this function, using on ordered list of features / drawbacks and maybe comparisons to other versions

7a Features

_______1. tries to use AutoIt to list files to array natively, nearly as fast as DOS and without DOS drawbacks; (foreign/ extended characters and PID problem)

_______2. All options as in DOS command (?), including recurse, folders/ files/ both/ , exclude/ wildcards/ multiple piped matches /remove dupes or not (latter not in _________Win 9x)

7b Drawbacks

_______1. a little slower than DOS sometimes; usually minimal difference

_______2. not fully tested yet due to lack of interest!

7c Comparisons to other versions

_______1. Current standard UDF is slow in large directories, and does not have any of the other DOS options as in 7a2

_______2. Most other options do not allow all options, or are only fast on specific options

_______3. Many other options have the faults of using DOS (foreign/ extended characters and PID problem)

_______4. smokN's version is the only version with as many options; Speed comparison with smokN's version in DOS shows mine is better in most, except "files only" (I think; see demo file in the zip for comparison)

_______5. Best comparison is to run the comparison example file in the zip..

Any other suggestions?; Maybe I should put this answer in first post, as you say?

Best, Randall

Edited November 26, 2007 by randallc

[MadBoy]

_______2. not fully tested yet due to lack of interest!

Trust me, there is interest. I would love to see it in AutoIt builtin. The function is amazing and it' superior to using dos commands. Mostly because it will be useful for more systems then English. I use your function and slowly i am trying to implement this into my program which uses my own listing function based on dos.

[randallc]

Trust me, there is interest. I would love to see it in AutoIt builtin. The function is amazing and it' superior to using dos commands. Mostly because it will be useful for more systems then English. I use your function and slowly i am trying to implement this into my program which uses my own listing function based on dos.

OK,

Thanks; I use it a lot myself too, and I like it!

Best, Randall

[weaponx]

Here is what I like to see. I didn't change any of the code except this line:

If $iRecurse Then

...

EndIf

The conditional check you had to determine if the exclude string is blank is unnecessary.

expandcollapse popup

Func _FileListToArray3($sPath, $sFilter = "*", $iFlag = 0, $iRecurse = 0, $iBaseDir = 1, $sExclude = "", $i_deleteduplicate = 1); ** still need to use RegExp to eliminate dupes? ***
    ;Declare local variables
    Local $hSearch, $sFile, $sFileString, $asList[1], $sep = "|", $sFileString1, $sFilter1 = $sFilter
    
    ;Set default filter to wildcard
    If $sFilter = -1 Then $sFilter = "*"
    
    ;Strip trailing slash from search path
    If StringRight($sPath, 1) == "\" Then $sPath = StringTrimRight($sPath, 1)
    
    ;Ensure search path exists
    If Not FileExists($sPath) Then Return SetError(1, 1, "")
    
    ;Return error if special characters are found in filter
    If (StringInStr($sFilter, "\")) Or (StringInStr($sFilter, "/")) Or (StringInStr($sFilter, ":")) Or (StringInStr($sFilter, ">")) Or (StringInStr($sFilter, "<")) Or (StringStripWS($sFilter, 8) = "") Then Return SetError(2, 2, "")
    
    ;Only allow 0,1,2 for flag options
    If Not ($iFlag = 0 Or $iFlag = 1 Or $iFlag = 2) Then Return SetError(3, 3, "");~     $sFilter = StringReplace("*" & $sFilter & "*", "**", "*")
    
    ;Append pipe to file filter if no semi-colons and pipe symbols are found
    If Not StringInStr($sFilter, ';') And Not StringInStr($sFilter, '|') Then $sFilter &= "|"
    
    ;Determine seperator character
    If StringInStr($sFilter, ';') Then $sep = ";"    ;$sFilter &= ';'
    If StringInStr($sFilter, ',') Then $sep = ","    ;$sFilter &= ';'
    
    ;Declare local variables, Implode file filter
    Local $aFilterSplit = StringSplit(StringStripWS($sFilter, 8), $sep), $sRead, $sHoldSplit, $arFolders[2] = [ $sPath, ""];~     $cw = ConsoleWrite("UBound($aFilterSplit) =" & UBound($aFilterSplit) & @LF)
    
    If $sExclude <> "" Then $sExclude = "(?i)" & StringReplace(StringReplace(StringReplace($sExclude, ".", "\."), "*", ".*"), "?", ".");change the filters to RegExp filters
    
    ;If recursion is desired, build an array of all sub-folders in search path (eliminates the need to run a conditional statement against FileAttrib)
    If $iRecurse Then;$cw = ConsoleWrite("UBound($aFilterSplit) =" & UBound($aFilterSplit) & @LF)
        
        ;Build string ($sFileString1) of foldernames within search path, recursion and exclusion options are passed from main function
        _FileListToArrayFolders($sPath, $sFileString1, "*", $iRecurse, $sExclude)
        
        ;Implode folder string
        Local $arFolders = StringSplit(StringTrimRight($sFileString1, 1), "*")
        
        ;Store search path in first element
        $arFolders[0] = $sPath
    EndIf
    
    ;Loop through folder array
    For $iCF = 0 To UBound($arFolders) - 1;    $cw = ConsoleWrite("$iCF=" & $iCF & " $arFolders[$iCF]    =" & @LF & $arFolders[$iCF] & @LF)
        
        ;Verify folder name isn't just whitespace
        If StringStripWS($arFolders[$iCF], 8) = '' Then ContinueLoop
        
        ;Loop through file filters
        For $iCC = 1 To UBound($aFilterSplit) - 1
            
            ;Verify file filter isn't just whitespace
            If StringStripWS($aFilterSplit[$iCC], 8) = '' Then ContinueLoop
            
            ;Append asterisk to file filter if a period is leading
            If StringLeft($aFilterSplit[$iCC], 1) == "." Then $aFilterSplit[$iCC] = "*" & $aFilterSplit[$iCC] ;, "**", "*")
            
            ;Replace multiple asterisks in file filter
            $sFilter = StringReplace("*" & $sFilter & "*", "**", "*")
            Select; options for not recursing; quicker than filtering after for single directory
                
                ;Below needs work, _FileListToArrayBrief1 and _FileListToArrayBrief2 should be consolidated with an option passed for the files / folders flag
                
                ;Fastest, with files and folders, not recursed
                Case Not $iRecurse And Not $iFlag And Not $iBaseDir
                    _FileListToArrayBrief2($arFolders[$iCF], $sFileString, $aFilterSplit[$iCC], $sExclude)
                    
                    ;Not $iRecurse and  And $iBaseDir ;fast, with files and folders, not recursed
                Case Not $iFlag
                    _FileListToArrayBrief1($arFolders[$iCF], $sFileString, $aFilterSplit[$iCC], $sExclude)
                    
                    ;Fast, with files only,  not recursed
                Case $iFlag = 1
                    _FileListToArrayFiles($arFolders[$iCF], $sFileString, $aFilterSplit[$iCC], $sExclude)
                    
                    ;Folders only , not recursed
                Case Not $iRecurse And $iFlag = 2
                    _FileListToArrayFolders($arFolders[$iCF], $sFileString, $aFilterSplit[$iCC], $iRecurse, $sExclude)
            EndSelect;$cw = ConsoleWrite("$iCC=" & $iCC & " $sFileString    =" & @LF & $sFileString & @LF)
            
            ;Append pipe symbol and current file filter onto $sHoldSplit ???????
            If $iCF = 0 Then $sHoldSplit &= '|' & $aFilterSplit[$iCC]; $cw = ConsoleWrite("$iCC=" & $iCC & " $sFileString    =" & @LF & $sFileString & @LF)
        Next
        
        ;Replace multiple asterisks
        If $iCF = 0 Then $sFilter = StringReplace(StringTrimLeft($sHoldSplit, 1), "**", "*");,$cw = ConsoleWrite("$iCC=" & $iCC & " $sFilter    =" & @LF & $sFilter & @LF)
    Next
    
    ;Below needs work....
    
    ;If recursive, folders-only, and filter ins't a wildcard
    If $iRecurse And ($iFlag = 2) And $sFilter <> "*" And $sFilter <> "*.*" And Not StringInStr($sFilter, "**") Then ; filter folders -------------------
        
        ;Trim trailing character
        $sFileString1 = StringTrimRight(StringReplace($sFileString1, "*", @CRLF), 1)
        
        ;Change the filters to RegExp filters
        $sFilter1 = StringReplace(StringReplace(StringReplace($sFilter1, ".", "\."), "*", ".*"), "?", ".")
        Local $pattern = '(?m)(^(?i)' & $sFilter1 & '$)'    ;, $cw = ConsoleWrite("$sFilter    =" & @LF & $sFilter1 & @LF), $cw = ConsoleWrite("$pattern    =" & @LF & $pattern & @LF)
        Local $asList = StringRegExp($sFileString1, $pattern, 3)
        
        ;If only relative file / folder names are desired
        If (Not $iBaseDir) Then
            
            ;ARRAY.AU3 DEPENDENCY
            $sFileString1 = _ArrayToString($asList, "*")
            $sFileString1 = StringReplace($sFileString1, $sPath & "\", "", 0, 2)
            $asList = StringSplit($sFileString1, "*")
        EndIf
    ElseIf $iRecurse And ($iFlag = 2) Then
        $sFileString = StringStripCR($sFileString1)
    EndIf;If UBound($asList) > 1 Then ConsoleWrite("$asList[1]     =" & @LF & $asList[1] & @LF);~
    
    ;ARRAY.AU3 DEPENDENCY
    If IsArray($asList) And UBound($asList) > 1 And $asList[0] <> "" And Not IsNumber($asList[0]) Then _ArrayInsert($asList, 0, UBound($asList))
    If IsArray($asList) And UBound($asList) > 1 And $asList[0] <> "" Then Return $asList
    If (Not $iBaseDir) Or (Not $iRecurse And Not $iFlag And Not $iBaseDir) Then $sFileString = StringReplace($sFileString, $sPath & "\", "", 0, 2)
    $arReturn = StringSplit(StringTrimRight($sFileString, 1), "*");~     local $a=ConsoleWrite("$sFileString :"&@lf&StringReplace($sFileString,"|",@crlf)&@lf),$timerstamp1=TimerInit()
    If $i_deleteduplicate And IsArray($arReturn) And UBound($arReturn) > 1 And $arReturn[1] <> "" And Not (UBound($aFilterSplit) = 3 And $aFilterSplit[2] == "") Then _ArrayDeleteDupes($arReturn);and  $arFolders[1]<>""
    Return $arReturn;~     Return StringSplit(StringTrimRight($sFileString, 1), "*")
EndFunc   ;==>_FileListToArray3

Edited November 26, 2007 by weaponx

[randallc]

Hi,

Thanks foer all that effort!

I will try to complete it that way and incorporate it.

I am sorry it is all so unclear.

The change you have suggested about the exclude;

The conditional check you had to determine if the exclude string is blank is unnecessary.

(the check is actually on the $flag parameter, to make the exclude blank temporarily while making the folder list before the loop of each directory)

would exclude finding files in foldernames depending on the exclude when it might only be intended to "exclude" in the filename search..

So I think that part should not be changed.

Best, Randall

[Zedna]

@randallc:

If you are going to submit your _FileListToArrayEx() function as standard UDF then please:

- don't implement unneccessary complicated "remove duplicates" functionality, look here

- don't use any _Array functions (if possible)

- don't use more helper functions, just one (if possible)

In summary: Make it as much simple as can be.

I prefer these parametres:

Func _FileListToArrayEx($sPath, $sFilter = '*.*', $iFlag = 0, $iRecurse = False, $iBaseDir = 1, $sExclude = '')

Edited November 26, 2007 by Zedna

[randallc]

@randallc:

If you are going to submit your _FileListToArrayEx() function as standard UDF then please:

Probably I won't ; I willl try to stimulate a better programmer to do it.

- don't implement unneccessary complicated "remove duplicates" functionality, look here

It only removes dupes from the list, not the directories! (?)

So far, there is no such UDF, and it seems to be impossible now to get new UDFs accepted...

I hate to send a "wrong" list of files (ie with dupes, too!; but happy to leave it that wat for Win 9x)

- don't use any _Array functions (if possible)

OK, I'll look at doing my own func, but another internal helper!

- don't use more helper functions, just one (if possible)

I don't think I'd know how, and harder to maintain?.

In summary, keep it simple

I think it is already; recursive funcs are hard to maintain, I think, if just a list of "case funcs", and I don't think I'd know how to do it.

I prefer these parametres:

Func _FileListToArrayEx($sPath, $sFilter = '*.*', $iFlag = 0, $iRecurse = False, $iBaseDir = 1, $sExclude = '')

OK, I'll look at that.

Best, randall

Edited December 6, 2007 by randallc