_FileListToArray new proposal; comments?

[randallc]

hi,

Here is proposed new faster, optioned "_FileListToArray " in AutoIt to avoid DOS problem with file search (misses foreign characters in file and folder names; and using PID gives incorrect results depending on indexing settings in Windows?).

[Current UDF function slow, no recursion, not so many options etc as in SmoKe_N's DOS version eg]

Based on scripts from all above; trial and comments if possible to test? [actually faster than Dos when not recursed, and some functions even on recursed!]

The exclude filter can't use wildcards other than "?" at present; can someone suggest a fix for that RegExp? I'm still interested to fix that regEx, but it is working now using a simpler Exlude RegExp if needed durig the original FindFiles sub...

Thanks, Randall

_FileListToArray3($sPath, $sFilter = "*", $iFlag = 0, $iRecurse = 0, $iBaseDir = 1, $sExclude = "", $i_deleteduplicate = 1)

7a Features

_______1. tries to use AutoIt to list files to array natively, nearly as fast as DOS and without DOS drawbacks; (foreign/ extended characters and PID problem)

_______2. All options as in DOS command (?), including recurse, folders/ files/ both/ , exclude/ wildcards/ multiple piped matches /remove dupes or not (latter not in _________Win 9x)

7b Drawbacks

_______1. a little slower than DOS sometimes; usually minimal difference

_______2. not fully tested yet due to lack of interest!

7c Comparisons to other versions

_______1. Current standard UDF is slow in large directories, and does not have any of the other DOS options as in 7a2

_______2. Most other options do not allow all options, or are only fast on specific options

_______3. Many other options have the faults of using DOS (foreign/ extended characters and PID problem)

_______4. smokN's version is the only version with as many options; Speed comparison with smokN's version in DOS shows mine is better in most, except "files only" (I think; see demo file in the zip for comparison)

_______5. Best comparison is to run the comparison example file in the zip..

** 2j version in post #1 fixes the multiple piped wildcards, by fixing the dupes removal; also quicker for that.

** 2m version in post #1 fixes extra @cr in Folder recurse list, and array[0] accurate after multiple wildcards

** 1d version in post #1 fixes speed for "*" and "*.*", as it is twice as fast and is such a commonly used filter... (thanks WeaponX for pointing it out..)

** 1e fixes folders not returned if only 1 present, and @CR prob

- 12thFeb08 - fixed exclude bug, I hope..

[ ** WeaponX function is probably simpler and I think has all the functions and speed now of mine; if you are happy to modify the RegExp he uses in the func parameters; (option #1 in post #1, I think)

FileListToArray comprehensive comparison] (although with multiple file filters, I am not sure if his function returns dupes....)

FileListToArray3.zip

Edited February 21, 2008 by randallc

[SmOke_N]

hi,

Here is proposed new faster, optioned "_FileListToArray " in AutoIt to avoid DOS problem with file search (misses foreign characters in file and folder names; and using PID gives incorrect results depending on indexing settings in Windows?).

[Current UDF function slow, no recursion, not so many options etc as in SmoKe_N's DOS version eg]

Based on scripts from all above; trial and comments if possible to test? [actually faster than Dos when not recursed, and some functions even on recursed!]

The exclude filter can't use wildcards other than "?" at present; can someone suggest a fix for that RegExp?

Thanks, Randall

_FileListToArray3($sPath, $sFilter = "*", $iFlag = 0, $iRecurse = 0, $iBaseDir = 1, $sExclude = "")

Nice work... I'm not understanding the RegExp to remove dupes though.

[randallc]

Hi,

Sorry; its not for dupes (I have just used array sort, remove dupes for that..)

Its to filter by "Exclude" as in your function, last parameter; exclude certain files;

If $sExclude <> "" Then ; filter files if exclude needed -------------------
        $sFileString = StringReplace($sFileString, "*", "**");$cw = ConsoleWrite("$sExclude    =" & $sExclude & @CR)
        $sExclude = StringReplace(StringReplace(StringReplace($sExclude, ".", "\."), "*", ".*"), "?", ".");change the filters to RegExp filters
        Local $pattern = '(\*(?i)[^\*]*(' & $sExclude & ')[^\\\*]*)\*';'(^(?i).*' & $sExclude & '[^\\]*$)
        $sFileString = StringRegExpReplace("*" & $sFileString, $pattern, "")
        $sFileString = StringReplace($sFileString, "**", "*")
        If StringLeft($sFileString, 1) = "*" Then $sFileString = StringTrimLeft($sFileString, 1)
    EndIf;If UBound($asList) > 1 Then ConsoleWrite("$asList[1]     =" & @LF & $asList[1] & @LF)oÝ÷ Ù«­¢+Ø1½°ÀÌØíÁÑÑɸôÌäì ÀäÈì¨ ý¤¥mxÀäÈì©t¨ ÌäìµÀìÀÌØíÍá±ÕµÀìÌäì¥mxÀäÈìÀäÈìÀäÈì©t¨¤ÀäÈì¨ÌäììÌäì¡x ý¤¤¸¨ÌäìµÀìÀÌØíÍá±ÕµÀìÌäímxÀäÈìÀäÈít¨ÀÌØì¤ÌäììÀÌØíÜô
½¹Í½±]É¥Ñ ÅÕ½ÐìÀÌØíÁÑÑɸôÅÕ½ÐìµÀìÀÌØíÁÑÑɸµÀì1¤oÝ÷ جµ§u©ejëh×6'(\*(?i)[^\*]*oÝ÷ ÚËZ®Ü"¶ªºjºZ²¦¥­æÊØ^±êZ­«h®)ßW¶¸§ëzË¥¶Æ§Ê{¦mêè~W­µêì4ä쵪ìjëh×6(' & $sExclude & ')oÝ÷ Ùùݶ,{­Ø^{%¹×©j¶¦z׫z
®¢Û*«¨¶'ò¢çhmÁ©íj|W§jg¬Â+a²©bØ^©êÞ!ƧßÛn±æ¬µêâ²F¬Â)]qªÝ¢yrjëh×6[^\\\*]*)

;end of that query Edited July 14, 2007 by randallc

[MrCreatoR]

For excludes i think you can try this one:

Func IsExclude($CheckPath, $ExclsPath, $ExcludeList)
    If $ExcludeList = -1 Then Return False
    Local $ExcludesArr[2] = [1, $ExcludeList]
    If StringInStr($ExcludeList, "|") Then $ExcludesArr = StringSplit($ExcludeList, "|")
    
    For $i = 1 To UBound($ExcludeList)-1
        $FindExcludeFile = FileFindFirstFile($ExclsPath & "\" & $ExcludeList[$i])
        While Not @error
            $CurrentExclFile = FileFindNextFile($FindExcludeFile)
            If @error Then ExitLoop
            If $ExclsPath & "\" & $CurrentExclFile = $CheckPath Or _
                $CurrentExclFile = $CheckPath Or $ExclsPath & "\" & $CurrentExclFile = $CheckPath Or _
                $ExclsPath =  $CheckPath Then Return True
        WEnd
    Next
    Return False
EndFunc

This way user (scripter) can also set the exlude list separated by | char.

[randallc]

For excludes i think you can try this one:

Func IsExclude($CheckPath, $ExclsPath, $ExcludeList)
    If $ExcludeList = -1 Then Return False
    Local $ExcludesArr[2] = [1, $ExcludeList]
    If StringInStr($ExcludeList, "|") Then $ExcludesArr = StringSplit($ExcludeList, "|")
    
    For $i = 1 To UBound($ExcludeList)-1
        $FindExcludeFile = FileFindFirstFile($ExclsPath & "\" & $ExcludeList[$i])
        While Not @error
            $CurrentExclFile = FileFindNextFile($FindExcludeFile)
            If @error Then ExitLoop
            If $ExclsPath & "\" & $CurrentExclFile = $CheckPath Or _
                $CurrentExclFile = $CheckPath Or $ExclsPath & "\" & $CurrentExclFile = $CheckPath Or _
                $ExclsPath =  $CheckPath Then Return True
        WEnd
    Next
    Return False
EndFunc

Best, randall

[SmOke_N]

Randal, that code looks good ... sounds like we need a macro to determine whether the code is unicode compiled or not

[randallc]

Randal, that code looks good ... sounds like we need a macro to determine whether the code is unicode compiled or not

HI,

That code is the one I have already, unfortunately; it does not give correct answers if "*" used as wildcard; only if ? used....

[PS there is a new macro alrady; "@unicode", , examles in bug reports somewhere; nont well documented]

I can keep the speed up and use asterisks as wildcards if I incorporate it in the original search subs; will post later; will lengthen the code even more to use pipes in that as well, though, for multiple options..

Randall

[randallc]

OK,

Looks like that RegExp within the subs works OK with a pipe in it as well as asterisks, and not any slower than my previous method; still reasonable and does not affect the ordinary search speed;

Randall

_FileListToArrayNew2g.zip ( 5.45K ) Number of downloads: 341 See post #1

Edited October 17, 2007 by randallc

[smashly]

Hi and thank you randallc.

This was just what I was looking for, puts my old, slow and outdated fl2a functions I've been using to shame

I'd say your function would be about 10 to 15 times faster then what I was previously using and yours offer more features as well ! w00t

Damn fine work m8.

After playing with your _FileListToArrayNew2g function for a couple of days I've decided to include it in to my latest autoit project/hobby. (hope you don't mind)

Currently I'm using your function in a loop to use multiple file extension filters for my searches.

For each filter looped it adds more time to the search but it's still speeds up my searches by 10 fold or more in comparison to what I was using... All Good.

Once again Thank You

'N'

Cheers.

[randallc]

Hi and thank you randallc.

This was just what I was looking for, puts my old, slow and outdated fl2a functions I've been using to shame

I'd say your function would be about 10 to 15 times faster then what I was previously using and yours offer more features as well ! w00t

Damn fine work m8.

After playing with your _FileListToArrayNew2g function for a couple of days I've decided to include it in to my latest autoit project/hobby. (hope you don't mind)

Currently I'm using your function in a loop to use multiple file extension filters for my searches.

For each filter looped it adds more time to the search but it's still speeds up my searches by 10 fold or more in comparison to what I was using... All Good.

Once again Thank You

'N'

Cheers.

Hi,

Good news!

thanks for the feedback..

Best, randall

[Manko]

Hi! Great stuff in there! I even modified one of the funcs to make me a USB syncer... (Looks at LastModified...)

It's much easier learning autoit by looking at good sources and playing around with them, I think

Thanks!

/Manko

[randallc]

Hi! Great stuff in there! I even modified one of the funcs to make me a USB syncer... (Looks at LastModified...)

It's much easier learning autoit by looking at good sources and playing around with them, I think

Thanks!

/Manko

Hi,

Glad it helped! Maybe post yours too for interest...?

Best, Randall

[Manko]

Hi,

Glad it helped! Maybe post yours too for interest...?

Best, Randall

Yahh... I guess if I mention it, I should show it. Was hoping to post it in its own thread but I don't have the post-count for it.

It takes two dirs as argument and copies the newer files (filemodified) in both directions. It also copies dirs/files not existing in both over to the other dir....

Source lacks somewhat in errorchecking, but that can be added as needed by those that need it.

expandcollapse popup

#include <Array.au3>

Local $sFileString1, $sEmptyDirString1, $sFileString2, $sEmptyDirString2

$sPath1 = $CmdLine[1]                                                                       ; Argument 1 from "commandline"
_ListFilesAndEmptyDirs($sPath1, $sFileString1, $sEmptyDirString1)                           ; Get strings with empty dirs and files from path1
$arFiles1 = StringSplit(StringTrimRight($sFileString1, 1), "*")             ; First remove last "*" and then make an array of the fileliststring
$arDirs1 = StringSplit(StringTrimRight($sEmptyDirString1, 1), "*")          ; First remove last "*" and then make an array of the emptydirsstring
$sPath2 = $CmdLine[2]
_ListFilesAndEmptyDirs($sPath2, $sFileString2, $sEmptyDirString2)
$arFiles2 = StringSplit(StringTrimRight($sFileString2, 1), "*")             
$arDirs2 = StringSplit(StringTrimRight($sEmptyDirString2, 1), "*")

; MakeEmptyDirs in the other path corresponding to first an vice versa...
; No need to check if they already exist.
$x = 0                                                                                      ; x to count synced files
$i = 1
Do
    DirCreate(StringReplace($arDirs1[$i], $sPath1, $sPath2))                                ; Create empty dirs from spath1 in spath2
    $i += 1
Until $i = UBound($arDirs1)                                                                 ; Past last item... Do no more.
$i = 1
Do
    DirCreate(StringReplace($arDirs2[$i], $sPath2, $sPath1))                                ; Create empty dirs from spath2 in spath1
    $i += 1
Until $i = UBound($arDirs2)                                                                 ; Past last item... Do no more.

; _ArrayBinarySearch must have sorted arrays.
_ArraySort($arFiles2)

; If I touch filelist2 I have to sort it again!!
; ...better copy it for working.
$arRest = $arFiles2

; Copy unique or newer files from path1 to path2, and if existing in both, 
; and files in path2 is newer, copy those files from path2 to path1...
$i = 1
$y = 1
Do
    $y = _ArrayBinarySearch($arFiles2, StringReplace($arFiles1[$i], $sPath1, $sPath2), 1)   ; Do items exist in both lists? $y==position if in list2
    If @error Then                                                                          ;If it dont exist in filelist2(path2)
        FileCopy($arFiles1[$i], StringReplace($arFiles1[$i], $sPath1, $sPath2), 8) ; copy unique item in list1 to path2, making dirs as needed (", 8")
        $x += 1                                                                             ; synced a file only existing in path1 to path2
    Else
        If FileGetTime($arFiles1[$i], 0, 1) > FileGetTime($arFiles2[$y], 0, 1) Then         ; checks if file1 was more recently modified
            FileCopy($arFiles1[$i], $arFiles2[$y], 1)                                       ; then copy over .. ", 1" == OverWrite
            $x += 1                                                                         ; synced newer file from path1 to path2
        Else
            If FileGetTime($arFiles1[$i], 0, 1) < FileGetTime($arFiles2[$y], 0, 1) Then     ; checks if file2 was more recently modified
                FileCopy($arFiles2[$y], $arFiles1[$i], 1)                                   ; then copy over .. ", 1" == OverWrite
                $x += 1                                                                     ; synced newer file from path2 to path1
            EndIf
                                                                                            ; if they were modified exact same second; Do nothing!
        EndIf
        $arRest[$y] = ""                                                ; Delete items existing in both paths, to single out Uniques in filelist2
    EndIf
    $i += 1
Until $i = UBound($arFiles1)                                                                ; Past last item...

; copy remaining unique files from spath2 to spath1
$i = 1
Do
    If $arRest[$i] > "" Then                                                                ; skip empty items, already handled remember?
        FileCopy($arRest[$i], StringReplace($arRest[$i], $sPath2, $sPath1), 8)              ; Just copy remaining unique from path2 to path1
        $x += 1                                                                             ; synced 1; copied unique item in filelist2 to path1
    EndIf
    $i += 1
Until $i = UBound($arRest)                                                              ; Past last item...

MsgBox(0, "Syncer", "Synced " & $x & " files.", 2)                              ;  Yup we are done, look how well we did.... Hrm... Look 2 secs...

; Get files and empty dirs. Return them in two large strings, 
; each item separated by an asterix: * 
; split to arrays with StringSplit
; Adapted from a _filelisttoarray UDF... 
Func _ListFilesAndEmptyDirs($sPathF, ByRef $sFileStringF, ByRef $sEmptyDirStringF)          ;ByRef==work directly on variable we used in call
    Local $hSearch = FileFindFirstFile($sPathF & "\*"), $sPathF2                    ; Open search in dir, searching for all files and folders=="*"
    If @error = 1 Then                                                                  ; Dir was empty!
        FileClose($hSearch)                                                                 ; release search-handles
        Return (1)                                                                          ; return dir empty
    EndIf
    While 1                                                                                 ; do "search filesandfolders" forever
        $sFileF = FileFindNextFile($hSearch)                                                ; Find next file!
        If @error Then                                                                      ; No more files in this dir
            FileClose($hSearch)                                                             ; release search-handles
            ExitLoop                                                                        ; exit searching!
        EndIf
        $sPathF2 = $sPathF & "\" & $sFileF                                                  ; combine searchpath and found dirname/filename
        If StringInStr(FileGetAttrib($sPathF2), "D") Then                                   ; Check if it is a directory and...
            If _ListFilesAndEmptyDirs($sPathF2, $sFileStringF, $sEmptyDirStringF) Then      ; ..find files/dirs in there too! ...but If dir is empty
                $sEmptyDirStringF &= $sPathF2 & "*"     ;write the dirname to our string, separating items with "*" ; use StringSplit to make Array
            EndIf
        Else                                                                                ; Not a directory
            $sFileStringF &= $sPathF2 & "*"             ;write the filename to our string, separating items with "*" ; use StringSplit to make Array
        EndIf
    WEnd

    FileClose($hSearch)
EndFunc   ;==>_ListFilesAndEmptyDirs

EDIT: Made "codebox" "autoitbox"...

/Manko

Edited October 30, 2007 by Manko

[GEOSoft]

Hi! Great stuff in there! I even modified one of the funcs to make me a USB syncer... (Looks at LastModified...)

It's much easier learning autoit by looking at good sources and playing around with them, I think

Thanks!

/Manko

What method did you use to get the drive letter for the USB drive(s)?

[Manko]

What method did you use to get the drive letter for the USB drive(s)?

The Pstart.exe (Menuapp for USBs...) is started by windows when it finds my USB (i have a autorun.inf)...

...well to be honest, I have to click it in the dialogue that comes up for all USB-storage devices, when attached...

I use PStart for all my apps on the USB and it exports the drive letter.

I also run the Syncer from the USB so I could also use "@AutoItExe" to get "executed_app/script-path" and stringright.

/Manko

[GEOSoft]

The Pstart.exe (Menuapp for USBs...) is started by windows when it finds my USB (i have a autorun.inf)...

...well to be honest, I have to click it in the dialogue that comes up for all USB-storage devices, when attached...

I use PStart for all my apps on the USB and it exports the drive letter.

I also run the Syncer from the USB so I could also use "@AutoItExe" to get "executed_app/script-path" and stringright.

/Manko

I'll take a look at Pstart.exe. I use my USB on 5 different systems and it gets used on other peoples systems so I just wrote a quick function to get the USB drives letter. I have now found a way through drive enumeration to determine if a fixed drive is actually a USB since USBs show as fixed drives.

[Xenobiologist]

Hi,

maybe also a solution:

Global $serial = 821811646

If _getDriveBySerial($serial) Then
    MsgBox(64, 'USB', 'Laufwerk --> ' & _getDriveBySerial($serial), 5)
Else
    MsgBox(16, 'USB', 'Serial nicht gefunden', 5)
EndIf

Func _getDriveBySerial($serial)
    For $i = 65 To 90
        If $serial = DriveGetSerial(Chr($i) & ':') Then Return Chr($i) & ':' 
    Next
    Return 0
EndFunc   ;==>_getDriveBySerial

So long,

Mega

[GEOSoft]

Hi,

maybe also a solution:

Global $serial = 821811646

If _getDriveBySerial($serial) Then
    MsgBox(64, 'USB', 'Laufwerk --> ' & _getDriveBySerial($serial), 5)
Else
    MsgBox(16, 'USB', 'Serial nicht gefunden', 5)
EndIf

Func _getDriveBySerial($serial)
    For $i = 65 To 90
        If $serial = DriveGetSerial(Chr($i) & ':') Then Return Chr($i) & ':' 
    Next
    Return 0
EndFunc   ;==>_getDriveBySerial

So long,

Mega

I use it in a file synchronizer which I run right from the USB drive so I just use

Func _DriveGetPortable()
   Return StringLeft(@ScriptDir, StringInStr(@ScriptDir, '\'))
EndFunc

You can actually get a list of USB drives using WMI

I was working on a Drive UDF when I quit scripting.

Edited October 31, 2007 by GEOSoft

[randallc]

hi,

** 2j version in post #1 fixes the multiple piped wildcards, by fixing the dupes removal; also quicker for that.

Randall

[MadBoy]

I use it in a file synchronizer which I run right from the USB drive so I just use

Func _DriveGetPortable()
    Return StringLeft(@ScriptDir, StringInStr(@ScriptDir, '\'))
 EndFunc

You can actually get a list of USB drives using WMI

I was working on a Drive UDF when I quit scripting.

GEOSoft: take a look at http://www.autoitscript.com/forum/index.php?showtopic=57433. I wrote it based on some code from general forum and it seems to work just fine