Improvement of included _FileListToArray function.

Spiff59 · July 4, 2009

Interesting...

CODE

$Repeat = 1000000

;---------------------------
$t1 = TimerInit() 
For $x = 1 To $Repeat
    Test1()
Next
$t1 = TimerDiff($t1)

;---------------------------
$t2 = TimerInit() 
For $x = 1 To $Repeat
    Test2()
Next
$t2 = TimerDiff($t2)

;---------------------------
MsgBox(1, "", "t1 = " & Int($t1/10)/100 & @CRLF & "t2 = " & Int($t2/10)/100)

;---------------------------
Func Test1()
    Return SetError(4,4,"")
EndFunc

;---------------------------
Func Test2()
    Return SetError(4)
EndFunc

Setting up the @extended mechanism takes some clock ticks. It is not documented in the current version that @extended is being set, and we're not passing a meaningful error message via the structure. I think we ought to drop returning empty @extended data from all error returns. jpm had said back on page 8 (post 113) there was no reason for such processing. It's not going to make much difference on a non-recursive call, but during a recursive call with filters/excludes and a lot of files/folders it can be executed quite a number of times. It's a negligable speed increase, but it is an increase, so why not. Opinions?

Edit: By the way, I've been pounding on this thing (testing), and can't seem to break it. It's looking pretty solid.

Edited July 4, 2009 by Spiff59

BaKaMu · July 4, 2009

Interesting...
Setting up the @extended mechanism takes some clock ticks. It is not documented in the current version that @extended is being set, and we're not passing a meaningful error message via the structure. I think we ought to drop returning empty @extended data from all error returns. jpm had said back on page 8 (post 113) there was no reason for such processing. It's not going to make much difference on a non-recursive call, but during a recursive call with filters/excludes and a lot of files/folders it can be executed quite a number of times. It's a negligable speed increase, but it is an increase, so why not. Opinions?

Yes, really interesting.

I have done the error setting like --> SetError(1, 1, "") <-- for 100% backward compatibility to _FileListToArray.

ex: _FileListToArray : If Not FileExists($sPath) Then Return SetError(1, 1, "")

If this is not so important (in this special case), i am on board.

Recommendation: we only change the SetError(4, 4, "") in the loop to SetError(4).

So we have 100% backward compatibility and more speed.

Has to been tested now ;-)

Edit1: The test results are curious:

Your test-routine shows a noticeable speed increase:

t1 = 2.96

t2 = 1.63

In FLTA: SetError(4, 4, "") set to SetError(4) there is a speed decrease (nothing changed else):

Test1: one path with 10584 files, 1560 folders, filter="*", return files/folders, full path, recursion, no exclusion, repeats 30

_FileListToArrayNT7: Sec = 15.89 , 12144 items

_FileListToArrayNT8: Sec = 16.04 , 12144 items

Test2: one path with 10584 files, 1560 folders, filter="*", return files/folders, full path, recursion, with one (folder) exclusion, repeats 30

_FileListToArrayNT7: Sec = 14.48 , 9719 items

_FileListToArrayNT8: Sec = 14.55 , 9719 items

I am just wondering whether this belongs to my machine only.

Please test yourself

Edit2: BTW: SetError(4...) is not called very often (even in recursion), cause error is only set if search result on folder is empty (= empty folder)

Edit3: so i think we should stay at _FileListToArrayNT7, my tests doesn't show any instability until now.

Edited July 4, 2009 by BaKaMu

Spiff59 · July 4, 2009

You'd have to do a call something like searching your entire hard drive, with a filter like ""*.ini". That would cause most folders searched to execute the SetError statement. In my tests, the difference is so small that sometimes (due to background activity on my PC) the 4,4,"" returns come out a little faster, but most of the time, the simpler 4 call is slightly faster.

I don't think there's any issue with compatibility. If someone is using the undocumented @extended flag to test for an error, instead of @error, then they are taking advanatge of an unsupported loophole in the existing FLTA. One that is not supposed to be there according to all documentation.

Again, this is no big deal, any increase would be very miniscule. I was just looking for one more i to dot, or t to cross.

In testing, I still can't break it.

Edit: I bit the bullet and took a year of life off my hard drives. I searched "C:\" for "*.ini" with recursion. I made one initial call outside the timer loop to let disk caching do what it could (I have 3 80gb SATA drives striped as my C: drive and 4gb of ram), then I went into the timer loop 100 times for each version of SetError. I got 151.72 for 4,4,"", and 151.39 for just 4. It takes tons of iterations for the difference, proven in the demo program from post 181, to manifest.

If we keep the SetError as it is, are we going to document that @extended is returned?

Edit2: 152.12 and 150.95 on my second run. I think this is more about are we going to recreate undocumented behavior of the current FLTA. The actions of this version should be fully documented, so returning the basic @error would match the exisitng FLTA documentation, if we return both @error and @extended, we'd need to document that error conditions return two flags, @error and @extended, and that both contain the same value (which doesn't make much sense).

Edited July 4, 2009 by Spiff59

BaKaMu · July 4, 2009

In my tests, the difference is so small that sometimes (due to background activity on my PC) the 4,4,"" returns come out a little faster, but most of the time, the simpler 4 call is slightly faster.

There is a point i can not understand:

On my harddrive i have a testing structure of 10584 files, 1560 folders (different filenames/foldernames).

This structure is related to the structure i have described in my first post in this thread.

For testing i make always two tests (and this not only once)

Test1: one path with 10584 files, 1560 folders, filter="*", return files/folders, full path, recursion, no exclusion, repeats 30

Test2: one path with 10584 files, 1560 folders, filter="*", return files/folders, full path, recursion, with one (folder) exclusion, repeats 30

so the test conditions are always the same (i even deactivated most of the background activities)

The only difference between _FileListToArrayNT7 and _FileListToArrayNT8 is SetError(4, 4, "") and SetError(4)

Your test routine shows constantly a faster speed for SetError(4) and

my test results shows constantly a slower speed for _FileListToArrayNT8 with SetError(4)

(i would understand if it's faster or even if it's equal, but not that it is slower, independent of the size of hard drives)

That makes me crazy.

What is the reason for this behavior? I want to understand the mechanism of this logic.

It's against all the rules of programming, i have ever learned.

Does it belong to my CPU or what else? I am perplexed.

If we keep the SetError as it is, are we going to document that @extended is returned?

Hm, in the original _FileListToArray it isn't documented, so we also shouldn't do that.

So this are goodies for insiders.

-----------------------

I have done a lot of tests now, none of them has been incorrect.

So maybe we have done the work now (just in time) :-) :-)

Edit1: I was answering before i see your edits (especially edit2). I need some time for translation. ;-)

Your arguments are acceptable as well.

Edited July 4, 2009 by BaKaMu

Spiff59 · July 4, 2009

Maybe my testing is messed up...

For me, the "b" version below is faster...

CODE

;#include<array.au3>

;Testing parameters
Dim $SearchPath = "C:\Program Files\Autoit"
Dim $Pattern = ""
Dim $SrchType = 0
Dim $PathType = 0
Dim $Recursiv = 1
Dim $Exclude = ""
Dim $Format = 1; 0-based, 1-based

$Repeat = 100000

$t1 = TimerInit(); with exclude test
For $x = 1 To $Repeat
    $aRet = _FileListToArrayNT7($SearchPath, $Pattern, $SrchType, $PathType, $Recursiv, $Exclude, $Format)
Next
$t1 = TimerDiff($t1)
;_ArrayDisplay($aRet, '')

$t2 = TimerInit(); without exclude test
For $x = 1 To $Repeat
    $aRet = _FileListToArrayNT7b($SearchPath, $Pattern, $SrchType, $PathType, $Recursiv, $Exclude, $Format)
Next
$t2 = TimerDiff($t2)
;_ArrayDisplay($aRet, '')
MsgBox(1, "", "t1 = " & Int($t1/10)/100 & @CRLF & "t2 = " & Int($t2/10)/100)



Func _FileListToArrayNT7($sPath = @ScriptDir, $sFilter = "*", $iSearchType = 0, $iPathType = 0, $bRecursive = False, $sExclude = "", $iRetFormat = 1, $sWorkPath = "(-:>firstcall<:-)")
  Local $hSearch, $iPCount, $iFCount, $sFile, $sFileList, $sTWorkPath

  If $sWorkPath == "(-:>firstcall<:-)" Then
    If $sPath = -1 Or $sPath = Default Then $sPath = @ScriptDir
;strip leading/trailing spaces and semi-colons, all adjacent semi-colons, and spaces surrounding semi-colons
    $sPath = StringRegExpReplace(StringRegExpReplace($sPath, "(\s*;\s*)+", ";"), "\A;|;\z", "")
;check that at least one path is set
    If $sPath = "" Then Return SetError(1, 1, "")
;-----
    If $sFilter = -1 Or $sFilter = Default Then $sFilter = "*"
;strip leading/trailing spaces and semi-colons, all adjacent semi-colons, and spaces surrounding semi-colons
    $sFilter = StringRegExpReplace(StringRegExpReplace($sFilter, "(\s*;\s*)+", ";"), "\A;|;\z", "")
;check that at least one filter is set
    If $sFilter = "" Then Return SetError(2, 2, "")
;-----
    If $iSearchType <> "1" and $iSearchType <> "2" Then $iSearchType = "0"
;-----
    If $iPathType <> "1" And $iPathType <> "2" Then $iPathType = "0"
;-----
    $bRecursive = ($bRecursive = "1")
;-----
    If $sExclude = -1 Or $sExclude = Default Then $sExclude = ""
    If $sExclude Then
;prepare $sExclude
;strip leading/trailing spaces and semi-colons, all adjacent semi-colons, and spaces surrounding semi-colons
      $sExclude = StringRegExpReplace(StringRegExpReplace($sExclude, "(\s*;\s*)+", ";"), "\A;|;\z", "")
;convert $sExclude to fit StringRegExp (not perfect but useable)
      $sExclude = StringRegExpReplace($sExclude, '([\Q\.+[^]$(){}=!\E])', '\\$1')
      $sExclude = StringReplace($sExclude, "?", ".")
      $sExclude = StringReplace($sExclude, "*", ".*?")
      $sExclude = "(?i)\A" & StringReplace($sExclude, ";", "|") & "\z"
    EndIf
;-----
    If $iRetFormat <> "0" And $iRetFormat <> "2" Then $iRetFormat = "1"

    $sWorkPath = ""
  EndIf

  Local $aPath = StringSplit($sPath, ';', 1);paths array
  Local $aFilter = StringSplit($sFilter, ';', 1);filters array

  If $sExclude Then

    For $iPCount = 1 To $aPath[0];Path loop
      Local $sDelim = "|";reset $sDelim

      If StringRight($aPath[$iPCount], 1) <> "\" Then $aPath[$iPCount] &= "\";ensure trailing slash
      If $iPathType = 2 Then $sDelim &= $aPath[$iPCount];return full-path

      For $iFCount = 1 To $aFilter[0];filter loop
        If StringRegExp($aFilter[$iFCount], "[\\/:<>|]") Then ContinueLoop;bypass filters with invalid chars
        $hSearch = FileFindFirstFile($aPath[$iPCount] & $aFilter[$iFCount])
        If @error Then ContinueLoop
        Switch $iSearchType
          Case 1;files Only
            While True
              $sFile = FileFindNextFile($hSearch)
              If @error Then ExitLoop
              If @extended Then ContinueLoop;bypass folder
;check for exclude files
              If StringRegExp($sFile, $sExclude) Then ContinueLoop
              $sFileList &= $sDelim & $sWorkPath & $sFile
            WEnd
          Case 2;folders Only
            While True
              $sFile = FileFindNextFile($hSearch)
              If @error Then ExitLoop
              If @extended Then;bypass file
        ;check for exclude folder
                If StringRegExp($sFile, $sExclude) Then ContinueLoop
                $sFileList &= $sDelim & $sWorkPath & $sFile
              EndIf
            WEnd
          Case Else;files and folders
            While True
              $sFile = FileFindNextFile($hSearch)
              If @error Then ExitLoop
;check for exclude files/folder
              If StringRegExp($sFile, $sExclude) Then ContinueLoop
              $sFileList &= $sDelim & $sWorkPath & $sFile
            WEnd
        EndSwitch
        FileClose($hSearch)
      Next;$iFCount - next filter

;---------------

;optional do a recursive search
      If $bRecursive Then
        $hSearch = FileFindFirstFile($aPath[$iPCount] & "*.*")
        If Not @error Then
          While True
            $sFile = FileFindNextFile($hSearch)
            If @error Then ExitLoop
            If @extended Then;bypass file
;check for exclude folder
              If StringRegExp($sFile, $sExclude) Then ContinueLoop
;call recursive search
              If $iPathType = 1 Then $sTWorkPath = $sWorkPath & $sFile & "\"
              $sFileList &= _FileListToArrayNT7($aPath[$iPCount] & $sFile & "\", $sFilter, $iSearchType, $iPathType, $bRecursive, $sExclude, 2, $sTWorkPath)
            EndIf
          WEnd
          FileClose($hSearch)
        EndIf
      EndIf

    Next;$iPCount - next path

  Else;If Not $sExclude

    For $iPCount = 1 To $aPath[0];path loop
      Local $sDelim = "|";reset $sDelim

      If StringRight($aPath[$iPCount], 1) <> "\" Then $aPath[$iPCount] &= "\";ensure trailing slash
      If $iPathType = 2 Then $sDelim &= $aPath[$iPCount];return full-path

      For $iFCount = 1 To $aFilter[0];filter loop
        If StringRegExp($aFilter[$iFCount], "[\\/:<>|]") Then ContinueLoop;bypass filters with invalid chars
        $hSearch = FileFindFirstFile($aPath[$iPCount] & $aFilter[$iFCount])
        If @error Then ContinueLoop
        Switch $iSearchType
          Case 1;files Only
            While True
              $sFile = FileFindNextFile($hSearch)
              If @error Then ExitLoop
              If @extended Then ContinueLoop;bypass folder
              $sFileList &= $sDelim & $sWorkPath & $sFile
            WEnd
          Case 2;folders Only
            While True
              $sFile = FileFindNextFile($hSearch)
              If @error Then ExitLoop
              If @extended Then;bypass file
                $sFileList &= $sDelim & $sWorkPath & $sFile
              EndIf
            WEnd
          Case Else;files and folders
            While True
              $sFile = FileFindNextFile($hSearch)
              If @error Then ExitLoop
              $sFileList &= $sDelim & $sWorkPath & $sFile
            WEnd
        EndSwitch
        FileClose($hSearch)
      Next;$iFCount - next filter

;---------------

;optional do a recursive search
      If $bRecursive Then
        $hSearch = FileFindFirstFile($aPath[$iPCount] & "*.*")
        If Not @error Then
          While True
            $sFile = FileFindNextFile($hSearch)
            If @error Then ExitLoop
            If @extended Then;bypass file
;call recursive search
              If $iPathType = 1 Then $sTWorkPath = $sWorkPath & $sFile & "\"
              $sFileList &= _FileListToArrayNT7($aPath[$iPCount] & $sFile & "\", $sFilter, $iSearchType, $iPathType, $bRecursive, $sExclude, 2, $sTWorkPath)
            EndIf
          WEnd
          FileClose($hSearch)
        EndIf
      EndIf

    Next;$iPCount - next path

  EndIf;If $sExclude

;---------------

;set according return value
  If $sFileList Then
    Switch $iRetFormat
      Case 2;return a delimited string
        Return $sFileList
      Case 0;return a 0-based array
        Return StringSplit(StringTrimLeft($sFileList, 1), "|", 2)
      Case Else;return a 1-based array
        Return StringSplit(StringTrimLeft($sFileList, 1), "|", 1)
    EndSwitch
  Else
    Return SetError(4, 4, "")
  EndIf

EndFunc;==>_FileListToArrayNT7

;===============================================================================
Func _FileListToArrayNT7b($sPath = @ScriptDir, $sFilter = "*", $iSearchType = 0, $iPathType = 0, $bRecursive = False, $sExclude = "", $iRetFormat = 1, $sWorkPath = "(-:>firstcall<:-)")
  Local $hSearch, $iPCount, $iFCount, $sFile, $sFileList, $sTWorkPath

  If $sWorkPath == "(-:>firstcall<:-)" Then
    If $sPath = -1 Or $sPath = Default Then $sPath = @ScriptDir
;strip leading/trailing spaces and semi-colons, all adjacent semi-colons, and spaces surrounding semi-colons
    $sPath = StringRegExpReplace(StringRegExpReplace($sPath, "(\s*;\s*)+", ";"), "\A;|;\z", "")
;check that at least one path is set
    If $sPath = "" Then Return SetError(1)
;-----
    If $sFilter = -1 Or $sFilter = Default Then $sFilter = "*"
;strip leading/trailing spaces and semi-colons, all adjacent semi-colons, and spaces surrounding semi-colons
    $sFilter = StringRegExpReplace(StringRegExpReplace($sFilter, "(\s*;\s*)+", ";"), "\A;|;\z", "")
;check that at least one filter is set
    If $sFilter = "" Then Return SetError(2)
;-----
    If $iSearchType <> "1" and $iSearchType <> "2" Then $iSearchType = "0"
;-----
    If $iPathType <> "1" And $iPathType <> "2" Then $iPathType = "0"
;-----
    $bRecursive = ($bRecursive = "1")
;-----
    If $sExclude = -1 Or $sExclude = Default Then $sExclude = ""
    If $sExclude Then
;prepare $sExclude
;strip leading/trailing spaces and semi-colons, all adjacent semi-colons, and spaces surrounding semi-colons
      $sExclude = StringRegExpReplace(StringRegExpReplace($sExclude, "(\s*;\s*)+", ";"), "\A;|;\z", "")
;convert $sExclude to fit StringRegExp (not perfect but useable)
      $sExclude = StringRegExpReplace($sExclude, '([\Q\.+[^]$(){}=!\E])', '\\$1')
      $sExclude = StringReplace($sExclude, "?", ".")
      $sExclude = StringReplace($sExclude, "*", ".*?")
      $sExclude = "(?i)\A" & StringReplace($sExclude, ";", "|") & "\z"
    EndIf
;-----
    If $iRetFormat <> "0" And $iRetFormat <> "2" Then $iRetFormat = "1"

    $sWorkPath = ""
  EndIf

  Local $aPath = StringSplit($sPath, ';', 1);paths array
  Local $aFilter = StringSplit($sFilter, ';', 1);filters array

  If $sExclude Then

    For $iPCount = 1 To $aPath[0];Path loop
      Local $sDelim = "|";reset $sDelim

      If StringRight($aPath[$iPCount], 1) <> "\" Then $aPath[$iPCount] &= "\";ensure trailing slash
      If $iPathType = 2 Then $sDelim &= $aPath[$iPCount];return full-path

      For $iFCount = 1 To $aFilter[0];filter loop
        If StringRegExp($aFilter[$iFCount], "[\\/:<>|]") Then ContinueLoop;bypass filters with invalid chars
        $hSearch = FileFindFirstFile($aPath[$iPCount] & $aFilter[$iFCount])
        If @error Then ContinueLoop
        Switch $iSearchType
          Case 1;files Only
            While True
              $sFile = FileFindNextFile($hSearch)
              If @error Then ExitLoop
              If @extended Then ContinueLoop;bypass folder
;check for exclude files
              If StringRegExp($sFile, $sExclude) Then ContinueLoop
              $sFileList &= $sDelim & $sWorkPath & $sFile
            WEnd
          Case 2;folders Only
            While True
              $sFile = FileFindNextFile($hSearch)
              If @error Then ExitLoop
              If @extended Then;bypass file
        ;check for exclude folder
                If StringRegExp($sFile, $sExclude) Then ContinueLoop
                $sFileList &= $sDelim & $sWorkPath & $sFile
              EndIf
            WEnd
          Case Else;files and folders
            While True
              $sFile = FileFindNextFile($hSearch)
              If @error Then ExitLoop
;check for exclude files/folder
              If StringRegExp($sFile, $sExclude) Then ContinueLoop
              $sFileList &= $sDelim & $sWorkPath & $sFile
            WEnd
        EndSwitch
        FileClose($hSearch)
      Next;$iFCount - next filter

;---------------

;optional do a recursive search
      If $bRecursive Then
        $hSearch = FileFindFirstFile($aPath[$iPCount] & "*.*")
        If Not @error Then
          While True
            $sFile = FileFindNextFile($hSearch)
            If @error Then ExitLoop
            If @extended Then;bypass file
;check for exclude folder
              If StringRegExp($sFile, $sExclude) Then ContinueLoop
;call recursive search
              If $iPathType = 1 Then $sTWorkPath = $sWorkPath & $sFile & "\"
              $sFileList &= _FileListToArrayNT7b($aPath[$iPCount] & $sFile & "\", $sFilter, $iSearchType, $iPathType, $bRecursive, $sExclude, 2, $sTWorkPath)
            EndIf
          WEnd
          FileClose($hSearch)
        EndIf
      EndIf

    Next;$iPCount - next path

  Else;If Not $sExclude

    For $iPCount = 1 To $aPath[0];path loop
      Local $sDelim = "|";reset $sDelim

      If StringRight($aPath[$iPCount], 1) <> "\" Then $aPath[$iPCount] &= "\";ensure trailing slash
      If $iPathType = 2 Then $sDelim &= $aPath[$iPCount];return full-path

      For $iFCount = 1 To $aFilter[0];filter loop
        If StringRegExp($aFilter[$iFCount], "[\\/:<>|]") Then ContinueLoop;bypass filters with invalid chars
        $hSearch = FileFindFirstFile($aPath[$iPCount] & $aFilter[$iFCount])
        If @error Then ContinueLoop
        Switch $iSearchType
          Case 1;files Only
            While True
              $sFile = FileFindNextFile($hSearch)
              If @error Then ExitLoop
              If @extended Then ContinueLoop;bypass folder
              $sFileList &= $sDelim & $sWorkPath & $sFile
            WEnd
          Case 2;folders Only
            While True
              $sFile = FileFindNextFile($hSearch)
              If @error Then ExitLoop
              If @extended Then;bypass file
                $sFileList &= $sDelim & $sWorkPath & $sFile
              EndIf
            WEnd
          Case Else;files and folders
            While True
              $sFile = FileFindNextFile($hSearch)
              If @error Then ExitLoop
              $sFileList &= $sDelim & $sWorkPath & $sFile
            WEnd
        EndSwitch
        FileClose($hSearch)
      Next;$iFCount - next filter

;---------------

;optional do a recursive search
      If $bRecursive Then
        $hSearch = FileFindFirstFile($aPath[$iPCount] & "*.*")
        If Not @error Then
          While True
            $sFile = FileFindNextFile($hSearch)
            If @error Then ExitLoop
            If @extended Then;bypass file
;call recursive search
              If $iPathType = 1 Then $sTWorkPath = $sWorkPath & $sFile & "\"
              $sFileList &= _FileListToArrayNT7b($aPath[$iPCount] & $sFile & "\", $sFilter, $iSearchType, $iPathType, $bRecursive, $sExclude, 2, $sTWorkPath)
            EndIf
          WEnd
          FileClose($hSearch)
        EndIf
      EndIf

    Next;$iPCount - next path

  EndIf;If $sExclude

;---------------

;set according return value
  If $sFileList Then
    Switch $iRetFormat
      Case 2;return a delimited string
        Return $sFileList
      Case 0;return a 0-based array
        Return StringSplit(StringTrimLeft($sFileList, 1), "|", 2)
      Case Else;return a 1-based array
        Return StringSplit(StringTrimLeft($sFileList, 1), "|", 1)
    EndSwitch
  Else
    Return SetError(4)
  EndIf

EndFunc;==>_FileListToArrayNT7

I don't think leaving hidden undocumented behavior in this routine is valid just because the old version was that way.

Actually, the production version only returns the @error = 4 "not found" error if the FileFindFirstFile call gets no results. If FileFindFirstFile() does get a return, and all those results are dropped because of file/folder restrictions, then @error = 4 is not returned. This is undocumented, flaky behavior of the production routine that we are not attempting to recreate. Why recreate other useless undocumented behavior?

Edit: I actually used these parms for my long tests:

Dim $SearchPath = "C:\"
Dim $Pattern = "*.ini"
Dim $SrchType = 0
Dim $PathType = 0
Dim $Recursiv = 1
Dim $Exclude = ""
Dim $Format = 1; 0-based, 1-based

$Repeat = 100

If the (purposeless) behavior is to remain then these lines:

;                  If no path is found, @error is set (to 1, for backwards compatibility)
;                  If no filter is found, @error is set (to 2, for backwards compatibility)
;                  If no data is found, @error is set (to 4, for backwards compatibility)

ought to be:

;                  If no path is found, @error and @extended are set to 1
;                  If no filter is found, @error and @extended are set to 2
;                  If no data is found, @error and @extended are set to 4

and similar changes made to the helpfile entry

Edited July 4, 2009 by Spiff59

BaKaMu · July 4, 2009

Maybe my testing is messed up...
$Repeat = 100

If have now increased the number of my iterations from 30 to 300.

And now, there is a little constantly speed improvement, contrary to 30 iterations.

This is now the next question:

Does SetError need a wake up call or a warm up phase. ;-)

(I believe. that's are the mysteries of AutoIt, so i don't ask anymore)

If the (purposeless) behavior is to remain then these lines:

;                  If no path is found, @error is set (to 1, for backwards compatibility)
;                  If no filter is found, @error is set (to 2, for backwards compatibility)
;                  If no data is found, @error is set (to 4, for backwards compatibility)

ought to be:

;                  If no path is found, @error and @extended are set to 1
;                  If no filter is found, @error and @extended are set to 2
;                  If no data is found, @error and @extended are set to 4

and similar changes made to the helpfile entry

OK, sounds good, taken over.

Edited post #175

KaFu · July 4, 2009

First 5 stars added... and if it's for all the effort gone into this one ...

Spiff59 · July 4, 2009

And now, there is a little constantly speed improvement, contrary to 30 iterations.
This is now the next question:
Does SetError need a wake up call or a warm up phase. ;-)

You got me there, there is apparently some overheard to defining the @extended structure and the text message that is passed with the @extended flag. I still think it would be cleaner disabling the @extended returns, but the documentation update is also an acceptable solution.

First 5 stars added...

I'm sure all who have contributed (you are on that list too!) appreciate your rating. I certainly do.

It's my hope that we have beaten the crap out of every nuance of this routine to the point that it will stand for a long time without generating all the forum topics and BugTracs that result from the existing FLTA function.

KaFu · July 4, 2009

It's my hope that we have beaten the crap out of every nuance of this routine to the point that it will stand for a long time without generating all the forum topics and BugTracs that result from the existing FLTA function.

True words! That's also why I appreciate all the work all of you have done in this thread... when I started AU in 2006 this function was the first one I really had a fight with, if I remembered correct it wasn't even a version in the official UDFs and I had to search the forum, just to stumble over this unicode character 'bug' with the DOS version. Nowadays I use my own version based on _WinAPI_FileFind() by ascendant (oh, worth a place in my signature btw

), which for sure I will x-check with the results of this thread! So, Kudos to all of you...

jpm · July 4, 2009

Just a comment setting @extended= @error in case only @extended is tested is not a good reason. @extended is aim to add extra info to @error.

Checking @error is the only way to know if an error occur .

For simple case the return value can be Ok . I assume that not the case of _FileListToArray()

You are really running good

BaKaMu · July 4, 2009

It's my hope that we have beaten the crap out of every nuance of this routine to the point that it will stand for a long time without generating all the forum topics and BugTracs that result from the existing FLTA function.

Five stars for this statement. ;-)

What we should do next.

I have seen a post from jpm, from his point of view setting @extended = @error in case only @extended is tested is not a good reason.

I agree with this statement also, especially it tends more in your direction, Really, i have no preference.

I was directed from the facts of my speed tests.

So i will edit _FileListToArrayNT7 (post #175) again and set all SetError(x, x, "") to SetError(x)

(and correct the header)

@jpm: should a final version be set in a new post now? And witch name should be set.

Function header will now be ok (in a few minutes), who updates the help documentation (please not i).

I'm glad that the work is done.

Many thanks to all for the impelling suggestions, especially to spiff59 (his hardheadedness and practically thoughts was a very good motor for this project)

So there is only a little tidy up remaining.

jpm · July 4, 2009

Five stars for this statement. ;-)
What we should do next.
I have seen a post from jpm, from his point of view setting @extended = @error in case only @extended is tested is not a good reason.
I agree with this statement also, especially it tends more in your direction, Really, i have no preference.
I was directed from the facts of my speed tests.
So i will edit _FileListToArrayNT7 (post #175) again and set all SetError(x, x, "") to SetError(x)
(and correct the header)
@jpm: should a final version be set in a new post now? And witch name should be set.
Function header will now be ok (in a few minutes), who updates the help documentation (please not i).
I'm glad that the work is done.
Many thanks to all for the impelling suggestions, especially to spiff59 (his hardheadedness and practically thoughts was a very good motor for this project)
So there is only a little tidy up remaining.

If everybody is OK i pickup post #175 and I will update the doc with my franglish ...

BaKaMu · July 4, 2009

If everybody is OK i pickup post #175 and I will update the doc with my franglish ...

Thank you very much for that (Merci beaucoup)

jpm · July 4, 2009

Done Congratulation to everybody

Valik · July 4, 2009

Not done. Not acceptable. A 27 line function was just turned into a 186 line function. I don't think so. I am not letting that can of worms get opened up.

And to anybody who thinks the function has had enough soak time to be stable... you're an idiot. It hasn't.

I'm reverting this change immediately. It will not be in 3.3.1.2.

BaKaMu · July 5, 2009

Not done. Not acceptable. A 27 line function was just turned into a 186 line function. I don't think so. I am not letting that can of worms get opened up.
And to anybody who thinks the function has had enough soak time to be stable... you're an idiot. It hasn't.
I'm reverting this change immediately. It will not be in 3.3.1.2.

You may be surprised, that i agree with you. (not with your diction)

But this doesn't care.

With the helpful assistance of the community we have now a good replacement for _FileListToArray

which is faster, more functional and returns the right results.

So what we want more?

Everyone can use it ore not, and it does not matter whether the function is in a general UDF or private UDF.

It is more relevant, that this function now exists and is working good.

Edited July 5, 2009 by BaKaMu

Valik · July 5, 2009

With the helpful assistance of the community we have now a good replacement for _FileListToArray

It is more relevant, that this function now exists and is working good.

These two statements cannot be made. It's very stupid to think that they can be. You need to quit programming the moment you think you've finally written a bug-free piece of code because that is the moment you've lost the plot.

Anyway, I have an idea on how to include this without violating the points I made previously on why it shouldn't be added.

Wus · July 5, 2009

I'm sorry, but I am failing to understand this whole dilemma. AutoIT has a _FileListToArray function that works fine. I've used it and have been quite happy with the results. If the built-in version is not fast enough for everyone then they can search the forum and find this version. Voila! Done. Everyone gets what they want.

On a side note, the need for speed is a dangerous one for programmers. Trying to optimize a function generally results in:

More bugs that were completely unforeseen
More overall complexity in the input and output arguments
Much more complexity in the code
An enormous amount of time wasted that could have been better used

If the function absolutely needs to be sped up then you just bite the bullet and accept the previous issues, but it should be avoided otherwise. In keeping with the KISS principle (Keep It Simple, Stupid) dependable, relatively slow, and concise is better than error-prone, blazing-fast, and complex any day.

**I'm sorry if this post has been a rehash of previous posts.. I only made it through the first couple pages in this thread before my eyes started bleeding.

EDIT:

Fixed formatting

Edited July 5, 2009 by Wus

GEOSoft · July 5, 2009

I have to agree with you Wus. It's wondeerfull that it has such great speed doing 100,00 files recursivly but I think the most I have ever had to return was around 250 and we would be talking a time less than the blink of an eye so all that added speed wouldn't show any noticable difference.

I'm interested in seeing what Valik comes up with.

BaKaMu · July 5, 2009

New approach after rollback ;-)

A 27 line function was just turned into a 186 line function.

What's wrong with that?

Normally more function and sometimes more speed = more code

(even AutoIt increases from version to version, and i don't see much more complexity in a 140 code line function)

If you have a better function (with same functionality, stability and speed) with less code, i will be the first to use this function. Really!

These two statements cannot be made. It's very stupid to think that they can be. You need to quit programming the moment you think you've finally written a bug-free piece of code because that is the moment you've lost the plot.

Don't agree (at least not in total), because in the logic of your statement, all the programming is open end forever.

(so i have understood your words, but may be i am wrong).

In practice there may be never a progamm that is 1000% bug-free and never fails in all imaginable and non imaginable circumstances.

(and if this programm exists, no one wants to use it) :-)

And in practice we normally finalize a programm if we are confident, that there is no more improvement.

(there may be some more improvements, but we are confident that there are no more, please notice the difference)

This is not a stupid approach, rather more practical.

On the other hand i know, there are a lot of projects in real world which have been terminated and all participants knows that those are junk,

but that's another question, we are discussing here about good programming.

Anyway i digress to philosophic questions.

It is my opinion that i am more on your side as you believe, nevertheless i'll think about your arguments further on.

Improvement of included _FileListToArray function.

Recommended Posts

Link to comment

Share on other sites

Top Posters In This Topic

Top Posters In This Topic

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members