limit files list in folder to array?

[tbaror]

Hello,

i am trying to write a function that will first check the amount of files exist in a folder and if its bigger for example 100 files will add only the first 100 files to array.

Since the function _FileListToArray i work with could get stack since sometimes i works with folders consist of thousand of files

Please advice

Thanks

[wakillon]

Something like this ?

#Include <File.au3>
#Include <Array.au3>
#Include <Math.au3>

$FileList = _FileListToArray ( @DesktopDir )
_ArrayDisplay ( $FileList, "$FileList" )
$FileList = _ArrayLimitSize ( $FileList, 100 )
_ArrayDisplay ( $FileList, "$FileList" )


Func _ArrayLimitSize ( $_OldArray, $_Size )
    Local $_NewArray[1], $_Start=0, $_Ubound=UBound ( $_OldArray ) -1
    If $_OldArray[0] = $_Ubound Then $_Start = 1
    For $_I = $_Start To _Min ( $_Size, $_Ubound )
        _ArrayAdd ( $_NewArray, $_OldArray[$_I] )
    Next
    $_NewArray[0] = UBound ( $_NewArray ) -1
    Return $_NewArray
EndFunc ;==> _ArrayLimitSize ( )

Edited May 14, 2011 by wakillon

[Tvern]

I did a quick adjustment to _FileListToArray to accept a maximum return count parameter:

Usage is identical to the original, but with one extra paramenter.

expandcollapse popup

; #FUNCTION# ====================================================================================================================
; Name...........: _FileListToArrayEx
; Description ...: Lists files and\or folders in a specified path (Similar to using Dir with the /B Switch)
; Syntax.........: _FileListToArrayEx($sPath[, $sFilter = "*"[, $iLimit = 0[,$iFlag = 0]]])
; Parameters ....: $sPath   - Path to generate filelist for.
;               $sFilter - Optional the filter to use, default is *. Search the Autoit3 helpfile for the word "WildCards" For details.
;               $iLimit - Maximum number of files to return. Default is 0, for unlimited.
;               $iFlag   - Optional: specifies whether to return files folders or both
;               |$iFlag=0(Default) Return both files and folders
;               |$iFlag=1 Return files only
;               |$iFlag=2 Return Folders only
; Return values .: @Error - 1 = Path not found or invalid
;               |2 = Invalid $sFilter
;               |3 = Invalid $iFlag
;               |4 = No File(s) Found
; Author ........: SolidSnake <MetalGX91 at GMail dot com>
; Modified.......: Tvern (added maximum return count)
; Remarks .......: The array returned is one-dimensional and is made up as follows:
;                               $array[0] = Number of Files\Folders returned
;                               $array[1] = 1st File\Folder
;                               $array[2] = 2nd File\Folder
;                               $array[3] = 3rd File\Folder
;                               $array[n] = nth File\Folder
; Related .......:
; Link ..........:
; Example .......: No
; Note ..........: Special Thanks to Helge and Layer for help with the $iFlag update speed optimization by code65536, pdaughe
; ===============================================================================================================================
Func _FileListToArrayEx($sPath, $sFilter = "*", $iLimit = 0, $iFlag = 0)
    Local $hSearch, $sFile, $sFileList, $sDelim = "|", $iCount = 0
    $sPath = StringRegExpReplace($sPath, "[\\/]+\z", "") & "\" ; ensure single trailing backslash
    If Not FileExists($sPath) Then Return SetError(1, 1, "")
    If StringRegExp($sFilter, "[\\/:><\|]|(?s)\A\s*\z") Then Return SetError(2, 2, "")
    If Not ($iFlag = 0 Or $iFlag = 1 Or $iFlag = 2) Then Return SetError(3, 3, "")
    $hSearch = FileFindFirstFile($sPath & $sFilter)
    If @error Then Return SetError(4, 4, "")
    While 1
        $sFile = FileFindNextFile($hSearch)
        If @error Then ExitLoop
        If ($iFlag + @extended = 2) Then ContinueLoop
        $iCount += 1
        $sFileList &= $sDelim & $sFile
        If $iLimit And $iLimit = $iCount Then ExitLoop
    WEnd
    FileClose($hSearch)
    If Not $sFileList Then Return SetError(4, 4, "")
    Return StringSplit(StringTrimLeft($sFileList, 1), "|")
EndFunc

[jchd]

Since the function _FileListToArray i work with could get stack since sometimes i works with folders consist of thousand of files

I don't see that happening anytime soon when run on typical stock PC (lest it's a 128Mb 386SX).

[Tvern]

I don't see that happening anytime soon when run on typical stock PC (lest it's a 128Mb 386SX).

I was thinking the same thing. Even with a system like you suggested you'd need a pretty large amount of files to cause significant speed/stability issues.

Still, I suppose the OP has his reasons to ask what he did and it was easy enough to make a quick adjustment for the function.

[tbaror]

I was thinking the same thing. Even with a system like you suggested you'd need a pretty large amount of files to cause significant speed/stability issues.

Still, I suppose the OP has his reasons to ask what he did and it was easy enough to make a quick adjustment for the function.

Hi All and thanks for the function,

The systems i worked with is usually from new Pc to remote storage's shares (Netapp usually) sometimes getting up to 70,000~30,000 files per for folder on Giga network based and those storage's are heavily used for storing HD,DV recording,playback.

Thanks

[jchd]

@tbaror

... and 70000 files of max NTFS filename (255 chars) are around 17Mchars, which we will round up (no TM here!) to a whopping 18 * 2 = 36Mb of array storage (worst case).

Knowing that average filenames hopefully tend to be much, much shorter than 255 characters, do you still really think it's such a problem for real world PC, that it would cause memory problems?

Just to clarify, I'd have to agree if you stated that you wanted to minimize time to collect that many filenames from a NAS, but not RAM storage of them.

[tbaror]

@tbaror

... and 70000 files of max NTFS filename (255 chars) are around 17Mchars, which we will round up (no TM here!) to a whopping 18 * 2 = 36Mb of array storage (worst case).

Knowing that average filenames hopefully tend to be much, much shorter than 255 characters, do you still really think it's such a problem for real world PC, that it would cause memory problems?

Just to clarify, I'd have to agree if you stated that you wanted to minimize time to collect that many filenames from a NAS, but not RAM storage of them.

Exactly "minimize time to collect that many file names from a NAS" which seems that the process is stack i wanted to emphasis that by writing its "getting stack".

as for the 70,000 limit i am aware of that there is a script that when files getting to certain amount level its getting divided into sub-folders.

in fact maybe you can help me with related performance script that i am currently writing i need opinion about certain test i am doing which is using the function GetFileSize and measuring the amount of time its took getting the response from remote storage , if threshold lets say 40ms is reached then alert is triggers.

My question would you by chance possibly knows how reliable this function in terms of pure network query spent time or its depends much on local machine cycle time?.

BTW the part of my initial function help asking where to get certain list of files and use it as rotation query so wont be used by storage cash.

Thanks

Edited May 14, 2011 by tbaror

[BrewManNH]

@jchd

I believe the problem isn't the stack, I believe he meant it gets stuck because it's taking so long. I may be mistaken but that's how I read it.

[jchd]

@BrewManNH

Right

@tbaror

TimerInit and TimerDiff will do pretty useful timing, independant of the hosting machine. In practice such query time may be vastly variable depending on NAS load and that is hard to foresee in enterprise environments.

Maybe your should monitor query response time and if you observe repeatitively slow response, decrease number of files limit accordingly.

[tbaror]

@BrewManNH

Right

@tbaror

TimerInit and TimerDiff will do pretty useful timing, independant of the hosting machine. In practice such query time may be vastly variable depending on NAS load and that is hard to foresee in enterprise environments.

Maybe your should monitor query response time and if you observe repeatitively slow response, decrease number of files limit accordingly.

Thanks for the info ,noted

[jchd]

Another, possibly simpler way: allow yourself a decent fixed amount of time to perform the task, say 100 files @ 40ms = 4s. Give yourself a safety margin and round this up to 5s.

Launch an AdLibRegister function with 5s timeout, pointing to a function which will raise a STOP_IT flag that your querying loop can poll at every new query, exiting the loop as soon as the timeout is done.

[tbaror]

Thanks again great idea i wasn't aware about AdLibRegister surly will use it.

I have more question but now for work.