tbaror Posted May 14, 2011 Posted May 14, 2011 Hello,i am trying to write a function that will first check the amount of files exist in a folder and if its bigger for example 100 files will add only the first 100 files to array.Since the function _FileListToArray i work with could get stack since sometimes i works with folders consist of thousand of filesPlease adviceThanks
wakillon Posted May 14, 2011 Posted May 14, 2011 (edited) Something like this ? #Include <File.au3> #Include <Array.au3> #Include <Math.au3> $FileList = _FileListToArray ( @DesktopDir ) _ArrayDisplay ( $FileList, "$FileList" ) $FileList = _ArrayLimitSize ( $FileList, 100 ) _ArrayDisplay ( $FileList, "$FileList" ) Func _ArrayLimitSize ( $_OldArray, $_Size ) Local $_NewArray[1], $_Start=0, $_Ubound=UBound ( $_OldArray ) -1 If $_OldArray[0] = $_Ubound Then $_Start = 1 For $_I = $_Start To _Min ( $_Size, $_Ubound ) _ArrayAdd ( $_NewArray, $_OldArray[$_I] ) Next $_NewArray[0] = UBound ( $_NewArray ) -1 Return $_NewArray EndFunc ;==> _ArrayLimitSize ( ) Edited May 14, 2011 by wakillon AutoIt 3.3.14.2 X86 - SciTE 3.6.0 - WIN 8.1 X64 - Other Example Scripts
Tvern Posted May 14, 2011 Posted May 14, 2011 I did a quick adjustment to _FileListToArray to accept a maximum return count parameter: Usage is identical to the original, but with one extra paramenter. expandcollapse popup; #FUNCTION# ==================================================================================================================== ; Name...........: _FileListToArrayEx ; Description ...: Lists files and\or folders in a specified path (Similar to using Dir with the /B Switch) ; Syntax.........: _FileListToArrayEx($sPath[, $sFilter = "*"[, $iLimit = 0[,$iFlag = 0]]]) ; Parameters ....: $sPath - Path to generate filelist for. ; $sFilter - Optional the filter to use, default is *. Search the Autoit3 helpfile for the word "WildCards" For details. ; $iLimit - Maximum number of files to return. Default is 0, for unlimited. ; $iFlag - Optional: specifies whether to return files folders or both ; |$iFlag=0(Default) Return both files and folders ; |$iFlag=1 Return files only ; |$iFlag=2 Return Folders only ; Return values .: @Error - 1 = Path not found or invalid ; |2 = Invalid $sFilter ; |3 = Invalid $iFlag ; |4 = No File(s) Found ; Author ........: SolidSnake <MetalGX91 at GMail dot com> ; Modified.......: Tvern (added maximum return count) ; Remarks .......: The array returned is one-dimensional and is made up as follows: ; $array[0] = Number of Files\Folders returned ; $array[1] = 1st File\Folder ; $array[2] = 2nd File\Folder ; $array[3] = 3rd File\Folder ; $array[n] = nth File\Folder ; Related .......: ; Link ..........: ; Example .......: No ; Note ..........: Special Thanks to Helge and Layer for help with the $iFlag update speed optimization by code65536, pdaughe ; =============================================================================================================================== Func _FileListToArrayEx($sPath, $sFilter = "*", $iLimit = 0, $iFlag = 0) Local $hSearch, $sFile, $sFileList, $sDelim = "|", $iCount = 0 $sPath = StringRegExpReplace($sPath, "[\\/]+\z", "") & "\" ; ensure single trailing backslash If Not FileExists($sPath) Then Return SetError(1, 1, "") If StringRegExp($sFilter, "[\\/:><\|]|(?s)\A\s*\z") Then Return SetError(2, 2, "") If Not ($iFlag = 0 Or $iFlag = 1 Or $iFlag = 2) Then Return SetError(3, 3, "") $hSearch = FileFindFirstFile($sPath & $sFilter) If @error Then Return SetError(4, 4, "") While 1 $sFile = FileFindNextFile($hSearch) If @error Then ExitLoop If ($iFlag + @extended = 2) Then ContinueLoop $iCount += 1 $sFileList &= $sDelim & $sFile If $iLimit And $iLimit = $iCount Then ExitLoop WEnd FileClose($hSearch) If Not $sFileList Then Return SetError(4, 4, "") Return StringSplit(StringTrimLeft($sFileList, 1), "|") EndFunc
jchd Posted May 14, 2011 Posted May 14, 2011 Since the function _FileListToArray i work with could get stack since sometimes i works with folders consist of thousand of filesI don't see that happening anytime soon when run on typical stock PC (lest it's a 128Mb 386SX). This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe hereRegExp tutorial: enough to get startedPCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta. SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)
Tvern Posted May 14, 2011 Posted May 14, 2011 I don't see that happening anytime soon when run on typical stock PC (lest it's a 128Mb 386SX).I was thinking the same thing. Even with a system like you suggested you'd need a pretty large amount of files to cause significant speed/stability issues.Still, I suppose the OP has his reasons to ask what he did and it was easy enough to make a quick adjustment for the function.
tbaror Posted May 14, 2011 Author Posted May 14, 2011 I was thinking the same thing. Even with a system like you suggested you'd need a pretty large amount of files to cause significant speed/stability issues.Still, I suppose the OP has his reasons to ask what he did and it was easy enough to make a quick adjustment for the function.Hi All and thanks for the function,The systems i worked with is usually from new Pc to remote storage's shares (Netapp usually) sometimes getting up to 70,000~30,000 files per for folder on Giga network based and those storage's are heavily used for storing HD,DV recording,playback.Thanks
jchd Posted May 14, 2011 Posted May 14, 2011 @tbaror ... and 70000 files of max NTFS filename (255 chars) are around 17Mchars, which we will round up (no TM here!) to a whopping 18 * 2 = 36Mb of array storage (worst case). Knowing that average filenames hopefully tend to be much, much shorter than 255 characters, do you still really think it's such a problem for real world PC, that it would cause memory problems? Just to clarify, I'd have to agree if you stated that you wanted to minimize time to collect that many filenames from a NAS, but not RAM storage of them. This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe hereRegExp tutorial: enough to get startedPCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta. SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)
tbaror Posted May 14, 2011 Author Posted May 14, 2011 (edited) @tbaror... and 70000 files of max NTFS filename (255 chars) are around 17Mchars, which we will round up (no TM here!) to a whopping 18 * 2 = 36Mb of array storage (worst case).Knowing that average filenames hopefully tend to be much, much shorter than 255 characters, do you still really think it's such a problem for real world PC, that it would cause memory problems?Just to clarify, I'd have to agree if you stated that you wanted to minimize time to collect that many filenames from a NAS, but not RAM storage of them.Exactly "minimize time to collect that many file names from a NAS" which seems that the process is stack i wanted to emphasis that by writing its "getting stack".as for the 70,000 limit i am aware of that there is a script that when files getting to certain amount level its getting divided into sub-folders. in fact maybe you can help me with related performance script that i am currently writing i need opinion about certain test i am doing which is using the function GetFileSize and measuring the amount of time its took getting the response from remote storage , if threshold lets say 40ms is reached then alert is triggers.My question would you by chance possibly knows how reliable this function in terms of pure network query spent time or its depends much on local machine cycle time?.BTW the part of my initial function help asking where to get certain list of files and use it as rotation query so wont be used by storage cash. Thanks Edited May 14, 2011 by tbaror
BrewManNH Posted May 14, 2011 Posted May 14, 2011 @jchd I believe the problem isn't the stack, I believe he meant it gets stuck because it's taking so long. I may be mistaken but that's how I read it. If I posted any code, assume that code was written using the latest release version unless stated otherwise. Also, if it doesn't work on XP I can't help with that because I don't have access to XP, and I'm not going to.Give a programmer the correct code and he can do his work for a day. Teach a programmer to debug and he can do his work for a lifetime - by Chirag GudeHow to ask questions the smart way! I hereby grant any person the right to use any code I post, that I am the original author of, on the autoitscript.com forums, unless I've specifically stated otherwise in the code or the thread post. If you do use my code all I ask, as a courtesy, is to make note of where you got it from. Back up and restore Windows user files _Array.au3 - Modified array functions that include support for 2D arrays. - ColorChooser - An add-on for SciTE that pops up a color dialog so you can select and paste a color code into a script. - Customizable Splashscreen GUI w/Progress Bar - Create a custom "splash screen" GUI with a progress bar and custom label. - _FileGetProperty - Retrieve the properties of a file - SciTE Toolbar - A toolbar demo for use with the SciTE editor - GUIRegisterMsg demo - Demo script to show how to use the Windows messages to interact with controls and your GUI. - Latin Square password generator
jchd Posted May 14, 2011 Posted May 14, 2011 @BrewManNH Right @tbaror TimerInit and TimerDiff will do pretty useful timing, independant of the hosting machine. In practice such query time may be vastly variable depending on NAS load and that is hard to foresee in enterprise environments. Maybe your should monitor query response time and if you observe repeatitively slow response, decrease number of files limit accordingly. This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe hereRegExp tutorial: enough to get startedPCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta. SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)
tbaror Posted May 14, 2011 Author Posted May 14, 2011 @BrewManNHRight@tbarorTimerInit and TimerDiff will do pretty useful timing, independant of the hosting machine. In practice such query time may be vastly variable depending on NAS load and that is hard to foresee in enterprise environments.Maybe your should monitor query response time and if you observe repeatitively slow response, decrease number of files limit accordingly.Thanks for the info ,noted
jchd Posted May 14, 2011 Posted May 14, 2011 Another, possibly simpler way: allow yourself a decent fixed amount of time to perform the task, say 100 files @ 40ms = 4s. Give yourself a safety margin and round this up to 5s. Launch an AdLibRegister function with 5s timeout, pointing to a function which will raise a STOP_IT flag that your querying loop can poll at every new query, exiting the loop as soon as the timeout is done. This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe hereRegExp tutorial: enough to get startedPCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta. SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)
tbaror Posted May 14, 2011 Author Posted May 14, 2011 Thanks again great idea i wasn't aware about AdLibRegister surly will use it.I have more question but now for work.
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now