Search the Community

Hello guys, I can not go forward on this pitch: _FileListToArrayRec("C:\files\", "*.ini", 1, 1, 0, True) I was shown this error from Include<File.au3>: "C:\Program Files (x86)\AutoIt3\Include\File.au3" (319) : ==> Incorrect number of parameters in function call.: $sName = FileFindNextFile($hSearch, 1) $sName = ^ ERROR Do you know the cause? Thanks

I'm basically processing text files where I'm reading them from a disk file into an input array, going through the input array and skipping what I don't want and writing it out to an output array and then writing the output array to a disk file. I encountered an "Error allocating memory" message. I have stripped down the program to illustrate how the memory usage grows in a looping routine (__ReadFiles). I have tried a _ReduceMemory routine (from https://www.autoitscript.com/forum/topic/134831-is-there-a-way-to-free-up-memory-being-used/ ) and it helps somewhat, however the peak working set continues to grow regardless. I believe I'm releasing resources (resetting arrays to 0) and am trying to figure out if the leak is from FileFindNextFile or something else I'm doing. To illustrate the program creates a subdirectory with 50000 simple text files and then runs showing the growing memory usage, Ctrl+q will pause program. Rerunning the program uses the already created files so it does not have to be re-created. Any help appreciated. ; use this to debug in console window <--- LOOK #AutoIt3Wrapper_run_debug_mode=Y ;#AutoIt3Wrapper_Run_Debug=off ;#AutoIt3Wrapper_Run_Debug=on #include <Array.au3> #include <File.au3> #include <MsgBoxConstants.au3> #include <GUIConstantsEx.au3> #include <EditConstants.au3> HotKeySet ( "^q", "EndProgram" ) ;v2e - added HotKeySet ( "^q", "EndProgram" ); CTRL+q so can pause/exit script Global $hEdit ;for scrolling text box Global $textarray[1] ;1 element $textarray[0] = 0 ;initial count Global $newtextarray[1] ;1 element $newtextarray[0] = 0 ;initial count Global $dir Global $k ;loop count Global $aMemory ;v1g for use in __ShowMemoryUsage("Location: In FileFindNextFile While 1 loop.") Global $debug = 0 ;0=off, 1=on , 2=deeper, - use debug > 1 to see arrays in Funcs $version = "v1g" ;v1g - we have an Autoit "Error allocating memory" somewhere - printing out WorkingSetSize and PeakWorkingSetSize from ProcessGetStats to try and find it __scrolling_text_box_init("--- Read files ---", 1050, 450) ;__scrolling_text_box_init($title, $width, $height) ;set up box - to display text use: __scrolltext($text) WinMove("--- Read files ---", "", 2000, 300) ;move so can see it - on single screen use 10, 30 on dual screen use 2000, 300 __scrolltext("Starting: " & @CRLF) ;illustrate error using current script directory by creating unique directotry and create file in it and put some text into it ----------- $dir = @ScriptDir ;for iterative testing see if test directory already exists - i.e. only need to build this once $dir = @ScriptDir & "\testdir_xcrgua" ;highly unlikely areadly exists unless this program was already run $str = "" ;null it For $j = 1 to 100 ;put 100 lines in each file $str = $str & $j & @CRLF Next If DirGetSize($dir) = -1 Then DirCreate($dir) ;create directory ;then create and fill it For $i = 1 to 50000 ;brute force (and I know it's slow, however straightforward and only 1 time FileWrite($dir & "\" & $i & ".txt", $str) If $i/1000 = Int($i/1000) Then __scrolltext("Writing test file " & $i & "/50000" & @CRLF) Next EndIf MsgBox(262144, "DEBUG", $dir & " <-- should have 50000 files each with 100 numbered lines 1- 100.") MsgBox(262144, "DEBUG", "Paused.") __scrolltext("Reading files in dir = " & $dir & @CRLF) ;__ShowMemoryUsage("Location: Before file count.") ;v1g $fc = __GetFileCount($dir) ;returns file count in $dir ;__ShowMemoryUsage("Location: After file count.") ;v1g __scrolltext("Number of files in dir = " & $fc & @CRLF) MsgBox(262144, "DEBUG", "Paused.") ;in this we get memory allocation erroroccurs between 25K and 50K on my machine __ReadFiles($dir) MsgBox(262144, "DEBUG", "Paused out of loop.") Exit ;----------------------------------- Functions ----------------------------------- Func __ShowMemoryUsage($title) $aMemory = ProcessGetStats() ; Retrieve memory details about the current process. ; If $aMemory is an array then display the following details about the process. If IsArray($aMemory) Then __scrolltext($title & " " & "WorkingSetSize: " & $aMemory[0] & " PeakWorkingSetSize: " & $aMemory[1] & @CRLF) Else __scrolltext("An error occurred in ProcessGetStats() in Func __ShowMemoryUsage($title).") EndIf EndFunc ;__ShowMemoryUsage($title) Func _ReduceMemory($i_PID = -1) ;from https://www.autoitscript.com/forum/topic/134831-is-there-a-way-to-free-up-memory-being-used/ If $i_PID <> -1 Then Local $ai_Handle = DllCall("kernel32.dll", 'int', 'OpenProcess', 'int', 0x1F0FFF, 'int', False, 'int', $i_PID) Local $ai_Return = DllCall("psapi.dll", 'int', 'EmptyWorkingSet', 'long', $ai_Handle[0]) DllCall('kernel32.dll', 'int', 'CloseHandle', 'int', $ai_Handle[0]) Else Local $ai_Return = DllCall("psapi.dll", 'int', 'EmptyWorkingSet', 'long', -1) EndIf Return $ai_Return[0] EndFunc ;==>_ReduceMemory Func __GetFileCount($dir) ;returns number of files in $dir - brute force but only about 50K files/directory ;MsgBox(0, "DEBUG", "$dir = '" & $dir & "'") #AutoIt3Wrapper_Run_Debug=off $fcount = 0 ;init count $search = FileFindFirstFile($dir & "\*.*") If $search = -1 Then ;no files found Return($fcount) ;MsgBox(0, "Error", "No files/directories matched the search pattern") ;Exit EndIf ;okay check number of files While 1 $file = FileFindNextFile($search) If @error Then ExitLoop ;exit loop when there are no more files Else $fcount = $fcount + 1 EndIf WEnd FileClose($search) ;be polite Return($fcount) #AutoIt3Wrapper_Run_Debug=on EndFunc Func __ReadFiles($dir) ;just reading to show memory alloc error Local $x Local $skipline #AutoIt3Wrapper_Run_Debug=off $tfc = __GetFileCount($dir) ;get total file count so we can show progress #AutoIt3Wrapper_Run_Debug=on $cfc = 0 ;init currect file count $search = FileFindFirstFile($dir & "\*.*") ;get search handle If $search = -1 Then ;no files found __scrolltext("Working on file: " & $cfc & "/" & $tfc & @CRLF) Return ;MsgBox(0, "Error", "No files/directories matched the search pattern") ;Exit EndIf ;okay go through the files one by one While 1 ;_ReduceMemory() ;this makes little difference - surprising $file = FileFindNextFile($search) If @error = 1 Then ExitLoop ;exit loop when there are no more to do If @extended = 1 Then $adir = 1 ;we have a directory so note it Else $adir = 0 ;a file EndIf ;MsgBox(4096, "File:", $directory&"\"&$file) $cfc = $cfc + 1 __scrolltext("Working on file: " & $cfc & "/" & $tfc & " " & $file & @CRLF) ;show the filename __ShowMemoryUsage("Location: In FileFindNextFile While 1 loop.") ;v1g ;clear out arrays so we have no artifacts from procesing the prior loop ;from here it looks like resetting it is better https://www.autoitscript.com/forum/topic/110933-solved-how-to-delete-whole-array/ $testarray = 0 $newtextarray = 0 Dim $textarray[1] ;1 element $textarray[0] = 0 ;initial count Dim $newtextarray[1] ;1 element $newtextarray[0] = 0 ;initial count ;bring file into memory for faster processing If $adir = 0 Then ;process file $x = _FileReadToArray($dir & "\" & $file, $textarray) ;bring file into memory for faster processing [0] has $textarray count If $x <> 1 Then MsgBox($MB_TOPMOST + $MB_ICONERROR, "ERROR", "Unable to read file into array. Error = " & $x & " @error = " & @error & " @extended = " & @extended) Exit EndIf If $debug > 0 Then _ArrayDisplay($textarray, "INPUT $textarray") ReDim $newtextarray[$textarray[0] + 1] ;create output array ($newtextarray) of the same size as input array ($textarray) and later ReDim the output array to the correct size $newtextarray[0] = 0 ;set the count of actual entries as $newtextarray is empty but has same size as $textarray If $debug > 0 Then _ArrayDisplay($newtextarray, "Starting $newtextarray") ;now the approach is to write out only lines we want then to file For $i = 1 to $textarray[0] ;process all elements (rows, lines) in input array ;If $debug > 0 Then __scrolltext("$textarray[" & $i & "] = '" & $textarray[$i] & "'" & @CRLF) ;show the line $skipline = 0 ;init value to not skip line ;----------- start stripping out stuff ----------- ;this rountine is not the cause of the memory leak so removed ;------------- If $skipline = 0 Then ;write it out $newtextarray[0] = $newtextarray[0] + 1 ;increase count and point to index where we're going to store it in $newtextarray $newtextarray[$newtextarray[0]] = $textarray[$i] Else ;skip the line EndIf Next $fn = $dir & "\processed-1\" & StringLeft($file, StringLen($file) - 4) & "-1.txt" __scrolltext("Processed file written to: " &$fn & @CRLF) ;show the line ;RECALL written out like an array so 1st entry which corresponds to index [0] has count of rows = lines in the file Else ;its a directory - skip it __scrolltext("NOT processed directory (i.e. skipped) : " & $dir & "\" & $file & @CRLF) ;show the line EndIf WEnd EndFunc ;__CleanFiles($dir) Func __scrolling_text_box_init($title, $width, $height) ;from http://www.autoitscript.com/forum/topic/110948-add-text-to-edit-box-and-scroll-it-down/page__p__971158__hl___guictrledit_scroll__fromsearch__1#entry971158 $hGUI = GUICreate($title, $width, $height) $hEdit = GUICtrlCreateEdit("", 10, 10, $width-20, $height-20, BitOr($GUI_SS_DEFAULT_EDIT, $ES_READONLY)) ;$limit = 9223372036854775807 ;2^63 $limit = 1000000000 ;works $x = GUICtrlSetLimit($hEdit, $limit) ;~ If $x = 0 Then ;~ MsgBox(0, "ERROR", "Limit of " & $limit & " is too large.") ;~ Else ;~ MsgBox(0, "Okay", "Limit of " & $limit & " is okay.") ;~ EndIf ;$hButton = GUICtrlCreateButton("Add", 10, 250, 80, 30) GUISetState() EndFunc Func __scrolltext($text) #AutoIt3Wrapper_Run_Debug=off ;prefix everything with date and time e.g. 2014-02-01 HH:MM:SS $dt = @YEAR & "-" & @MON & "-" & @MDAY & " " & @HOUR & ":" & @MIN & ":" & @SEC & " " $text = $dt & $text GUICtrlSetData($hEdit, $text, 1) #AutoIt3Wrapper_Run_Debug=on EndFunc Func EndProgram() $x = MsgBox($MB_TOPMOST + $MB_YESNO + $MB_ICONWARNING, "Paused", "Do you want to continue?") If $x = $IDNO Then Exit ;exit program Else ;just a pause - so continue EndIf EndFunc 04 - Clean files - v1g - test for memory leak v1c.au3