[Solved] Memory Leak

[ahha]

I'm basically processing text files where I'm reading them from a disk file into an input array, going through the input array and skipping what I don't want and writing it out to an output array and then writing the output array to a disk file.  I encountered an "Error allocating memory" message.  I have stripped down the program to illustrate how the memory usage grows in a looping routine (__ReadFiles).  I have tried a _ReduceMemory routine (from https://www.autoitscript.com/forum/topic/134831-is-there-a-way-to-free-up-memory-being-used/ ) and it helps somewhat, however the peak working set continues to grow regardless.  I believe I'm releasing resources (resetting arrays to 0) and am trying to figure out if the leak is from FileFindNextFile or something else I'm doing.   To illustrate the program creates a subdirectory with 50000 simple text files and then runs showing the growing memory usage, Ctrl+q will pause program.  Rerunning the program uses the already created files so it does not have to be re-created.  Any help appreciated.

expandcollapse popup

; use this to debug in console window <--- LOOK
#AutoIt3Wrapper_run_debug_mode=Y
;#AutoIt3Wrapper_Run_Debug=off
;#AutoIt3Wrapper_Run_Debug=on


#include <Array.au3>
#include <File.au3>
#include <MsgBoxConstants.au3>
#include <GUIConstantsEx.au3>
#include <EditConstants.au3>

HotKeySet ( "^q", "EndProgram" )    ;v2e - added HotKeySet ( "^q", "EndProgram" ); CTRL+q so can pause/exit script

Global $hEdit   ;for scrolling text box

Global $textarray[1]    ;1 element
$textarray[0] = 0       ;initial count
Global $newtextarray[1] ;1 element
$newtextarray[0] = 0    ;initial count

Global $dir
Global $k       ;loop count

Global $aMemory ;v1g    for use in __ShowMemoryUsage("Location: In FileFindNextFile While 1 loop.")

Global $debug = 0       ;0=off, 1=on , 2=deeper,  - use debug > 1 to see arrays in Funcs


$version = "v1g"
;v1g - we have an Autoit "Error allocating memory" somewhere - printing out WorkingSetSize and  PeakWorkingSetSize from ProcessGetStats to try and find it


__scrolling_text_box_init("--- Read files ---", 1050, 450)  ;__scrolling_text_box_init($title, $width, $height)     ;set up box - to display text use: __scrolltext($text)
WinMove("--- Read files ---", "", 2000, 300)    ;move so can see it - on single screen use  10, 30 on dual screen use 2000, 300
__scrolltext("Starting:  " & @CRLF)


;illustrate error using current script directory by creating unique directotry and create file in it and put some text into it -----------
$dir = @ScriptDir

;for iterative testing see if test directory already exists - i.e. only need to build this once
$dir = @ScriptDir & "\testdir_xcrgua"   ;highly unlikely areadly exists unless this program was already run
$str = ""   ;null it
For $j = 1 to 100   ;put 100 lines in each file
    $str = $str & $j & @CRLF
Next

If DirGetSize($dir) = -1 Then
    DirCreate($dir) ;create directory   ;then create and fill it
    For $i = 1 to 50000     ;brute force (and I know it's slow, however straightforward and only 1 time
        FileWrite($dir & "\" & $i & ".txt", $str)
        If $i/1000 = Int($i/1000) Then __scrolltext("Writing test file " & $i & "/50000" & @CRLF)
    Next
EndIf
MsgBox(262144, "DEBUG", $dir & "  <-- should have 50000 files each with 100 numbered lines 1- 100.")

MsgBox(262144, "DEBUG", "Paused.")


__scrolltext("Reading files in dir = " & $dir & @CRLF)
;__ShowMemoryUsage("Location: Before file count.")  ;v1g
$fc = __GetFileCount($dir)  ;returns file count in $dir
;__ShowMemoryUsage("Location: After file count.")   ;v1g
__scrolltext("Number of files in dir = " & $fc & @CRLF)
MsgBox(262144, "DEBUG", "Paused.")

;in this we get memory allocation erroroccurs between 25K and 50K on my machine
__ReadFiles($dir)


MsgBox(262144, "DEBUG", "Paused out of loop.")

Exit


;----------------------------------- Functions -----------------------------------

Func    __ShowMemoryUsage($title)

        $aMemory = ProcessGetStats()    ; Retrieve memory details about the current process.

        ; If $aMemory is an array then display the following details about the process.
        If IsArray($aMemory) Then
            __scrolltext($title & "   " & "WorkingSetSize: " & $aMemory[0] & "      PeakWorkingSetSize: " & $aMemory[1] & @CRLF)
        Else
            __scrolltext("An error occurred in ProcessGetStats() in Func __ShowMemoryUsage($title).")
        EndIf

EndFunc     ;__ShowMemoryUsage($title)


Func _ReduceMemory($i_PID = -1) ;from https://www.autoitscript.com/forum/topic/134831-is-there-a-way-to-free-up-memory-being-used/
    If $i_PID <> -1 Then
        Local $ai_Handle = DllCall("kernel32.dll", 'int', 'OpenProcess', 'int', 0x1F0FFF, 'int', False, 'int', $i_PID)
        Local $ai_Return = DllCall("psapi.dll", 'int', 'EmptyWorkingSet', 'long', $ai_Handle[0])
        DllCall('kernel32.dll', 'int', 'CloseHandle', 'int', $ai_Handle[0])
    Else
        Local $ai_Return = DllCall("psapi.dll", 'int', 'EmptyWorkingSet', 'long', -1)
    EndIf
    Return $ai_Return[0]
EndFunc   ;==>_ReduceMemory



Func    __GetFileCount($dir)    ;returns number of files in $dir - brute force but only about 50K files/directory

    ;MsgBox(0, "DEBUG", "$dir = '" & $dir & "'")
#AutoIt3Wrapper_Run_Debug=off

    $fcount = 0     ;init count
    $search = FileFindFirstFile($dir & "\*.*")

    If $search = -1 Then    ;no files found
        Return($fcount)
        ;MsgBox(0, "Error", "No files/directories matched the search pattern")
        ;Exit
    EndIf

    ;okay check number of files
    While 1
        $file = FileFindNextFile($search)
        If @error Then
            ExitLoop        ;exit loop when there are no more files
        Else
            $fcount = $fcount + 1
        EndIf
    WEnd

    FileClose($search)  ;be polite

    Return($fcount)

#AutoIt3Wrapper_Run_Debug=on

EndFunc


Func    __ReadFiles($dir)   ;just reading to show memory alloc error

    Local $x
    Local $skipline

#AutoIt3Wrapper_Run_Debug=off
    $tfc = __GetFileCount($dir) ;get total file count so we can show progress
#AutoIt3Wrapper_Run_Debug=on

    $cfc = 0    ;init currect file count
    $search = FileFindFirstFile($dir & "\*.*")  ;get search handle

    If $search = -1 Then    ;no files found
        __scrolltext("Working on file: " & $cfc & "/" & $tfc & @CRLF)
        Return
        ;MsgBox(0, "Error", "No files/directories matched the search pattern")
        ;Exit
    EndIf

    ;okay go through the files one by one
    While 1

        ;_ReduceMemory()    ;this makes little difference - surprising

        $file = FileFindNextFile($search)
        If @error = 1 Then ExitLoop     ;exit loop when there are no more to do
        If @extended = 1 Then
            $adir = 1   ;we have a directory so note it
        Else
            $adir = 0   ;a file
        EndIf

        ;MsgBox(4096, "File:", $directory&"\"&$file)
        $cfc = $cfc + 1
        __scrolltext("Working on file: " & $cfc & "/" & $tfc & "     " & $file & @CRLF) ;show the filename

        __ShowMemoryUsage("Location: In FileFindNextFile While 1 loop.")    ;v1g

        ;clear out arrays so we have no artifacts from procesing the prior loop
        ;from here it looks like resetting it is better https://www.autoitscript.com/forum/topic/110933-solved-how-to-delete-whole-array/
        $testarray = 0
        $newtextarray = 0

        Dim $textarray[1]       ;1 element
        $textarray[0] = 0       ;initial count
        Dim $newtextarray[1]    ;1 element
        $newtextarray[0] = 0    ;initial count

        ;bring file into memory for faster processing
        If $adir = 0 Then   ;process file
            $x = _FileReadToArray($dir & "\" & $file, $textarray)       ;bring file into memory for faster processing [0] has $textarray count
            If $x <> 1 Then
                MsgBox($MB_TOPMOST + $MB_ICONERROR, "ERROR", "Unable to read file into array.  Error = " & $x & "    @error = " & @error & "    @extended = " & @extended)
                Exit
            EndIf
            If $debug > 0 Then _ArrayDisplay($textarray, "INPUT $textarray")

            ReDim $newtextarray[$textarray[0] + 1]  ;create output array ($newtextarray) of the same size as input array ($textarray) and later ReDim the output array to the correct size
            $newtextarray[0] = 0        ;set the count of actual entries as $newtextarray is empty but has same size as $textarray
            If $debug > 0 Then _ArrayDisplay($newtextarray, "Starting $newtextarray")

            ;now the approach is to write out only lines we want then to file
            For $i = 1 to $textarray[0] ;process all elements (rows, lines) in input array
                ;If $debug > 0 Then __scrolltext("$textarray[" & $i & "] = '" & $textarray[$i] & "'" & @CRLF)   ;show the line
                $skipline = 0   ;init value to not skip line
                ;----------- start stripping out stuff -----------
                ;this rountine is not the cause of the memory leak so removed
                ;-------------
                If $skipline = 0 Then   ;write it out
                    $newtextarray[0] = $newtextarray[0] + 1     ;increase count and point to index where we're going to store it in $newtextarray
                    $newtextarray[$newtextarray[0]] = $textarray[$i]
                Else
                    ;skip the line
                EndIf
            Next

            $fn = $dir & "\processed-1\" & StringLeft($file, StringLen($file) - 4) & "-1.txt"

            __scrolltext("Processed file written to: " &$fn & @CRLF)    ;show the line
            ;RECALL written out like an array so 1st entry which corresponds to index [0] has count of rows = lines in the file
        Else    ;its a directory - skip it
            __scrolltext("NOT processed directory (i.e. skipped) : " & $dir & "\" & $file & @CRLF)  ;show the line
        EndIf

    WEnd

EndFunc ;__CleanFiles($dir)



Func __scrolling_text_box_init($title, $width, $height)
    ;from http://www.autoitscript.com/forum/topic/110948-add-text-to-edit-box-and-scroll-it-down/page__p__971158__hl___guictrledit_scroll__fromsearch__1#entry971158

    $hGUI = GUICreate($title, $width, $height)
    $hEdit = GUICtrlCreateEdit("", 10, 10, $width-20, $height-20, BitOr($GUI_SS_DEFAULT_EDIT, $ES_READONLY))
    ;$limit = 9223372036854775807   ;2^63
    $limit = 1000000000 ;works
    $x = GUICtrlSetLimit($hEdit, $limit)
    ;~ If $x = 0 Then
    ;~  MsgBox(0, "ERROR", "Limit of " &  $limit & " is too large.")
    ;~ Else
    ;~  MsgBox(0, "Okay", "Limit of " &  $limit & " is okay.")
    ;~ EndIf

    ;$hButton = GUICtrlCreateButton("Add", 10, 250, 80, 30)

    GUISetState()

EndFunc


Func __scrolltext($text)
#AutoIt3Wrapper_Run_Debug=off
    ;prefix everything with date and time e.g. 2014-02-01 HH:MM:SS
    $dt = @YEAR & "-" & @MON & "-" & @MDAY & " " & @HOUR & ":" & @MIN & ":" & @SEC & "  "
    $text = $dt & $text
    GUICtrlSetData($hEdit, $text, 1)
#AutoIt3Wrapper_Run_Debug=on
EndFunc


Func EndProgram()
    $x = MsgBox($MB_TOPMOST + $MB_YESNO + $MB_ICONWARNING, "Paused", "Do you want to continue?")
    If $x = $IDNO Then
        Exit    ;exit program
    Else
        ;just a pause - so continue
    EndIf
EndFunc

 

04 - Clean files - v1g - test for memory leak v1c.au3

Edited February 3, 2016 by ahha
Added as a file.

[Bowmore]

Hi ahha

This is the line in your __scrolltext() function that is using all your memory.

GUICtrlSetData($hEdit, $text,1)

It's appending to the existing contents of the control each time it is executed. If you change it to this your memory problems will disappear.

GUICtrlSetData($hEdit, $text)

 

[ahha]

Bowmore,

Thanks - I totally missed that - duh - I feel stupid.  I do have a question however.  Why when using scrolling text does the PeakWorkingSetSize increase but not the WorkingSetSize.  I would have thought the WorkingSetSize would increase if the text is being stored/displayed. I've forgotten how to indicate solved at the topic top.

Edited February 1, 2016 by ahha

[EmilyLove]

On 2/1/2016 at 7:44 PM, ahha said:

Bowmore,

Thanks - I totally missed that - duh - I feel stupid.  I do have a question however.  Why when using scrolling text does the PeakWorkingSetSize increase but not the WorkingSetSize.  I would have thought the WorkingSetSize would increase if the text is being stored/displayed. I've forgotten how to indicate solved at the topic top.

Edit your original post and change the title to [Solved] Memory Leak