RAM Usage of Scripts doing external program calls

[rudi]

Hi.

The final question first: Is there a reason, that Autoit doesn't clean up RAM usage as a standard feature?

 

This posting by @guinness was pointing me to the solution for solving my problem:

https://www.autoitscript.com/forum/topic/131315-accumulating-memory-usage/?do=findComment&comment=914208

DllCall("psapi.dll", "int", "EmptyWorkingSet", "long", -1)

this simple, single line called on a regular basis stopped the script consuming more and more RAM.

CU, Rudi.

[Melba23]

rudi,

As I understand it, Windows does its own RAM clearing over time and using that particular  line of code merely pushes a little more of the less-often-used pages in RAM onto a Windows file on the harddisk. So it is a bit of a 2-edged sword as it will slow down your program if it suddenly decides it needs some pages back in RAM for usage.

In most cases I have seen over the years, ever-increasing RAM means you have a memory leak in your program - usually a handle or object not correctly disposed of once no longer needed - and fixing that underlying cause, rather than the symptom, has been the key to preventing it.

M23

 

[rudi]

Hi @Melba23,

 

thanks for your reply.

As I couldn't see any such program leaks, I played with these two scripts. I used CMD.EXE, as it's available on every Windows PC, the external program I had the problems with is a tiny console  prog (no GUI) to do connectivity checks.

 

I noticed, that killing the prog, when the output has the info I'm looking for for speed reasons, the RAM consumption is much higher then when I wait for natural program exit.

 

In the original script the connectivity test is done just every 2 minutes. Anyway, when I let that script run (compiled) when I leave my desk, I have even seen RAM consuptions beyond 1GB, when I came back next day or on Monday.

 

Setting $CleanupRAM = True will activate the RAM cleanup for both sample scripts.

Code sample 1: waiting for natural program exit

expandcollapse popup

; RAM-USAGE-with-ProcessWaitClose.au3

$sDone = "control.exe"
$loops = 0

$CleanupRAM = False

$aWS = ProcessGetStats()
$WSram = $aWS[0]
$WSramPeak = $aWS[1]

While 1
    $loops += 1
    $Timer = TimerInit()
    $PID = Run(@ComSpec & " /c dir c:\windows /s", @TempDir, @SW_HIDE, $STDERR_CHILD + $STDOUT_CHILD)
    ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $PID = ' & $PID & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console
    $sOut = ""
    ProcessWaitClose($PID)
    $sOut = StdoutRead($PID)
    UpdToolTip()
WEnd


Func UpdToolTip()
    $sResult = "Loops: " & $loops & ", Duration: " & Round(TimerDiff($Timer) / 1000, 2) & " sec." & @CRLF & StringLen($sOut) & " Chars received"
    $aWS = ProcessGetStats()
    $WS_TXT = "    WorkingSetSize (4k)= " & Round($aWS[0] / 1024 / 4, 2) & @CRLF & "PeakWorkingSetSize (4k)= " & Round($aWS[1] / 1024 / 4, 2) & @CRLF & _
            "RAM  Delta 4k pages : " & Round(($aWS[0] - $WSram) / 1024 / 4, 2) & @CRLF & _
            "Peak Delta 4k pages : " & Round(($aWS[1] - $WSramPeak) / 1024 / 4, 2)

    If $CleanupRAM Then
        DllCall("psapi.dll", "int", "EmptyWorkingSet", "long", -1)
        $aWS = ProcessGetStats()

        $WS_TXT &= @CRLF & _
                "RAM  Delta (after cleanup) = " & Round(($aWS[0] - $WSram) / 1024 / 4, 2) & @CRLF & _
                "Peak Delta (after cleanup) = " & Round(($aWS[1] - $WSramPeak) / 1024 / 4, 2)
    EndIf
    $WSram = $aWS[0]
    $WSramPeak = $aWS[1]
    If StringInStr($sOut, $sDone) Then
        ; ConsoleWrite($sOut & @CRLF)
        ToolTip("process ended: " & $PID & @CRLF & $WS_TXT & @CRLF & $sResult, 500, 100, "String found: " & $sDone)
    Else
        ToolTip("Process ended." & @CRLF & "Search string *NOT* found! " & $sDone & @CRLF & $WS_TXT & @CRLF & $sResult, 500, 100, "String not found: " & $sDone)
    EndIf
EndFunc   ;==>UpdToolTip

 

code sample 2: watching out for "my result", killing the PID ASAP

expandcollapse popup

; RAM-USAGE-with-ProcessClose.au3

#include <AutoItConstants.au3>
$sDone = "control.exe"
$loops = 0

$CleanupRAM = False

$aWS = ProcessGetStats()
$WSram = $aWS[0]
$WSramPeak = $aWS[1]

While 1
    $loops += 1
    $Timer=TimerInit()
    $PID = Run(@ComSpec & " /c dir c:\windows\system32 /s", @TempDir, @SW_HIDE, $STDERR_CHILD + $STDOUT_CHILD)
    ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $PID = ' & $PID & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console
    $sOut = ""
    While 1
        $sOut &= StdoutRead($PID)
        If StringInStr($sOut, $sDone) Then
            ProcessClose($PID)
            ExitLoop
        EndIf
    WEnd
    UpdToolTip()
WEnd

Func UpdToolTip()
    $sResult = "Loops: " & $loops & ", Duration: " & Round(TimerDiff($Timer) / 1000, 2) & " sec." & @CRLF & StringLen($sOut) & " Chars received"
    $aWS = ProcessGetStats()
    $WS_TXT = "    WorkingSetSize (4k)= " & Round($aWS[0] / 1024 / 4, 2) & @CRLF & "PeakWorkingSetSize (4k)= " & Round($aWS[1] / 1024 / 4, 2) & @CRLF & _
            "RAM  Delta 4k pages : " & Round(($aWS[0] - $WSram) / 1024 / 4, 2) & @CRLF & _
            "Peak Delta 4k pages : " & Round(($aWS[1] - $WSramPeak) / 1024 / 4, 2)

    If $CleanupRAM Then
        DllCall("psapi.dll", "int", "EmptyWorkingSet", "long", -1)
        $aWS = ProcessGetStats()

        $WS_TXT &= @CRLF & _
                "RAM  Delta (after cleanup) = " & Round(($aWS[0] - $WSram) / 1024 / 4, 2) & @CRLF & _
                "Peak Delta (after cleanup) = " & Round(($aWS[1] - $WSramPeak) / 1024 / 4, 2)
    EndIf
    $WSram = $aWS[0]
    $WSramPeak = $aWS[1]
    If StringInStr($sOut, $sDone) Then
        ; ConsoleWrite($sOut & @CRLF)
        ToolTip("process ended: " & $PID & @CRLF & $WS_TXT & @CRLF & $sResult, 500, 100, "String found: " & $sDone)
    Else
        ToolTip("Process ended." & @CRLF & "Search string *NOT* found! " & $sDone & @CRLF & $WS_TXT & @CRLF & $sResult, 500, 100, "String not found: " & $sDone)
    EndIf
EndFunc   ;==>UpdToolTip

 

Edited May 24, 2019 by rudi

[rudi]

@Melba23

<cite>
... using that particular  line of code merely pushes a little more of the less-often-used pages in RAM onto a Windows file on the harddisk
</cite>

Hm. That means the RAM not freed correctly for some reason is just swapped to disk, if I get you correctly? (I missed that aspect before)

To the Windows swap file, or to some dedicated "Swap-file-for-this-app"?

Hm. Then this is definitely no option, as spoiling disk space, especially the Windows PAGEFILE.SYS to free RAM is not a good idea.

 

Can you see in the sample scripts how to avoid the continuosly growing RAM consumption?

TIA Rudi.

[rudi]

I think I've caught the problem:

 

It happens, when I use Run() with $STDERR_MERGED to catch the Output.

 

This script is demonstrating the situation. Press F12 to toggle between using and not using $STDERR_MERGED

expandcollapse popup

; *** Start added by AutoIt3Wrapper ***
#include <AutoItConstants.au3>
; *** End added by AutoIt3Wrapper ***
#Region ;**** Directives created by AutoIt3Wrapper_GUI ****
#AutoIt3Wrapper_Add_Constants=n
#EndRegion ;**** Directives created by AutoIt3Wrapper_GUI ****
#include <Misc.au3>



$loops = 0

$CleanupRAM = False

$aWS = ProcessGetStats()
$WSram = $aWS[0]
$WSramPeak = $aWS[1]

$xMax=9999
$yMax=99
Dim $aTest[$xMax+1][$yMax+1]


AdlibRegister(UpdToolTip)

$PID=False

$GrabOutput=False




While 1
    if _IsPressed("7B") Then
        if $GrabOutput=False Then
            $GrabOutput=True
        else
            $GrabOutput=False
        EndIf
        Sleep(100)
    EndIf

    $RandX=Random(0,$xMax,1)
    $RandY=Random(0,$yMax,1)
    $aTest[$RandX][$RandY]=$RandX * $RandY * Random(1,100)
    if ProcessExists($PID) then ProcessClose($PID)
    if $GrabOutput Then
    $PID=Run(@ComSpec & " /c echo hello",@TempDir,@SW_HIDE, $STDERR_MERGED)
    Else
    $PID=Run(@ComSpec & " /c echo hello",@TempDir,@SW_HIDE)
    EndIf
    $loops+=1
WEnd

Func UpdToolTip()
    $sResult = "Loops: " & $loops & @CRLF
    $aWS = ProcessGetStats()
    $WS_TXT = "    WorkingSetSize (4k)= " & Round($aWS[0] / 1024 / 4, 2) & @CRLF & "PeakWorkingSetSize (4k)= " & Round($aWS[1] / 1024 / 4, 2) & @CRLF & _
            "RAM  Delta 4k pages : " & Round(($aWS[0] - $WSram) / 1024 / 4, 2) & @CRLF & _
            "Peak Delta 4k pages : " & Round(($aWS[1] - $WSramPeak) / 1024 / 4, 2)

    If $CleanupRAM Then
        DllCall("psapi.dll", "int", "EmptyWorkingSet", "long", -1)
        $aWS = ProcessGetStats()

        $WS_TXT &= @CRLF & _
                "RAM  Delta (after cleanup) = " & Round(($aWS[0] - $WSram) / 1024 / 4, 2) & @CRLF & _
                "Peak Delta (after cleanup) = " & Round(($aWS[1] - $WSramPeak) / 1024 / 4, 2)
    EndIf
    $WSram = $aWS[0]
    $WSramPeak = $aWS[1]
        if $GrabOutput Then
        ToolTip($WS_TXT & @CRLF & $sResult & @CRLF & "With $STDERR_MERGED", 500, 100, "Random Array Fill, loops: " & $loops)
        Else
        ToolTip($WS_TXT & @CRLF & $sResult & @CRLF & "Without $STDERR_MERGED", 500, 100, "Random Array Fill, loops: " & $loops)
        EndIf
EndFunc   ;==>UpdToolTip

 

Helpfile for Run():

$STDERR_MERGED (0x8) = Provides the same handle for STDOUT and STDERR. Implies both $STDOUT_CHILD and $STDERR_CHILD.
 

Shot in the dark: This handle is not destroyed automatically.

 

Howto do that explicitely? Maybe StdioClose() , on my way to try that one.

Edited May 24, 2019 by rudi
edit typos. 2nd edit about "handle"

[rudi]

Okay, I suspect, that it's this combination:

 

1. Start a Programm using Run() with the opt_flag $STDERR_MERGED
2. killing this process

 

When doing so, it doesn't matter whether StdioClose() is used or not, and in case it *IS* used, whether it's used before or after the ProcessClose($PID), the RAM consumption is increasing with each and every Run() used.

 

 

Edited May 24, 2019 by rudi

[KaFu]

This works for me

If ProcessExists($PID) Then
    While ProcessExists($PID)
        ProcessClose($PID)
    WEnd
    StdioClose($PID)
EndIf

Seems like StdioClose must be called to flush the stream, and processclose alone does not ensure that process is closed when StdioClose is called, so fragments pile up. Ensure that process is closed before the StdioClose call seems to fix this.

Whole modified script

expandcollapse popup

; *** Start added by AutoIt3Wrapper ***
#include <AutoItConstants.au3>
; *** End added by AutoIt3Wrapper ***
#Region ;**** Directives created by AutoIt3Wrapper_GUI ****
#AutoIt3Wrapper_Add_Constants=n
#EndRegion ;**** Directives created by AutoIt3Wrapper_GUI ****
#include <Misc.au3>

$loops = 0

$CleanupRAM = False

$aWS = ProcessGetStats()
$WSram = $aWS[0]
$WSramPeak = $aWS[1]

$xMax = 9999
$yMax = 99
Dim $aTest[$xMax + 1][$yMax + 1]


AdlibRegister(UpdToolTip)

$PID = False
$sOutput = ""

$GrabOutput = True


While 1
    If _IsPressed("7B") Then
        If $GrabOutput = False Then
            $GrabOutput = True
        Else
            $GrabOutput = False
        EndIf
        Sleep(100)
    EndIf

    $RandX = Random(0, $xMax, 1)
    $RandY = Random(0, $yMax, 1)
    $aTest[$RandX][$RandY] = $RandX * $RandY * Random(1, 100)

    If ProcessExists($PID) Then
        While ProcessExists($PID)
            ProcessClose($PID)
        WEnd
        StdioClose($PID)
    EndIf

    If $GrabOutput Then
        $PID = Run(@ComSpec & " /c echo hello", @TempDir, @SW_HIDE, $STDERR_MERGED)
    Else
        $PID = Run(@ComSpec & " /c echo hello", @TempDir, @SW_HIDE)
    EndIf
    $loops += 1
WEnd

Func UpdToolTip()
    $sResult = "Loops: " & $loops & @CRLF
    $aWS = ProcessGetStats()
    $WS_TXT = "    WorkingSetSize (4k)= " & Round($aWS[0] / 1024 / 4, 2) & @CRLF & "PeakWorkingSetSize (4k)= " & Round($aWS[1] / 1024 / 4, 2) & @CRLF & _
            "RAM  Delta 4k pages : " & Round(($aWS[0] - $WSram) / 1024 / 4, 2) & @CRLF & _
            "Peak Delta 4k pages : " & Round(($aWS[1] - $WSramPeak) / 1024 / 4, 2)

    If $CleanupRAM Then
        DllCall("psapi.dll", "int", "EmptyWorkingSet", "long", -1)
        $aWS = ProcessGetStats()

        $WS_TXT &= @CRLF & _
                "RAM  Delta (after cleanup) = " & Round(($aWS[0] - $WSram) / 1024 / 4, 2) & @CRLF & _
                "Peak Delta (after cleanup) = " & Round(($aWS[1] - $WSramPeak) / 1024 / 4, 2)
    EndIf
    $WSram = $aWS[0]
    $WSramPeak = $aWS[1]
    If $GrabOutput Then
        ToolTip($WS_TXT & @CRLF & $sResult & @CRLF & "With $STDERR_MERGED", 500, 100, "Random Array Fill, loops: " & $loops)
    Else
        ToolTip($WS_TXT & @CRLF & $sResult & @CRLF & "Without $STDERR_MERGED", 500, 100, "Random Array Fill, loops: " & $loops)
    EndIf
EndFunc   ;==>UpdToolTip

 

Edited May 26, 2019 by KaFu