rudi Posted May 24, 2019 Posted May 24, 2019 Hi. The final question first: Is there a reason, that Autoit doesn't clean up RAM usage as a standard feature? This posting by @guinness was pointing me to the solution for solving my problem: https://www.autoitscript.com/forum/topic/131315-accumulating-memory-usage/?do=findComment&comment=914208 DllCall("psapi.dll", "int", "EmptyWorkingSet", "long", -1) this simple, single line called on a regular basis stopped the script consuming more and more RAM. CU, Rudi. Earth is flat, pigs can fly, and Nuclear Power is SAFE!
Moderators Melba23 Posted May 24, 2019 Moderators Posted May 24, 2019 rudi, As I understand it, Windows does its own RAM clearing over time and using that particular line of code merely pushes a little more of the less-often-used pages in RAM onto a Windows file on the harddisk. So it is a bit of a 2-edged sword as it will slow down your program if it suddenly decides it needs some pages back in RAM for usage. In most cases I have seen over the years, ever-increasing RAM means you have a memory leak in your program - usually a handle or object not correctly disposed of once no longer needed - and fixing that underlying cause, rather than the symptom, has been the key to preventing it. M23 Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind Open spoiler to see my UDFs: Spoiler ArrayMultiColSort ---- Sort arrays on multiple columnsChooseFileFolder ---- Single and multiple selections from specified path treeview listingDate_Time_Convert -- Easily convert date/time formats, including the language usedExtMsgBox --------- A highly customisable replacement for MsgBoxGUIExtender -------- Extend and retract multiple sections within a GUIGUIFrame ---------- Subdivide GUIs into many adjustable framesGUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView itemsGUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeViewMarquee ----------- Scrolling tickertape GUIsNoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxesNotify ------------- Small notifications on the edge of the displayScrollbars ----------Automatically sized scrollbars with a single commandStringSize ---------- Automatically size controls to fit textToast -------------- Small GUIs which pop out of the notification area
rudi Posted May 24, 2019 Author Posted May 24, 2019 (edited) Hi @Melba23, thanks for your reply. As I couldn't see any such program leaks, I played with these two scripts. I used CMD.EXE, as it's available on every Windows PC, the external program I had the problems with is a tiny console prog (no GUI) to do connectivity checks. I noticed, that killing the prog, when the output has the info I'm looking for for speed reasons, the RAM consumption is much higher then when I wait for natural program exit. In the original script the connectivity test is done just every 2 minutes. Anyway, when I let that script run (compiled) when I leave my desk, I have even seen RAM consuptions beyond 1GB, when I came back next day or on Monday. Setting $CleanupRAM = True will activate the RAM cleanup for both sample scripts. Code sample 1: waiting for natural program exit expandcollapse popup; RAM-USAGE-with-ProcessWaitClose.au3 $sDone = "control.exe" $loops = 0 $CleanupRAM = False $aWS = ProcessGetStats() $WSram = $aWS[0] $WSramPeak = $aWS[1] While 1 $loops += 1 $Timer = TimerInit() $PID = Run(@ComSpec & " /c dir c:\windows /s", @TempDir, @SW_HIDE, $STDERR_CHILD + $STDOUT_CHILD) ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $PID = ' & $PID & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console $sOut = "" ProcessWaitClose($PID) $sOut = StdoutRead($PID) UpdToolTip() WEnd Func UpdToolTip() $sResult = "Loops: " & $loops & ", Duration: " & Round(TimerDiff($Timer) / 1000, 2) & " sec." & @CRLF & StringLen($sOut) & " Chars received" $aWS = ProcessGetStats() $WS_TXT = " WorkingSetSize (4k)= " & Round($aWS[0] / 1024 / 4, 2) & @CRLF & "PeakWorkingSetSize (4k)= " & Round($aWS[1] / 1024 / 4, 2) & @CRLF & _ "RAM Delta 4k pages : " & Round(($aWS[0] - $WSram) / 1024 / 4, 2) & @CRLF & _ "Peak Delta 4k pages : " & Round(($aWS[1] - $WSramPeak) / 1024 / 4, 2) If $CleanupRAM Then DllCall("psapi.dll", "int", "EmptyWorkingSet", "long", -1) $aWS = ProcessGetStats() $WS_TXT &= @CRLF & _ "RAM Delta (after cleanup) = " & Round(($aWS[0] - $WSram) / 1024 / 4, 2) & @CRLF & _ "Peak Delta (after cleanup) = " & Round(($aWS[1] - $WSramPeak) / 1024 / 4, 2) EndIf $WSram = $aWS[0] $WSramPeak = $aWS[1] If StringInStr($sOut, $sDone) Then ; ConsoleWrite($sOut & @CRLF) ToolTip("process ended: " & $PID & @CRLF & $WS_TXT & @CRLF & $sResult, 500, 100, "String found: " & $sDone) Else ToolTip("Process ended." & @CRLF & "Search string *NOT* found! " & $sDone & @CRLF & $WS_TXT & @CRLF & $sResult, 500, 100, "String not found: " & $sDone) EndIf EndFunc ;==>UpdToolTip code sample 2: watching out for "my result", killing the PID ASAP expandcollapse popup; RAM-USAGE-with-ProcessClose.au3 #include <AutoItConstants.au3> $sDone = "control.exe" $loops = 0 $CleanupRAM = False $aWS = ProcessGetStats() $WSram = $aWS[0] $WSramPeak = $aWS[1] While 1 $loops += 1 $Timer=TimerInit() $PID = Run(@ComSpec & " /c dir c:\windows\system32 /s", @TempDir, @SW_HIDE, $STDERR_CHILD + $STDOUT_CHILD) ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $PID = ' & $PID & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console $sOut = "" While 1 $sOut &= StdoutRead($PID) If StringInStr($sOut, $sDone) Then ProcessClose($PID) ExitLoop EndIf WEnd UpdToolTip() WEnd Func UpdToolTip() $sResult = "Loops: " & $loops & ", Duration: " & Round(TimerDiff($Timer) / 1000, 2) & " sec." & @CRLF & StringLen($sOut) & " Chars received" $aWS = ProcessGetStats() $WS_TXT = " WorkingSetSize (4k)= " & Round($aWS[0] / 1024 / 4, 2) & @CRLF & "PeakWorkingSetSize (4k)= " & Round($aWS[1] / 1024 / 4, 2) & @CRLF & _ "RAM Delta 4k pages : " & Round(($aWS[0] - $WSram) / 1024 / 4, 2) & @CRLF & _ "Peak Delta 4k pages : " & Round(($aWS[1] - $WSramPeak) / 1024 / 4, 2) If $CleanupRAM Then DllCall("psapi.dll", "int", "EmptyWorkingSet", "long", -1) $aWS = ProcessGetStats() $WS_TXT &= @CRLF & _ "RAM Delta (after cleanup) = " & Round(($aWS[0] - $WSram) / 1024 / 4, 2) & @CRLF & _ "Peak Delta (after cleanup) = " & Round(($aWS[1] - $WSramPeak) / 1024 / 4, 2) EndIf $WSram = $aWS[0] $WSramPeak = $aWS[1] If StringInStr($sOut, $sDone) Then ; ConsoleWrite($sOut & @CRLF) ToolTip("process ended: " & $PID & @CRLF & $WS_TXT & @CRLF & $sResult, 500, 100, "String found: " & $sDone) Else ToolTip("Process ended." & @CRLF & "Search string *NOT* found! " & $sDone & @CRLF & $WS_TXT & @CRLF & $sResult, 500, 100, "String not found: " & $sDone) EndIf EndFunc ;==>UpdToolTip Edited May 24, 2019 by rudi Earth is flat, pigs can fly, and Nuclear Power is SAFE!
rudi Posted May 24, 2019 Author Posted May 24, 2019 @Melba23 <cite> ... using that particular line of code merely pushes a little more of the less-often-used pages in RAM onto a Windows file on the harddisk </cite> Hm. That means the RAM not freed correctly for some reason is just swapped to disk, if I get you correctly? (I missed that aspect before) To the Windows swap file, or to some dedicated "Swap-file-for-this-app"? Hm. Then this is definitely no option, as spoiling disk space, especially the Windows PAGEFILE.SYS to free RAM is not a good idea. Can you see in the sample scripts how to avoid the continuosly growing RAM consumption? TIA Rudi. Earth is flat, pigs can fly, and Nuclear Power is SAFE!
rudi Posted May 24, 2019 Author Posted May 24, 2019 (edited) I think I've caught the problem: It happens, when I use Run() with $STDERR_MERGED to catch the Output. This script is demonstrating the situation. Press F12 to toggle between using and not using $STDERR_MERGED expandcollapse popup; *** Start added by AutoIt3Wrapper *** #include <AutoItConstants.au3> ; *** End added by AutoIt3Wrapper *** #Region ;**** Directives created by AutoIt3Wrapper_GUI **** #AutoIt3Wrapper_Add_Constants=n #EndRegion ;**** Directives created by AutoIt3Wrapper_GUI **** #include <Misc.au3> $loops = 0 $CleanupRAM = False $aWS = ProcessGetStats() $WSram = $aWS[0] $WSramPeak = $aWS[1] $xMax=9999 $yMax=99 Dim $aTest[$xMax+1][$yMax+1] AdlibRegister(UpdToolTip) $PID=False $GrabOutput=False While 1 if _IsPressed("7B") Then if $GrabOutput=False Then $GrabOutput=True else $GrabOutput=False EndIf Sleep(100) EndIf $RandX=Random(0,$xMax,1) $RandY=Random(0,$yMax,1) $aTest[$RandX][$RandY]=$RandX * $RandY * Random(1,100) if ProcessExists($PID) then ProcessClose($PID) if $GrabOutput Then $PID=Run(@ComSpec & " /c echo hello",@TempDir,@SW_HIDE, $STDERR_MERGED) Else $PID=Run(@ComSpec & " /c echo hello",@TempDir,@SW_HIDE) EndIf $loops+=1 WEnd Func UpdToolTip() $sResult = "Loops: " & $loops & @CRLF $aWS = ProcessGetStats() $WS_TXT = " WorkingSetSize (4k)= " & Round($aWS[0] / 1024 / 4, 2) & @CRLF & "PeakWorkingSetSize (4k)= " & Round($aWS[1] / 1024 / 4, 2) & @CRLF & _ "RAM Delta 4k pages : " & Round(($aWS[0] - $WSram) / 1024 / 4, 2) & @CRLF & _ "Peak Delta 4k pages : " & Round(($aWS[1] - $WSramPeak) / 1024 / 4, 2) If $CleanupRAM Then DllCall("psapi.dll", "int", "EmptyWorkingSet", "long", -1) $aWS = ProcessGetStats() $WS_TXT &= @CRLF & _ "RAM Delta (after cleanup) = " & Round(($aWS[0] - $WSram) / 1024 / 4, 2) & @CRLF & _ "Peak Delta (after cleanup) = " & Round(($aWS[1] - $WSramPeak) / 1024 / 4, 2) EndIf $WSram = $aWS[0] $WSramPeak = $aWS[1] if $GrabOutput Then ToolTip($WS_TXT & @CRLF & $sResult & @CRLF & "With $STDERR_MERGED", 500, 100, "Random Array Fill, loops: " & $loops) Else ToolTip($WS_TXT & @CRLF & $sResult & @CRLF & "Without $STDERR_MERGED", 500, 100, "Random Array Fill, loops: " & $loops) EndIf EndFunc ;==>UpdToolTip Helpfile for Run(): $STDERR_MERGED (0x8) = Provides the same handle for STDOUT and STDERR. Implies both $STDOUT_CHILD and $STDERR_CHILD. Shot in the dark: This handle is not destroyed automatically. Howto do that explicitely? Maybe StdioClose() , on my way to try that one. Edited May 24, 2019 by rudi edit typos. 2nd edit about "handle" Earth is flat, pigs can fly, and Nuclear Power is SAFE!
rudi Posted May 24, 2019 Author Posted May 24, 2019 (edited) Okay, I suspect, that it's this combination: Start a Programm using Run() with the opt_flag $STDERR_MERGED killing this process When doing so, it doesn't matter whether StdioClose() is used or not, and in case it *IS* used, whether it's used before or after the ProcessClose($PID), the RAM consumption is increasing with each and every Run() used. Edited May 24, 2019 by rudi Earth is flat, pigs can fly, and Nuclear Power is SAFE!
KaFu Posted May 26, 2019 Posted May 26, 2019 (edited) This works for me If ProcessExists($PID) Then While ProcessExists($PID) ProcessClose($PID) WEnd StdioClose($PID) EndIf Seems like StdioClose must be called to flush the stream, and processclose alone does not ensure that process is closed when StdioClose is called, so fragments pile up. Ensure that process is closed before the StdioClose call seems to fix this. Whole modified script expandcollapse popup; *** Start added by AutoIt3Wrapper *** #include <AutoItConstants.au3> ; *** End added by AutoIt3Wrapper *** #Region ;**** Directives created by AutoIt3Wrapper_GUI **** #AutoIt3Wrapper_Add_Constants=n #EndRegion ;**** Directives created by AutoIt3Wrapper_GUI **** #include <Misc.au3> $loops = 0 $CleanupRAM = False $aWS = ProcessGetStats() $WSram = $aWS[0] $WSramPeak = $aWS[1] $xMax = 9999 $yMax = 99 Dim $aTest[$xMax + 1][$yMax + 1] AdlibRegister(UpdToolTip) $PID = False $sOutput = "" $GrabOutput = True While 1 If _IsPressed("7B") Then If $GrabOutput = False Then $GrabOutput = True Else $GrabOutput = False EndIf Sleep(100) EndIf $RandX = Random(0, $xMax, 1) $RandY = Random(0, $yMax, 1) $aTest[$RandX][$RandY] = $RandX * $RandY * Random(1, 100) If ProcessExists($PID) Then While ProcessExists($PID) ProcessClose($PID) WEnd StdioClose($PID) EndIf If $GrabOutput Then $PID = Run(@ComSpec & " /c echo hello", @TempDir, @SW_HIDE, $STDERR_MERGED) Else $PID = Run(@ComSpec & " /c echo hello", @TempDir, @SW_HIDE) EndIf $loops += 1 WEnd Func UpdToolTip() $sResult = "Loops: " & $loops & @CRLF $aWS = ProcessGetStats() $WS_TXT = " WorkingSetSize (4k)= " & Round($aWS[0] / 1024 / 4, 2) & @CRLF & "PeakWorkingSetSize (4k)= " & Round($aWS[1] / 1024 / 4, 2) & @CRLF & _ "RAM Delta 4k pages : " & Round(($aWS[0] - $WSram) / 1024 / 4, 2) & @CRLF & _ "Peak Delta 4k pages : " & Round(($aWS[1] - $WSramPeak) / 1024 / 4, 2) If $CleanupRAM Then DllCall("psapi.dll", "int", "EmptyWorkingSet", "long", -1) $aWS = ProcessGetStats() $WS_TXT &= @CRLF & _ "RAM Delta (after cleanup) = " & Round(($aWS[0] - $WSram) / 1024 / 4, 2) & @CRLF & _ "Peak Delta (after cleanup) = " & Round(($aWS[1] - $WSramPeak) / 1024 / 4, 2) EndIf $WSram = $aWS[0] $WSramPeak = $aWS[1] If $GrabOutput Then ToolTip($WS_TXT & @CRLF & $sResult & @CRLF & "With $STDERR_MERGED", 500, 100, "Random Array Fill, loops: " & $loops) Else ToolTip($WS_TXT & @CRLF & $sResult & @CRLF & "Without $STDERR_MERGED", 500, 100, "Random Array Fill, loops: " & $loops) EndIf EndFunc ;==>UpdToolTip Edited May 26, 2019 by KaFu OS: Win10-22H2 - 64bit - German, AutoIt Version: 3.3.16.1, AutoIt Editor: SciTE, Website: https://funk.eu AMT - Auto-Movie-Thumbnailer (2024-Oct-13) BIC - Batch-Image-Cropper (2023-Apr-01) COP - Color Picker (2009-May-21) DCS - Dynamic Cursor Selector (2024-Oct-13) HMW - Hide my Windows (2024-Oct-19) HRC - HotKey Resolution Changer (2012-May-16) ICU - Icon Configuration Utility (2018-Sep-16) SMF - Search my Files (2024-Oct-20) - THE file info and duplicates search tool SSD - Set Sound Device (2017-Sep-16)
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now