_ProcessListProperties()

[PsaltyDS]

I have a memory leak in one of my programs and seemed to have narrowed it down to using this function. My program is a program to help testing programs we make that tracks memory, cpu and thread usage over time intervals. It uses _ProcessListProperties to check CPU usage for a specified executable.

Is there a memory leak? Here is an example script showing memory usage blowing up from calling this function.

(Paste the _ProcessListProperties function at the end of the script)

#include <array.au3>

While 1
    $array = _ProcessListProperties("AutoIt3.exe")  ;Replace "AutoIt3.exe" with any running process
    If IsArray($array) Then
        For $i = 0 To UBound($array, 2) - 1
            Switch $i
                Case 0
                    $string = $array[1][0]
                Case Else
                    $string = $string & @TAB & $array[1][$i]
            EndSwitch
        Next
        ConsoleWrite("_ArrayToString($array): " & $string & @CRLF)
    Else
        ConsoleWrite("$array: " & $array & @CRLF)
    EndIf
WEnd

I have reproduced your finding, though I used this to get a tighter loop, and to have an ESC plan:

HotKeySet("{ESC}", "_Quit")

Global $avArray

$n = 1
While 1
    $avArray = _ProcessListProperties("AutoIt3.exe")
    ConsoleWrite("Debug: Pass: " & $n & "  Memory usage: " & $avArray[1][7] & @LF)
    $n += 1
WEnd

Func _Quit()
    Exit
EndFunc  ;==>_Quit

I tried explicitly zeroizing the COM objects when finished, with no change. Will investigate further.

I also stumbled accross a horrendous logic bug I never noticed before: Because the function starts out with a local ProcessList() the remote computer access parameter is useless! I'll have to think about how I want it work now. I'll probably just remove ProcessList() inside the function and start the array from scratch based on $colProcs.

Rats!

[PsaltyDS]

On the memory leak issue: It only appears for me when the script is monitoring itself (AutoIt3.exe). When I monitor some other process, I don't get the problem:

HotKeySet("{ESC}", "_Quit")

Global $avArray

$n = 1
While 1
    $avArray = _ProcessListProperties("Firefox.exe")
    ConsoleWrite("Debug: Pass: " & $n & "  Memory usage: " & $avArray[1][7] & @LF)
    $n += 1
WEnd

Func _Quit()
    Exit
EndFunc  ;==>_Quit

On the remote access: I intend to keep the remote computer option, so I will be removing the local-only ProcessList() parts once I get a replacement coded.

[weaponx]

On the memory leak issue: It only appears for me when the script is monitoring itself (AutoIt3.exe). When I monitor some other process, I don't get the problem:

HotKeySet("{ESC}", "_Quit")
 
 Global $avArray
 
 $n = 1
 While 1
     $avArray = _ProcessListProperties("Firefox.exe")
     ConsoleWrite("Debug: Pass: " & $n & "  Memory usage: " & $avArray[1][7] & @LF)
     $n += 1
 WEnd
 
 Func _Quit()
     Exit
 EndFunc ;==>_Quit

On the remote access: I intend to keep the remote computer option, so I will be removing the local-only ProcessList() parts once I get a replacement coded.

I never understood why you even needed ProcessList() when you were already enumerating the process collection which has everything you need.

[PsaltyDS]

I never understood why you even needed ProcessList() when you were already enumerating the process collection which has everything you need.

Slow mental reflexes... comes with age...

Here is my current work-in-progress version:

expandcollapse popup

;===============================================================================
; Function Name:    _ProcessListProperties()
; Description:   Get various properties of a process, or all processes
; Call With:       _ProcessListProperties( [$Process [, $sComputer]] )
; Parameter(s):  (optional) $Process - PID or name of a process, default is "" (all)
;          (optional) $sComputer - remote computer to get list from, default is local
; Requirement(s):   AutoIt v3.2.4.9+
; Return Value(s):  On Success - Returns a 2D array of processes, as in ProcessList()
;            with additional columns added:
;            [0][0] - Number of processes listed (can be 0 if no matches found)
;            [1][0] - 1st process name
;            [1][1] - 1st process PID
;            [1][2] - 1st process Parent PID
;            [1][3] - 1st process owner
;            [1][4] - 1st process priority (0 = low, 31 = high)
;            [1][5] - 1st process executable path
;            [1][6] - 1st process CPU usage
;            [1][7] - 1st process memory usage
;            [1][8] - 1st process creation date/time = "MM/DD/YYY hh:mm:ss" (hh = 00 to 23)
;            [1][9] - 1st process command line string
;            ...
;            [n][0] thru [n][9] - last process properties
; On Failure:      Returns array with [0][0] = 0 and sets @Error to non-zero (see code below)
; Author(s):        PsaltyDS at http://www.autoitscript.com/forum
; Date/Version:   06/10/2008  --  v2.0.0
; Notes:            If a numeric PID or string process name is provided and no match is found,
;            then [0][0] = 0 and @error = 0 (not treated as an error, same as ProcessList)
;          This function requires admin permissions to the target computer.
;          All properties come from the Win32_Process class in WMI.
;           To get time-base properties (CPU and Memory usage), a 100ms SWbemRefresher is used.
;===============================================================================
Func _ProcessListProperties($Process = "", $sComputer = ".")
    Local $sUserName, $sMsg, $sUserDomain, $avProcs, $dtmDate
    Local $avProcs[1][2] = [[0, ""]], $n = 1
    
; Connect to WMI and get process objects
    $oWMI = ObjGet("winmgmts:{impersonationLevel=impersonate,authenticationLevel=pktPrivacy}!\\" & $sComputer & "\root\cimv2")
    If IsObj($oWMI) Then
    ; Get collection of all processes from Win32_Process
        $colProcs = $oWMI.ExecQuery("select * from win32_process")
        If IsObj($colProcs) Then
        ; Return for no matches
            If $colProcs.count = 0 Then Return $avProcs
            
        ; Size the array
            ReDim $avProcs[$colProcs.count + 1][10]
            $avProcs[0][0] = UBound($avProcs) - 1
            
        ; For each process...
            For $oProc In $colProcs
            ; [n][0] = Process name
                $avProcs[$n][0] = $oProc.name
            ; [n][1] = Process PID
                $avProcs[$n][1] = $oProc.ProcessId
            ; [n][2] = Parent PID
                $avProcs[$n][2] = $oProc.ParentProcessId
            ; [n][3] = Owner
                If $oProc.GetOwner($sUserName, $sUserDomain) = 0 Then $avProcs[$n][3] = $sUserDomain & "\" & $sUserName
            ; [n][4] = Priority
                $avProcs[$n][4] = $oProc.Priority
            ; [n][5] = Executable path
                $avProcs[$n][5] = $oProc.ExecutablePath
            ; [n][8] = Creation date/time
                $dtmDate = $oProc.CreationDate
                If $dtmDate <> "" Then
                    $dtmDate = StringMid($dtmDate, 5, 2) & "/" & _
                            StringMid($dtmDate, 7, 2) & "/" & _
                            StringLeft($dtmDate, 4) & " " & _
                            StringMid($dtmDate, 9, 2) & ":" & _
                            StringMid($dtmDate, 11, 2) & ":" & _
                            StringMid($dtmDate, 13, 2)
                EndIf
                $avProcs[$n][8] = $dtmDate
            ; [n][9] = Command line string
                $avProcs[$n][9] = $oProc.CommandLine
                
            ; increment index
                $n += 1
            Next
        Else
            SetError(2); Error getting process collection from WMI
        EndIf
    ; release the collection object
        $colProcs = 0
        
    ; Get collection of all processes from Win32_PerfFormattedData_PerfProc_Process
    ; Have to use an SWbemRefresher to pull the collection, or all Perf data will be zeros
        Local $oRefresher = ObjCreate("WbemScripting.SWbemRefresher")
        $colProcs = $oRefresher.AddEnum($oWMI, "Win32_PerfFormattedData_PerfProc_Process" ).objectSet
        $oRefresher.Refresh
        
    ; Time delay before calling refresher
        Local $iTime = TimerInit()
        Do
            Sleep(20)
        Until TimerDiff($iTime) >= 100
        $oRefresher.Refresh
        
    ; Get PerfProc data
        For $oProc In $colProcs
        ; Find it in the array
            For $n = 1 To $avProcs[0][0]
                If $avProcs[$n][1] = $oProc.IDProcess Then
                ; [n][6] = CPU usage
                    $avProcs[$n][6] = $oProc.PercentProcessorTime
                ; [n][7] = memory usage
                    $avProcs[$n][7] = $oProc.WorkingSet
                    ExitLoop
                EndIf
            Next
        Next
    Else
        SetError(1); Error connecting to WMI
    EndIf
    
; Return array
    Return $avProcs
EndFunc  ;==>_ProcessListProperties

Can't put the time in for thorough testing at the moment.

[weaponx]

I'm not sure if anyone has done this before but you can format wmi dates using regex back references:

$original = "20080613104425.484375-240"
;Convert WMI date to YYYY/MM/DD HH:MM:SS
$new = StringRegExpReplace($original, "\A(\d{4})(\d{2})(\d{2})(\d{2})(\d{2})(\d{2})(?:.*)","$1/$2/$3 $4:$5:$6")
ConsoleWrite($new)

;Convert WMI date to MM/DD/YYYY HH:MM:SS
$new = StringRegExpReplace($original, "\A(\d{4})(\d{2})(\d{2})(\d{2})(\d{2})(\d{2})(?:.*)","$2/$3/$1 $4:$5:$6") 
ConsoleWrite($new)

This will knock off a few lines.

Edited June 12, 2008 by weaponx

[ptrex]

@weaponx

I like your method !!

Here are a few other options :

expandcollapse popup

#include <Date.au3>

$strComputer = "."

$objWMIService = ObjGet("winmgmts:\\" & $strComputer & "\root\cimv2")

$colOperatingSystems = _
    $objWMIService.ExecQuery ("Select * from Win32_OperatingSystem")

For $objOperatingSystem in _
    $colOperatingSystems
    ConsoleWrite( "Not Converted : " & $objOperatingSystem.InstallDate & @LF)
    $temp = WMIDateTimeConvert($objOperatingSystem.InstallDate)
    ConsoleWrite( "Converted : " &  $temp & @LF)
    $stemp = WMIDateStringToDate_EURO($objOperatingSystem.InstallDate)
    ConsoleWrite( "String Converted : " &  $stemp & @LF)
    ConsoleWrite( "String Converted au3 : " & WMIDateStringToDate_AU3($objOperatingSystem.InstallDate) & @LF)
    ConsoleWrite( "Number of Seconds since : " & _DateDiff( 's',WMIDateStringToDate_AU3($objOperatingSystem.InstallDate),_NowCalc()) & @LF)
    ConsoleWrite( "Number of Minutes since : " &  _DateDiff( 'n',WMIDateStringToDate_AU3($objOperatingSystem.InstallDate),_NowCalc()) & @LF)
    ConsoleWrite( "Number of Hours since : " &  _DateDiff( 'h',WMIDateStringToDate_AU3($objOperatingSystem.InstallDate),_NowCalc()) & @LF)
    ConsoleWrite( "Number of Days since : " &  _DateDiff( 'd',WMIDateStringToDate_AU3($objOperatingSystem.InstallDate),_NowCalc()) & @LF)
    ConsoleWrite( "Number of Weeks since : " &  _DateDiff( 'w',WMIDateStringToDate_AU3($objOperatingSystem.InstallDate),_NowCalc()) & @LF)
    ConsoleWrite( "Number of Months since : " &  _DateDiff( 'm',WMIDateStringToDate_AU3($objOperatingSystem.InstallDate),_NowCalc()) & @LF)
    ConsoleWrite( "Number of Years since : " &  _DateDiff( 'y',WMIDateStringToDate_AU3($objOperatingSystem.InstallDate),_NowCalc()) & @LF)
    ConsoleWrite("Convert WMI date to YYYY/MM/DD HH:MM:SS : " & WMIDateStringToDateTime_EURO($objOperatingSystem.InstallDate)& @CRLF)
    ConsoleWrite("Convert WMI date to MM/DD/YYYY HH:MM:SS : " & WMIDateStringToDateTime_US($objOperatingSystem.InstallDate)& @CRLF)
Next

Func WMIDateTimeConvert($dtmDate)
local $Ret
    If $dtmDate <> "" Then
    $objSWbemDateTime = ObjCreate("WbemScripting.SWbemDateTime")
    
    $objSWbemDateTime.Value = $dtmDate 
    $Ret = $objSWbemDateTime.GetVarDate(true) ; True or False is corrected for local TimeZone or not
        Return $Ret
    Else
        Return
    EndIf
EndFunc

Func WMIDateStringToDate_US($dtmDate)
    Return (StringMid($dtmDate, 5, 2) & "/" & _
    StringMid($dtmDate, 7, 2)  & "/" & StringLeft($dtmDate, 4) _
    & " " & StringMid($dtmDate, 9, 2) & ":" & StringMid($dtmDate, 11, 2) & ":" & StringMid($dtmDate,13, 2))
EndFunc

Func WMIDateStringToDate_EURO($dtmDate)
    Return (StringMid($dtmDate, 7, 2) & "/" & _
     StringMid($dtmDate, 5, 2) & "/" & StringLeft($dtmDate, 4) _
    & " " & StringMid($dtmDate, 9, 2) & ":" & StringMid($dtmDate, 11, 2) & ":" & StringMid($dtmDate,13, 2))
EndFunc

Func WMIDateStringToDate_AU3($dtmDate)
       Return (StringLeft($dtmDate, 4) & "/" & _
     StringMid($dtmDate, 5, 2) & "/" &  StringMid($dtmDate, 7, 2) _
    & " " & StringMid($dtmDate, 9, 2) & ":" & StringMid($dtmDate, 11, 2) & ":" & StringMid($dtmDate,13, 2))
EndFunc

Func WMIDateStringToDateTime_EURO($dtmDate)
    Return(StringRegExpReplace($dtmDate, "\A(\d{4})(\d{2})(\d{2})(\d{2})(\d{2})(\d{2})(?:.*)","$1/$2/$3 $4:$5:$6"))
EndFunc

Func WMIDateStringToDateTime_US($dtmDate)
Return(StringRegExpReplace($dtmDate, "\A(\d{4})(\d{2})(\d{2})(\d{2})(\d{2})(\d{2})(?:.*)","$2/$3/$1 $4:$5:$6"))
EndFunc

I know it's a bit of topic,but I 'm sure PsaltyDS won't mind.

Regards,

ptrex

[PsaltyDS]

Stop it! Both of you!

The RegExp geekiness is making me go blind!

expandcollapse popup

#include <Array.au3>; Only for _ArrayDisplay()

$avRET = _ProcessListProperties()
_ArrayDisplay($avRET, "$avRET")

;===============================================================================
; Function Name:    _ProcessListProperties()
; Description:   Get various properties of a process, or all processes
; Call With:       _ProcessListProperties( [$Process [, $sComputer]] )
; Parameter(s):  (optional) $Process - PID or name of a process, default is "" (all)
;          (optional) $sComputer - remote computer to get list from, default is local
; Requirement(s):   AutoIt v3.2.4.9+
; Return Value(s):  On Success - Returns a 2D array of processes, as in ProcessList()
;            with additional columns added:
;            [0][0] - Number of processes listed (can be 0 if no matches found)
;            [1][0] - 1st process name
;            [1][1] - 1st process PID
;            [1][2] - 1st process Parent PID
;            [1][3] - 1st process owner
;            [1][4] - 1st process priority (0 = low, 31 = high)
;            [1][5] - 1st process executable path
;            [1][6] - 1st process CPU usage
;            [1][7] - 1st process memory usage
;            [1][8] - 1st process creation date/time = "MM/DD/YYY hh:mm:ss" (hh = 00 to 23)
;            [1][9] - 1st process command line string
;            ...
;            [n][0] thru [n][9] - last process properties
; On Failure:      Returns array with [0][0] = 0 and sets @Error to non-zero (see code below)
; Author(s):        PsaltyDS at http://www.autoitscript.com/forum
; Date/Version:   06/12/2008  --  v2.0.1
; Notes:            If a numeric PID or string process name is provided and no match is found,
;            then [0][0] = 0 and @error = 0 (not treated as an error, same as ProcessList)
;          This function requires admin permissions to the target computer.
;          All properties come from the Win32_Process class in WMI.
;            To get time-base properties (CPU and Memory usage), a 100ms SWbemRefresher is used.
;===============================================================================
Func _ProcessListProperties($Process = "", $sComputer = ".")
    Local $sUserName, $sMsg, $sUserDomain, $avProcs, $dtmDate
    Local $avProcs[1][2] = [[0, ""]], $n = 1

; Connect to WMI and get process objects
    $oWMI = ObjGet("winmgmts:{impersonationLevel=impersonate,authenticationLevel=pktPrivacy}!\\" & $sComputer & "\root\cimv2")
    If IsObj($oWMI) Then
    ; Get collection of all processes from Win32_Process
        $colProcs = $oWMI.ExecQuery("select * from win32_process")
        If IsObj($colProcs) Then
        ; Return for no matches
            If $colProcs.count = 0 Then Return $avProcs

        ; Size the array
            ReDim $avProcs[$colProcs.count + 1][10]
            $avProcs[0][0] = UBound($avProcs) - 1

        ; For each process...
            For $oProc In $colProcs
            ; [n][0] = Process name
                $avProcs[$n][0] = $oProc.name
            ; [n][1] = Process PID
                $avProcs[$n][1] = $oProc.ProcessId
            ; [n][2] = Parent PID
                $avProcs[$n][2] = $oProc.ParentProcessId
            ; [n][3] = Owner
                If $oProc.GetOwner($sUserName, $sUserDomain) = 0 Then $avProcs[$n][3] = $sUserDomain & "\" & $sUserName
            ; [n][4] = Priority
                $avProcs[$n][4] = $oProc.Priority
            ; [n][5] = Executable path
                $avProcs[$n][5] = $oProc.ExecutablePath
            ; [n][8] = Creation date/time
                $dtmDate = $oProc.CreationDate
                If $dtmDate <> "" Then
                ; Back referencing RegExp pattern from weaponx
                    Local $sRegExpPatt = "\A(\d{4})(\d{2})(\d{2})(\d{2})(\d{2})(\d{2})(?:.*)"
                    $dtmDate = StringRegExpReplace($dtmDate, $sRegExpPatt, "$2/$3/$1 $4:$5:$6")
                EndIf
                $avProcs[$n][8] = $dtmDate
            ; [n][9] = Command line string
                $avProcs[$n][9] = $oProc.CommandLine

            ; increment index
                $n += 1
            Next
        Else
            SetError(2); Error getting process collection from WMI
        EndIf
    ; release the collection object
        $colProcs = 0

    ; Get collection of all processes from Win32_PerfFormattedData_PerfProc_Process
    ; Have to use an SWbemRefresher to pull the collection, or all Perf data will be zeros
        Local $oRefresher = ObjCreate("WbemScripting.SWbemRefresher")
        $colProcs = $oRefresher.AddEnum($oWMI, "Win32_PerfFormattedData_PerfProc_Process" ).objectSet
        $oRefresher.Refresh

    ; Time delay before calling refresher
        Local $iTime = TimerInit()
        Do
            Sleep(20)
        Until TimerDiff($iTime) >= 100
        $oRefresher.Refresh

    ; Get PerfProc data
        For $oProc In $colProcs
        ; Find it in the array
            For $n = 1 To $avProcs[0][0]
                If $avProcs[$n][1] = $oProc.IDProcess Then
                ; [n][6] = CPU usage
                    $avProcs[$n][6] = $oProc.PercentProcessorTime
                ; [n][7] = memory usage
                    $avProcs[$n][7] = $oProc.WorkingSet
                    ExitLoop
                EndIf
            Next
        Next
    Else
        SetError(1); Error connecting to WMI
    EndIf

; Return array
    Return $avProcs
EndFunc  ;==>_ProcessListProperties

Edited June 12, 2008 by PsaltyDS

[weaponx]

Stop it! Both of you!

The RegExp geekiness is making me go blind!

The StringRegExp method shaved off 18 ms for me, thats like 189 milliseconds in dog years.

[duckling78]

This updated function seems to work and not have that memory leak thing. Are you done working on it now?

Anyways thanks again for the function and fix!

[PsaltyDS]

This updated function seems to work and not have that memory leak thing. Are you done working on it now?

Anyways thanks again for the function and fix!

It's only "done" like the dishes, until I have to do it again... but, yeah. Done for now.

I haven't had a chance to try out the remote access part. Seemed to work, or not, based on what two OS's were talking. The WMI security stuff may not be straight yet.

[Valuater]

You guys are nuts!!!

... But, Awesome too!!

8)

[PsaltyDS]

New version 2.0.2 posted at the top of the topic. When ProcessList() was removed, the functionality to find by PID or Name was lost. I put it back in.

[PsaltyDS]

Edited the demo script part in post #1 to show more of the functionality and test more inputs.

muttley

[ludocus]

take a look what I made:

SBA Task Manager

[amokoura]

I found a memory leak (using 3.2.12.0)

while 1
    _ProcessListProperties()
    Sleep(500)
WEnd

I'm just watching task manager and the script process memory usage grows slightly all the time. Or am I getting it wrong?

[amokoura]

EDIT: In previous post I used taskmans "mem usage" column for analyze but I think it's the wrong way. To conclusions below I ended up by watching the process's "VM size" column, which is the right column to monitor mem usage. Maybe.

Possible memory leak reasons:

1. The Process objects GetOwner method

2. Using the $oRefresher thingie

The refresher object seems to be more agressive on memory usage.

I simply isolated the code to small parts and found out those things.

I also googled for WMI problems and found out that "WMI is notorious for memory problems", as someone mentioned

Edited August 4, 2008 by amokoura

[PsaltyDS]

EDIT: In previous post I used taskmans "mem usage" column for analyze but I think it's the wrong way. To conclusions below I ended up by watching the process's "VM size" column, which is the right column to monitor mem usage. Maybe.

Possible memory leak reasons:

1. The Process objects GetOwner method

2. Using the $oRefresher thingie

The refresher object seems to be more agressive on memory usage.

I simply isolated the code to small parts and found out those things.

I also googled for WMI problems and found out that "WMI is notorious for memory problems", as someone mentioned

I see it too, with XP Pro and on Windows 2003. Commenting out the entire section where "the $oRefresher thingie" is used made no difference. I also made sure the $oWMI and $colProcs objects got zeroed out with no change.

I don't know any tricks for this... anybody got any ideas?

[amokoura]

I don't know any tricks for this... anybody got any ideas?

It's probably impossible to fix if the faults are inside WMI. Luckily we can have workarounds

I commented out the parts I mentioned because I didn't need them anyways.

From this forum thread (http://objectmix.com/dotnet/109090-wmi-memory-trouble.html) I found a hint. Maybe trivial.. :

"1. Try to minimize such leaks by trying to reuse the objects if possible.

2. Also try to extract the pertinent data from the WMI objects and quickly dispose of them. Don't hold on to them any longer than necessary.

3. If after doing the above steps have this process shut down after a certain amount of time to be restarted."

Maybe by using global WMI and Refresher objects additional memory would not be wasted. However, the _ProcessListProperties() is currently a really neat and simple package that works inside it's own scope. Making it a some kind of a "system" would be sad

[weaponx]

If WMI is having memory problems, wouldn't the working set of the WMI service grow? This should't effect your program. There must be something else in AutoIt that isn't being closed properly.

NOTE: Commenting out everything after "$colProcs = 0" (the refresher) stopped the leak for me.

Edited August 5, 2008 by weaponx

[amokoura]

If WMI is having memory problems, wouldn't the working set of the WMI service grow? This should't effect your program. There must be something else in AutoIt that isn't being closed properly.

NOTE: Commenting out everything after "$colProcs = 0" (the refresher) stopped the leak for me.

So you're suggesting that AutoIt has some COM object releasing bug?

I'm not an expert in COM but for example if I create an ADO object in AutoIt, the AutoIt process's working set grows (I'm not 100% sure about this but I have an image it happened that way). The same way WMI objects would "attach" into AutoIt process and grow it's working set? And some voodoo here and there. I'm confused.

I also found out that the $oProc.GetOwner($str, $str) leaks. I tested it by calling the _ProcessListProperties() inside an infinite loop with a sleep(100) and noticed that the AutoIt Script was eating memory a little bit all the time. Do you notice it?

(Note: I stripped everything from the function except for the stuff that was needed to use the GetOwner() method).