ProcessExists stops working?

[therks]

Is there any reason that ProcessExists would start returning false on a process that still .. exists?

I'm having an issue that I'm so far unable to reproduce reliably, so this is more a general question/advice thread. I have a rather elaborate script running and interacting with a server application. Because the server can crash, one of the purposes of my script is to relaunch the server if it stops.

I'm accomplishing this by storing the PID whenever I Run() the server, and I have an if statement with ProcessExists() in a loop to relaunch. This is a snippet:

While 1
    If Not ProcessExists($I_PID) Then
        _LogWrite('Process lost: ' & $I_PID)

        If TimerDiff($iRelaunchTimer) < 5000 Then
            $iRelaunchCount += 1
        Else
            $iRelaunchCount = 1
        EndIf

        If $iRelaunchCount <= 5 Then
            _LogWrite('Relaunching... (attempt ' & $iRelaunchCount & ')')
            _Launch()
        Else
            Local $sRelaunchExceeded = 'Relaunch looped ' & $iRelaunchCount & ' times in ' & Round(TimerDiff($iRelaunchTimer)/1000) & ' seconds. Check that server is not already running. Exiting.'
            _LogWrite($sRelaunchExceeded)
            MsgBox(0x10, $APP_NAME, $sRelaunchExceeded)
            ExitLoop
        EndIf
        $iRelaunchTimer = TimerInit()
    Else
        ; Do a bunch of other stuff
    EndIf
WEnd

Func _Launch()
    Global $I_PID = Run($SERVER_CMD, $SERVER_DIR, @SW_HIDE, $STDERR_MERGED)
    If @error Then
        MsgBox(0x10, $APP_NAME, 'Error running command:' & @LF & $SERVER_CMD & @LF & @LF & 'In directory:' & @LF & $SERVER_DIR) ; OK: 1
        Exit 600
    EndIf
    _LogWrite('Server launched (PID:' & $I_PID & ').')
    IniWrite($INI_FILE, 'Config', 'PID', $I_PID)
EndFunc

That's not really runnable, but you get the general idea. As I suggested above, the issue I'm experiencing is that sometimes ProcessExists returns false even though the process does still exist (getting the PID from the log, and checking task manager I can see it's still running with the same PID), and the server won't relaunch if it's already running. And the major problem I'm having with diagnosing this is that it happens completely intermittently. It could go for days just fine, or only hours (it's never quick though of course). The server runs on our media computer all the time and the computer and server sometimes go for a few days without being checked on, but ideally we'd like it running all the time.

Anyway, I'm stumped, so any advice on offer will be gratefully accepted.

Edited April 2, 2018 by therks

[Bowmore]

Instead of using ProcessExists() try using ProcessWait() with a suitable time-out value

[therks]

Hmm, I'm not sure if that will have any different results, but I'll give it a try. Thanks. 

[therks]

Well it took several days but it still managed to fail in the same way. It's reacting as if the processes don't exist, even when freshly created.

Like, the code can literally be shrunken down to:

While 1
    Global $I_PID = Run('whatever.exe')
    ConsoleWrite('Server launched (PID:' & $I_PID & ').')
    If Not ProcessWait($I_PID, 1) Then
        ConsoleWrite('Process lost: ' & $I_PID)
    Else
        ExitLoop
    EndIf
WEnd

My console looks like this:

Quote

Server launched (PID:5628).
Process lost: 5628
Server launched (PID:11240).
Process lost: 11240
Server launched (PID:8680).
Process lost: 8680
Server launched (PID:10968).
Process lost: 10968

And if I check task manager, all those processes DO exist.

[Jos]

Actually the whole code looks flawed and can't generate the shown output unless there is more code to it.
So, simply post a proper replicator script please!

Jos

Edited April 9, 2018 by Jos

[therks]

Ok, so I just realized that ProcessWait() isn't supposed to accept a PID (although it does), so that code is technically flawed but I threw that together as a quick example (without testing; my mistake).

Here's an example (that I tested) that is more or less what my original script is doing:

#include <AutoItConstants.au3>

Global $PID

_Launch()
_Main()

Func _Main()
    Local $iRelaunchTrack, $iRelaunchMax = 5
    Local $sStdout

    While 1
        If Not ProcessExists($PID) Then
            ConsoleWrite('Process lost (' & $PID & ')' & @LF)
            If $iRelaunchTrack < $iRelaunchMax Then
                $iRelaunchTrack += 1
                ConsoleWrite('Relaunch attempt #' & $iRelaunchTrack & @LF)
                _Launch()
            Else
                ConsoleWrite('Relaunch has been attempted over ' & $iRelaunchTrack & ' times. Exiting for safety.')
                ExitLoop
            EndIf
        Else
            $sStdout = StdoutRead($PID)
            If Not @error And $sStdout Then
                ConsoleWrite('<Server> ' & $sStdout & @LF)
            EndIf
        EndIf
    WEnd
EndFunc

Func _Launch()
    Global $PID = Run(@ComSpec, @MyDocumentsDir, @SW_SHOW, BitOR($STDIN_CHILD, $STDERR_MERGED))
    ConsoleWrite('Process has been launched (' & $PID & ')' & @LF)
EndFunc

(Obviously in actual use I'm running the server application, not @ComSpec.)

The script runs fine, until suddenly ProcessExists() starts returning false, even though the process still exists (I can see the matching PID in task manager). The main problem is there's no consistency to when this error starts happening. The server and my script run fine for several hours (or days), and then suddenly the script can no longer see the process and starts trying to relaunch the server (which in my case would fail anyway because the server is still running and the ports are inaccessible). Also, as I said above, once it hits this point ProcessExists() doesn't see the new processes either.

[BrewManNH]

I can't duplicate the problem, but I rewrote your code to avoid the Globals that you were using. Plus, you redeclared the $PID variable inside the function, which wasn't needed or desired.

#include <AutoItConstants.au3>

_Main()

Func _Main()
    Local $iRelaunchTrack, $iRelaunchMax = 5
    Local $sStdout
    Local $PID = _Launch()

    While 1
        If Not ProcessExists($PID) Then
            ConsoleWrite('Process lost (' & $PID & ')' & @LF)
            If $iRelaunchTrack < $iRelaunchMax Then
                $iRelaunchTrack += 1
                ConsoleWrite('Relaunch attempt #' & $iRelaunchTrack & @LF)
                $PID = _Launch()
            Else
                ConsoleWrite('Relaunch has been attempted over ' & $iRelaunchTrack & ' times. Exiting for safety.' & @CRLF)
                ExitLoop
            EndIf
        Else
            $sStdout = StdoutRead($PID)
            If Not @error And $sStdout Then
                ConsoleWrite('<Server> ' & $sStdout & @LF)
            EndIf
        EndIf
    WEnd
EndFunc   ;==>_Main

Func _Launch()
    $Temp = Run(@ComSpec, @MyDocumentsDir, @SW_SHOW, BitOR($STDIN_CHILD, $STDERR_MERGED))
    ConsoleWrite('Process has been launched (' & $Temp & ')' & @LF)
    Return $Temp
EndFunc   ;==>_Launch

 

[therks]

Oh, thank you, but it's just a repro script. My original script uses that global in a registered exit function that ensures the process has been shutdown.

It's more like this (I wrote this as the repro first, but I figured it was unnecessarily large)

expandcollapse popup

#include <AutoItConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <GUIEdit.au3>

Global $PID, $ED_LOG

OnAutoItExitRegister(_OnExit)

_Launch()
_Main()

Func _Main()
    Local $iRelaunchTrack, $iRelaunchMax = 5, $iGUIGetMsg
    Local $sStdout

    Local $hGUI = GUICreate('Server watch', 300, 300, Default, Default, $WS_OVERLAPPEDWINDOW)
    $ED_LOG = GUICtrlCreateEdit('', 0, 0, 300, 300)
        GUICtrlSetResizing(-1, $GUI_DOCKBORDERS)
    GUISetState()

    While 1
        $iGUIGetMsg = GUIGetMsg()
        If $iGUIGetMsg = $GUI_EVENT_CLOSE Then ExitLoop

        If Not ProcessExists($PID) Then
            _Log('Process lost (' & $PID & ')')
            If $iRelaunchTrack < $iRelaunchMax Then
                $iRelaunchTrack += 1
                _Log('========================================')
                _Log('Relaunch attempt #' & $iRelaunchTrack)
                _Launch()
            Else
                _Log('Relaunch has been attempted over ' & $iRelaunchTrack & ' times. Exiting for safety.')
                ExitLoop
            EndIf
        Else
            $sStdout = StdoutRead($PID)
            If Not @error And $sStdout Then
                _Log('<Server> ' & $sStdout)
            EndIf
        EndIf
    WEnd
EndFunc

Func _Launch()
    $PID = Run(@ComSpec, @MyDocumentsDir, @SW_SHOW, BitOR($STDIN_CHILD, $STDERR_MERGED))
    _Log('Process has been launched (' & $PID & ')')
EndFunc

Func _OnExit()
    If ProcessExists($PID) Then
        _Log('Exiting process.')
        StdinWrite($PID, 'exit' & @LF)
        If Not ProcessWaitClose($PID, 5) Then
            _Log('Process still running. Forcing closed.')
            ProcessClose($PID)
        EndIf
    EndIf
EndFunc

Func _Log($sMsg)
    _GUICtrlEdit_AppendText($ED_LOG, $sMsg & @CRLF)
    ConsoleWrite($sMsg & @CRLF)
EndFunc

 

[Earthshine]

maybe the error is in the Server code, because, you know, the client can't talk to it anymore, becomes unresponsive probably, maybe in a loop? What is the server written in?

I ran your earlier test code quite a while on 2008R2 and it worked as expected.

Edited April 9, 2018 by Earthshine

[therks]

I had actually thought of that as well, and I would be satisfied to chalk it up to that, except for the fact that new processes fail to be recognized as well.

Like, while trying to debug this, I rewrote my script so it would prompt me before trying to relaunch. So I actually killed the server manually, then let my script relaunch the server, and the server starts up and runs fine, but the script immediately says the process doesn't exist. The only thing that works is closing and relaunching my script, then everything works fine.

If we can't figure this out here, I'm thinking what I might end up doing is when ProcessExists fails:

• Write the "lost" PID to INI file
• Terminate/Relaunch script
• Read PID from INI
• Kill process
• Relaunch server

I even have to kill the process after the relaunch because ProcessClose() stops working as well. I would have done this already, but it just seems like such a clunky way to do things.

[Earthshine]

have you tried to rule out your PC as the problem? Check the HD and run a memory test? this is very strange. could even be a memory issue as I have used ProcessExist forever now and it works great.

Edited April 9, 2018 by Earthshine

[JohnOne]

What if you use the process name rather that pid?

[therks]

@Earthshine: I must admit I haven't tried that. The PC runs fine otherwise, so it didn't seem like it would be the issue. We use it for managing media, downloads, games etc and it hasn't had any issues (besides this server having to be restarted manually now and then).

@JohnOne I'm not sure that would work as it's a java based server (so java.exe) and we sometimes play Minecraft on that computer as well (so also uses java.exe) so I'm sure the script would confuse the two.

Edited April 9, 2018 by therks

[JohnOne]

You are monitoring more than one process in your script proper?

[therks]

Nope, just the one. But if I were to monitor for "java.exe" then when one of us runs any other Java application the script may think that is the server.

[BrewManNH]

1 hour ago, therks said:

My original script uses that global in a registered exit function that ensures the process has been shutdown.

You can still do it without the Global by not using the OnAutoItExit function and just calling it from inside your _Main function when the loop is exited.

[JohnOne]

Try testing for the presence of the pid using ProcessList instead of ProcessExists.

All sounds a bit spooky to me though.

[therks]

So I just added this to my actual script:

_LogWrite('RUNNING DEBUG TESTS:')
_LogWrite('ProcessGetStats(' & $I_PID & ', 0)')
Local $aProcStats = ProcessGetStats($I_PID, 0)
If @error Then
    _LogWrite('Error: ' & @error)
Else
    _LogWrite($aProcStats[0])
    _LogWrite($aProcStats[1])
EndIf

_LogWrite('ProcessGetStats(' & $I_PID & ', 1)')
Local $aProcStats = ProcessGetStats($I_PID, 1)
If @error Then
    _LogWrite('Error: ' & @error)
Else
    For $i = 0 To 5
        _LogWrite($aProcStats[$i])
    Next
EndIf

Local $aProcList = ProcessList()
_LogWrite('ProcessList(): ' & $aProcList[0][0] & ' results')
_LogWrite('  PID PE:PID PE:Name Name')
For $i = 1 to $aProcList[0][0]
    _LogWrite(StringFormat('%5d %6d %7d %s', $aProcList[$i][1], ProcessExists($aProcList[$i][1]), ProcessExists($aProcList[$i][0]), $aProcList[$i][0]))
Next

Maybe I'll get some new insights heh. Unfortunately with the sporadic nature of this bug, I probably won't be back here for a week.

Also, I'm going to try running it on another computer as well, to try and see if it's just a problem with the PC like @Earthshine suggested.

Wish me luck!

Edited April 9, 2018 by therks

[JohnOne]

Func _ProcessExixts($pid)
    $aProcessList = ProcessList("java.exe")
    For $i = 1 To $aProcessList[0][0]
        if $aProcessList[$i][1] = $pid Then
            Return True
        EndIf    
    Next
    Return False
EndFunc

 

Edited April 9, 2018 by JohnOne

[therks]

Oh yeesh, I didn't even think about doing it like that. Thanks for the advice. I'll give that a try depending on how my latest tests turn out.

On a semi-related note, there's no way to attach AutoIt to a process that's already running and get the output, right? Like, in my program I'm Run()'ing the server with $STDERR_MERGED because I'm reading output. So if I have to restart my AutoIt script, I can't re-attach to the existing process and continue StdoutRead()'ing from it, right? I have to kill and restart the server? I'm assuming so, it sounds like it would be a security issue.