Autoit sometimes freezes when polling StdOutRead

[mbit90]

I have run into a problem with Run(...,  $STDOUT_CHILD)  and then calling StdOutRead($pid) in a loop to get all data.
I'm starting the same child application over and over, collecting and then evaluating the child's output.
At a certain point Autoit freezes. It appears to be a random deadlock.
The deadlock does not happen when I do ProcessWaitClose($pid) and afterwards StdOutRead($pid). But the helpfile says it's advisable to read STDOUT in a loop to get all data.
Here's the minimum example code. Make sure to watch the console output e.g. in SCiTE. It will stop spewing new output after a few seconds or minutes:

; AutoIt Version: 3.3.14.2
; Script Function: Autoit sometimes deadlocks (stops responding) when using Run() repeatedly with a StdoutRead loop.
; Just leave this running for a certain time, it will freeze eventually.
#include <AutoItConstants.au3>
While True
    $pid = Run('tasklist.exe', @ScriptDir, @SW_HIDE, $STDOUT_CHILD)
    $sOutput = ""
    Do
        $sOutput &= StdoutRead($pid) ; collect new output
    Until @error ; EOF reached
    ProcessWaitClose($pid) ; These two don't make a difference...
    StdioClose($pid)       ; ... you can leave them out
    ConsoleWrite($sOutput)
    Sleep(50)
WEnd

Am I doing something wrong here or is there a bug in AutoIT?

[zeenmakr]

1 hour ago, mbit90 said:

Make sure to watch the console output e.g. in SCiTE. It will stop spewing new output after a few seconds or minutes

the title says 'freezes' did you mean SciTE crashed and became unresponsive which require termination/end task from task manager?

code above only stopped producing console output after about 47seconds, it pretty common to me -still have control over SciTE. need to terminate by heading over to 'tools > stop executing'

i don't know why but try this you'll get the same thing -console write would halt.

While 1
    ConsoleWrite('!>!'&@ScriptLineNumber&':'&$str&' please stop'&@CRLF) ;Read/Brown
WEmd

 

Edited September 25, 2020 by zeenmakr

[Danp2]

The issue appears to be that the launched application doesn't exit, so your code gets stuck in a loop. Change @SW_HIDE to @SW_SHOW in your code and you'll see what I mean.

[zeenmakr]

6 minutes ago, Danp2 said:

The issue appears to be that the launched application doesn't exit, so your code gets stuck in a loop. Change @SW_HIDE to @SW_SHOW in your code and you'll see what I mean.

like this? still freezes up

#include <AutoItConstants.au3>

$hGui = GUICreate('')
GUISetState(@SW_SHOW, $hGui)

While True
    $pid = Run('tasklist.exe', @ScriptDir, @SW_HIDE, $STDOUT_CHILD)
    $sOutput = ""
    Do
        $sOutput &= StdoutRead($pid) ; collect new output
    Until @error ; EOF reached
    ProcessWaitClose($pid) ; These two don't make a difference...
    StdioClose($pid)       ; ... you can leave them out
    ConsoleWrite($sOutput)
    Sleep(50)
WEnd

 

[Danp2]

Nope... those lines didn't exist in your original example. Make the change in your Run command.

[zeenmakr]

1 minute ago, Danp2 said:

Nope... those lines didn't exist in your original example. Make the change in your Run command.

im not the op btw (: 

but from  this

$pid = Run('tasklist.exe', @ScriptDir, @SW_HIDE, $STDOUT_CHILD)

to this, would result in continuous cmd window popingup

$pid = Run('tasklist.exe', @ScriptDir, @SW_SHOW, $STDOUT_CHILD)

 

[mbit90]

6 hours ago, zeenmakr said:

the title says 'freezes' did you mean SciTE crashed and became unresponsive which require termination/end task from task manager?

Nope, not SCiTE freezes, the AutoIT process (the runtime interpreter) freezes. And no crashes. A freeze means the program stops doing anything, it deadlocks, and becomes unresponsive (e.g. the try icon dos not react to anything in this example). You can in fact see the same result without running in SCite but directly. You just don't get the ConsoleWrite() output.

6 hours ago, zeenmakr said:

i don't know why but try this you'll get the same thing -console write would halt.

While 1
    ConsoleWrite('!>!'&@ScriptLineNumber&':'&$str&' please stop'&@CRLF) ;Read/Brown
WEmd

$str is not defined. But when I remove $str, Autoit Crashes after a few minutes with a DialogBox saying "Error allocating memory", which is also strange but surely a different bug?

1 minute ago, Danp2 said:

The issue appears to be that the launched application doesn't exit, so your code gets stuck in a loop. Change @SW_HIDE to @SW_SHOW in your code and you'll see what I mean.

When I tried this, AutoIt froze again, but the last console window that popped up closed. However, tasklist.exe still ran. Terminating tasklist.exe did not un-freeze AutoIt.
Note that the call to ProcessWaitClose($pid) does not cause the issue -- it can be removed from the script with the same result. It's just there for 'best practice' reasons.
It's also not tasklist.exe misbehaving in some way, I tried this with other console applications as well with the same result.

[Danp2]

10 minutes ago, zeenmakr said:

would result in continuous cmd window popingup

Exactly... that way you can observe the program when it fails to exit.

17 minutes ago, mbit90 said:

Terminating tasklist.exe did not un-freeze AutoIt.

From what I can see, after the initial "hang", the launched process will continue to hang each time it is launched. Here's my version of your code --

#include <AutoItConstants.au3>
$x = 1

While True
    $pid = Run('tasklist.exe', @ScriptDir,  @SW_HIDE, $STDOUT_CHILD)

    If @error Then Exit

    $sOutput = ""
    $y = 0
    Do
        $y += 1

        If $y > 500000 Then
            ProcessClose($pid)
            ExitLoop
        EndIf

;       Sleep(100)
        $sOutput &= StdoutRead($pid) ; collect new output
    Until @error ; EOF reached
    ConsoleWrite("$y=" & $y & @CRLF)
    ProcessWaitClose($pid) ; These two don't make a difference...
    StdioClose($pid)       ; ... you can leave them out
    ConsoleWrite("$x=" & $x & @CRLF)
;   ConsoleWrite($sOutput)
    $x += 1
    Sleep(50)
WEnd

And here's the console output from the most recent run --

expandcollapse popup

$y=170008
$x=1
$y=209266
$x=2
$y=206772
$x=3
$y=206552
$x=4
$y=208768
$x=5
$y=201445
$x=6
$y=207141
$x=7
$y=200126
$x=8
$y=204915
$x=9
$y=204160
$x=10
$y=208952
$x=11
$y=208278
$x=12
$y=208166
$x=13
$y=200109
$x=14
$y=208952
$x=15
$y=204353
$x=16
$y=211114
$x=17
$y=203153
$x=18
$y=202704
$x=19
$y=199369
$x=20
$y=208926
$x=21
$y=201927
$x=22
$y=208349
$x=23
$y=211657
$x=24
$y=213872
$x=25
$y=209144
$x=26
$y=211106
$x=27
$y=202671
$x=28
$y=203840
$x=29
$y=206928
$x=30
$y=208421
$x=31
$y=209180
$x=32
$y=201213
$x=33
$y=205221
$x=34
$y=197454
$x=35
$y=202601
$x=36
$y=211849
$x=37
$y=205381
$x=38
$y=198597
$x=39
$y=195326
<snip>
$x=328
$y=214865
$x=329
$y=202048
$x=330
$y=202573
$x=331
$y=212213
$x=332
$y=198373
$x=333
$y=206837
$x=334
$y=500001
$x=335
$y=500001
$x=336
$y=500001
$x=337
$y=500001
$x=338
$y=500001
$x=339
$y=500001
$x=340
$y=500001
$x=341
$y=500001
$x=342
$y=500001
$x=343
$y=500001
$x=344
$y=500001
$x=345
$y=500001
$x=346
$y=500001
$x=347
$y=500001
$x=348
$y=500001
$x=349
$y=500001
$x=350
$y=500001
$x=351
$y=500001
$x=352
$y=500001
$x=353
$y=500001
$x=354
$y=500001
$x=355
$y=500001
$x=356
$y=500001
$x=357
$y=500001
$x=358
$y=500001
$x=359
$y=500001
$x=360
$y=500001
$x=361
$y=500001
$x=362
$y=500001
$x=363
$y=500001
$x=364
$y=500001
$x=365
$y=500001
$x=366
$y=500001
$x=367

 

[TheXman]

I would structure the logic a bit differently than in the original post.  Unless there is a reason to look at the output before the command finishes, then it's much easier to just grab the complete output of the command after the command finishes.  The example below runs the command in a for next loop in order not to have an endless loop.  No matter how many times it loops, it shouldn't freeze.

#include <Constants.au3>

cmd_output_example()

Func cmd_output_example()
    Local $iPID = 0

    For $i = 1 To 15
        ;execute command
        $iPID = Run(@ComSpec & ' /c tasklist.exe', "", @SW_HIDE, $STDERR_MERGED)
        If @error Then Exit MsgBox($MB_ICONERROR, "ERROR", "An error occurred executing command.")

        ;wait for process to end
        ProcessWaitClose($iPID)

        ;Display output
        ConsoleWrite("Tasklist execution #" & $i & @CRLF)
        ConsoleWrite(StdoutRead($iPID) & @CRLF)
    Next
EndFunc

 

Edited September 25, 2020 by TheXman
Change script to display output each loop instead of aggregating it and displaying it once.

[mbit90]

 

12 hours ago, Danp2 said:

From what I can see, after the initial "hang", the launched process will continue to hang each time it is launched. Here's my version of your code --

#include <AutoItConstants.au3>
$x = 1

While True
    $pid = Run('tasklist.exe', @ScriptDir,  @SW_HIDE, $STDOUT_CHILD)

    If @error Then Exit

    $sOutput = ""
    $y = 0
    Do
        $y += 1

        If $y > 500000 Then
            ProcessClose($pid)
            ExitLoop
        EndIf

;       Sleep(100)
        $sOutput &= StdoutRead($pid) ; collect new output
    Until @error ; EOF reached
    ConsoleWrite("$y=" & $y & @CRLF)
    ProcessWaitClose($pid) ; These two don't make a difference...
    StdioClose($pid)       ; ... you can leave them out
    ConsoleWrite("$x=" & $x & @CRLF)
;   ConsoleWrite($sOutput)
    $x += 1
    Sleep(50)
WEnd

And here's the console output from the most recent run --

Hmm... so you inserted a watchdog sort of thing into the loop, in case StdoutRead does not give @error when it should. Nice idea. However, when I run it, it just freezes again instead of giving the $y=500001 outputs. AutoIT does not accumulate any cycles when I check in Process Hacker, meaning it's not stuck in a loop, but an actual deadlock.

 

@TheXman, your example works just fine for most applications, but the point of my original post was to demonstrate that AutoIT freezes when calling StdoutRead in a loop.

Calling StdoutRead in a loop is mentioned as good practice in the help file, so it should not cause the runtime to deadlock.
Furthermore, as you mentioned, there might be reasons to "stream" the output to the host application continuously. I currently build such an application and stumbled upon this bug, then constructed the minimal example to isolate the issue.

I might not have made this clear in my original post, if so, I'm sorry
This was meant to be a bug report, kind of. Is there a better way of reporting this issue to the devs?
I'm still interested in any intermediate workarounds that call StdoutRead repeatedly without freezing though

cheers

[TheXman]

There's no bug.  You just need to tighten up your logic, or should I say, make it a little more robust.  Here's a UDF I wrote to capture cmd output as it happens.  I've used it for years and it has never frozen up on me.  However, I've really never had the need to run it in a tight loop.  But it doesn't seem to have any issue in my example below.  If you are wondering why there is a loop to wait for output before reading the output, it is because it was written for an application that took 1-2 seconds before it started generating output.

expandcollapse popup

#include <Constants.au3>

cmd_output_example()

Func cmd_output_example()
    Local $sOutput = ""

    For $i = 1 To 10
        ;execute command
        $sOutput = _RunCaptureOutput("tasklist.exe", "")
        If @error Then Exit MsgBox($MB_ICONERROR, "ERROR", "An error occurred executing command. @error = " & @error)

        ;Display output
        ConsoleWrite("Tasklist execution #" & $i & @CRLF)
        ConsoleWrite($sOutput & @CRLF)
    Next
EndFunc

;==========================================================================
; Function Name:    _RunCaptureOutput()
; Description:      Execute a command and capture the output
; Parameter(s):     $strCmd, command to be executed
;                   $strWorkingDir, optional, working directory
;                   $iTimeoutSecs, optional, timeout in seconds
; Requirement(s):   None
; Return Value(s):  Success: Output of the executed command
;                   Failure: "" and sets @error (see notes)
; Author(s):        TheXman
; Note(s):          @error = 1, Error executing the RUN function
;                   @error = 2, Process closed before displaying any output
;                   @error = 3, Timed out before displaying any output
;==========================================================================
Func _RunCaptureOutput($strCmd, $strWorkingDir = @WorkingDir, $iTimeoutSecs = 5)
    Local $intPID     = 0
    Local $strOutput  = ""
    Local $hndTimer   = -1

    ;execute command
    $intPID = Run($strCmd, $strWorkingDir, @SW_HIDE, $STDERR_MERGED)

    ;return if error
    If @error Then
        ConsoleWrite("Error: Unable to execute command - " & $strCmd & @CRLF)
        Return SetError(1, 0, "")
    EndIf

    ; loop until there is data to display, process closes, or timed out
    $hndTimer = TimerInit()
    While Not StdoutRead($intPID, True)
        ;process still exists
        If ProcessExists($intPID) Then
            ;sleep
            Sleep(250)
        Else
            ;display warning and return
            ConsoleWrite("Warning: Process closed before displaying any output" & @CRLF)
            Return SetError(2,0,"")
        EndIf

        ;If timeout has been reached
        If TimerDiff($hndTimer) > $iTimeoutSecs * 1000 Then
            ;display warning and return
            ConsoleWrite("Warning: Process timed out waiting for output" & @CRLF)
            Return SetError(3,0,"")
        EndIf
    WEnd

    ;loop while there is data to display
    While StdoutRead($intPID, True)
        ;display stdout data
        $strOutput &= StdoutRead($intPID)

        ;sleep for .5 secs
        Sleep(500)
    WEnd

    ;return output
    Return $strOutput
EndFunc

 

Edited September 26, 2020 by TheXman

[mbit90]

I don't get the full Tasklist output with the Sleeps removed - code like this must not rely on getting the Sleep timings just right.
You can't just assume that when StdoutRead returns an empty string once, that the output is finished. More output may come later. Of course it works with tasklist followed by 500ms of sleeping - in the common case.
The only (intended) reliable indication for end-of-stream seems to be @error after StdoutRead, or am I missing something?

[TheXman]

I disagree with your assertion regarding timing.  I also don't get any truncation of the tasklist output.  The stdout stays in the buffer until read so I don't know why you aren't seeing the full tasklist output.  My tasklist is pretty large and I get the whole thing each time thru the loops.

My UDF was written to handle the output of the application I was dealing with.  Once it started writing to stdout, it continued until it was done.  If you need something else, then I have given you a pretty good platform to jump off from.

Edited September 26, 2020 by TheXman

[mbit90]

Well, assume you want the reading of stdout to be fast, then you could not use Sleeping, but you would have to rely only on Locks and Busy waiting. Looking at this from a C programmer perspective.

i get the truncation of the tasklist output only if I remove the Sleeps from your code, and it's perfectly clear why. The last loop...

37 minutes ago, TheXman said:

;loop while there is data to display     While StdoutRead($intPID, True)         ;display stdout data         $strOutput &= StdoutRead($intPID)         ;sleep for .5 secs         Sleep(500)     WEnd

... without the Sleep will collect some initial output into the string, but in the second iteration, StdoutRead returns "" (most likely) and the loop exits (perhaps) without the remainder of output that tasklist has yet to print. It depends on the timing between the two processes and thus has undefined behaviour.

[TheXman]

You could make plenty other changes to the code to make it not work too, so what?  It has a sleep in it because that's the way I wrote it and it worked perfectly for my needs.  AutoIt isn't C so I don't care to look at it from a C-perspective.  If it needs to run like a C application, then write it in C.  I'm showing you how it can be done in AutoIt.  You are starting to try my patience.  So I will just leave it to someone else to entertain your observations, assertions, arguments.  Have a good evening or day, where ever you may be.

[mbit90]

Yep my needs are clearly different. I'm just trying to stick to the topic, which is about the very simple example code in my first post, and that it should be working according to the help file, while it causes autoit to deadlock.

Good night to you as well.

[Dan_555]

Guys, this is an auto'it forum where people try to help others. (at least that is my perspective).

Please concentrate on problem-solving things, not on personal viewpoints.

I'v made few tests with the code.

The code from the first post is freezing somewhere in between 45 and 200 rounds.

I'v tested the example code from TheXman, and i stopped it after around 700 rounds, because it did not freezed. (i changed the for loop into a while loop )

Then i took the code from the 1st post and experimented a bit with it.

As you can see, i tried few things: (some things are in other posts  too)

expandcollapse popup

#include <GuiEdit.au3>
#include <GuiConstantsEx.au3>
#include <WindowsConstants.au3>

    Global $hGUI = GUICreate("CMD", 500, 400, -1, -1, BitOR($GUI_SS_DEFAULT_GUI, $WS_SIZEBOX, $WS_THICKFRAME), BitOR($WS_EX_ACCEPTFILES, $WS_EX_WINDOWEDGE))
    Global $g_idMemo = GUICtrlCreateEdit("", 0, 31, 499, 349)

    GUISetState(@SW_SHOW)

Local $i=0,$j=0, $k=0
While True
    $k=0
    $j=$j+1
    $pid = Run('tasklist.exe', @ScriptDir, @SW_HIDE, $STDOUT_CHILD)
    sleep (200)
    $sOutput = ""
    ;If ProcessExists($pid) Then
        $i=$i+1
        WinSetTitle ($hGUI,"",$i & "/" & $j & "/" & $pid)
        Do
            $k=$k+1
            $sOutput &= StdoutRead($pid) ; collect new output
            if $k>540000 then
                ConsoleWrite ("emergency exit" & @CRLF)
                ExitLoop
            EndIf
        Until @error ; EOF reached
        ;StdioClose($pid)   ; ... you can leave them out
        ;ProcessWaitClose($pid) ; These two don't make a difference...
        MemoWrite("",1)
        MemoWrite($sOutput)
    ;EndIf
    Sleep(50)
WEnd


Func MemoWrite($sMessage = "", $clr = 0)
    Local $CRLF = ""

    If $clr = 1 Then
        GUICtrlSetData($g_idMemo, "")
    Else
        $CRLF = @CRLF
    EndIf

    GUICtrlSetData($g_idMemo, $sMessage & $CRLF, 0)
EndFunc   ;==>MemoWrite

P.S. i'v added a gui and an edit field, where the output is written.

This code is running, at the moment of writing, with 1500 loops without freezing.

The important part here is the Sleep (200) just after the Run('tasklist.exe") line 

(edit #5: i'v just tried out to increase the sleep(50) and the freezing have started (not using the sleep (200) line), so it is not up to this)

If you lower it, the freezing may start happening  - the lower the number - the sooner the freezes.

The emergency exit has not been reached in the normal run, but only with the lowered sleep amount.

If a freeze (and it is not the script which is freezing, just the capturing of the output with the StdOutRead) and this, @DanP2, has allready mentioned: All consecutive run calls will fail to capture the output.

Why is this happening - IDK, maybe a bug, but the question is, is it a bug in autoIt or an windows bug ... 

(currently the loop has reached 4200 (after few edits) as it is running in the background).

Maybe someone wants to open a ticket in the bugtracker, so that the developers may look into it (if it is a bug at all) ?

 

Edited September 26, 2020 by Dan_555