StdoutRead Memory Problem (EDIT: and StdioClose)

[ferbfletcher]

Windows 7 64 bit.
Running AutoIt in x86.
Autoit 3.3.10.2

I am monitoring the output of a process.  If the output is empty, I close it.  That works fine.

However, I think the constant reading stdout is causing the memory of the autoit process to continually increase.  After about 12 hours, it goes from 25M up to 100M of memory usage.



As you can see, I'm not keeping the contents of stdoutread, but it still must be accumulating all of the stdout in memory?  Is there anyway to fix this?

Thank you.

$pid = Run("mycommand.exe","",@SW_HIDE,$STDERR_MERGED)  ;Read from both the stdout and the stderr

While 1
     If StringLen(StdoutRead($pid)) = 0 Then
          ProcessClose($pid)
     EndIf
     Sleep(10)
Wend

 

 

Edited October 3, 2015 by ferbfletcher
Edit Title

[JohnOne]

Are you going to tell us, 20 posts down the line that, that is not all of your code?

[ferbfletcher]

Are you going to tell us, 20 posts down the line that, that is not all of your code?

That is not all of the code, this is just the part I am concerned about.  If I remove "STDERR_MERGED" from the Run command, it no longer has the large memory usage. So, I have narrowed it down to that.

 

[MHz]

I am curious, how do you get out of that While loop? I see no break out of loop. The solution seems to me is to check @error from StdOutRead() in which to break out of the loop and then progress with exitloop.

Edited September 29, 2015 by MHz
underlined the exitloop as a way to exit the loop

[ferbfletcher]

I am curious, how do you get out of that While loop? I see no break out of loop. The solution seems to me is to check @error from StdOutRead() in which to break out of the loop and then progress with exitloop.

The entire code is much larger, but the part that seems to have the issue is just this part where it reads StdOut.  I've been watching it, doing some testing, and this is what seems to be happening:

StdoutRead will read everything in the $pid stdio buffer.  At this point, the memory goes up (obviously, because it is getting data).  Then, StdoutRead should immediately dump the data because nothing is being done with it (just reading the StringLen).  At this point, the memory goes back down (obviously, because it is releasing that stdio data).

However, when the memory goes back down, it doesn't go all the way back down to the original level.  So, it goes up, comes down most of the way but not all the way, then it goes up, then comes down most of the way but not all the way.  Eventually, this builds up to a very high memory usage.

If the sending program ($pid being read) doesn't send as much output to stdio, the memory leak goes up slower, and if I remove STDERR_MERGED so it can't read stdio at all, the memory doesn't go up at all.  Both of those things tend to prove a memory leak inside StdOutRead.


 

[TheDcoder]

I need some information to make my hypothesis:

1. Did you try your code with latest version of AutoIt?

2. How much output does "mycommand.exe" produce in 1 hour? Does the rate of output remains constant? If not, give me the peak rate of the output.

 

TD

[jvanegmond]

I made a little reproducer but couldn't get AutoIt (3.3.14.2) process memory above 3 megabytes after a run of 30 minutes, so a short test says it is unlikely to be the problem. However, you never know what happens if you run something for a longer period of time. 25 megabytes total for an AutoIt process seems a bit much in the first place though. What else is that script doing?

If the sending program ($pid being read) doesn't send as much output to stdio, the memory leak goes up slower, and if I remove STDERR_MERGED so it can't read stdio at all, the memory doesn't go up at all.  Both of those things tend to prove a memory leak inside StdOutRead.

The code, as you posted it, kills the process if it has not produced any output for a period of 10 milliseconds. If you remove STDERR_MERGED, the process you are running produces less data in STDOUT because it no longer contains the error stream. This can result in the process being killed more often. If the memory leak is in the part of cleaning up the process or restarting it (do you have such a mechanism?) that can influence the results (though you'd expect the opposite behavior). I would focus my efforts there, first upgrading AutoIt (just in case!) and then for example by making a test where you continuously stop and start the process, since a short test says that StdoutRead is probably not leaking memory.

Edited September 30, 2015 by jvanegmond

[ferbfletcher]

I'm using 3.3.10.2.  There is nothing in the changelog regarding stdio between that version and the newest version.  Regardless, I will try the newest version as you suggest just to make sure there isn't an issue there.  I will report back here what happens with the newest version.
 

[ferbfletcher]

As expected, there was no change between 3.3.10.2 and the newest version, as there were no mentions of this in the changelog.

Someone had asked what the amount of output from mycommand.exe is in one hour. 

It is extremely large.  This is the trace log output for ffmpeg's ffplay.exe program. It is about 150,000 characters every 3 seconds, and it is actually monitoring 12 programs ($pid1, $pid2, etc).

So, the amount of characters in one hour would be 150000/3*60 = 3 million per minute per program monitored * 60 minutes = 180 million per program monitored per hour * 12 programs = 2 billion characters per hour.

The reason for monitoring ffplay.exe is that ffplay cannot determine what a live stream connection is disconnected.  It just keeps playing the last frame received forever, waiting for more video to arrive, which will never happen until it is restarted.  The trace output stops during this time, so that is the only way to determine when the stream is broken.  The trace output is enormous, though.

As another experiment, I have recompiled ffplay.exe (that is mycommand.exe) to only allow it to collect trace for 1 second every 10 seconds, and each trace output is limited to 2 characters during that time.  That will cut the output down by about 99%.  So, I'm thinking that now, the leak will increase at only 1% of the rate it did before, or not at all if the leak was only due to having such a large input.

I'll let this run for a while and see what happens.

 

[TheDcoder]

@ferbfletcher Ah, I made my hypothesis: 

 

StdoutRead is not leaking memory, It needs to store all the output of 2 billion characters per hour, multiplied by 12 gives you 24 billion characters... when you call StdoutRead needs to store all the characters internally to return them when called...

 

Since you have recompiled it, lets see what happens...

[jvanegmond]

StdoutRead has a limit to the maximum number of characters returned. Suppose this limit is 2048 hypothetically (I don't know what it is really but 2048 seems reasonable and is found throughout AutoIt), and we only run StdoutRead at a maximum rate of 100x per second because of the 10ms sleep in between, that gives us a maximum of 2048*100 bytes per second which is ~205kb/s. At 2 billion characters per hour in the input stream, that's roughly 4800kb/s. So taking that into account, your standard stream is probably filling up around 20-25 times faster than you are reading from it, which explains the insane memory usage.

What happens if you trigger the scenario you are trying to prevent, where the process does not produce output? Don't just force close the process, that might cause AutoIt to discard some of the buffer and causes StdoutRead to return empty. What happens to memory in that time? Or does memory usage gradually decrease and thén it triggers the forced close?

You probably should have written something like this in the first place:

$pid = Run("mycommand.exe","",@SW_HIDE,$STDERR_MERGED)  ;Read from both the stdout and the stderr
$lastData = TimerInit()

Sleep(100) ; Wait for process to start producing output
AdlibRegister("CheckLastStreamOutput", 10)

While 1
    If StringLen(StdoutRead($pid)) == 0 Then
        Sleep(1)
    EndIf
    $lastData = TimerInit()
 Wend
 
Func CheckLastStreamOutput()
    If TimerDiff($lastData) > 10 Then
        ProcessClose($pid)
    EndIf
EndFunc

But in any case your change to reduce the input by 99% probably already fixes the issue.

Edited October 1, 2015 by jvanegmond

[ferbfletcher]

Yes, that did help.  I think it is still slightly increasing, but I would need to wait a long time to know for sure.

Regardless, this code that I found elsewhere in the forum seems to fix everything, no matter how bad it leaks.  Just run this periodically and it goes WAY down (to like 3mb)

Func _ReduceMemory()
    Local $aReturn = DllCall("psapi.dll", "int", "EmptyWorkingSet", "long", -1)
    If @error = 1 Then
        Return SetError(1, 0, 0)
    EndIf
    Return $aReturn[0]
EndFunc   ;==>_ReduceMemory

So far, I don't see any downside to running this function every so often (every minute or every hour or whatever).  I'll probably throw it in all my scripts, just to prevent any unknown problems or poor coding from eventually crashing a computer.

 

 

[BrewManNH]

Bad idea to run this snippet.

It is not reducing the memory used by the program, it is reducing the physical memory used, and dumping it onto the hard drive into your page file. Your page file is probably only as large as the amount of memory in the computer, eventually you will run out of page file as well. Not to mention, hard drives are much much slower than memory chips, even SSD drives can't compete, so anything that you will use that data for is going to take longer to access.

This snippet was a better idea when people used XP, and didn't know what they were doing, and thought having tons of free memory meant something. Since Vista and above, the OS is going to use as much of your free memory as possible instead of a page file, because memory is so much faster. Let Windows handle your memory usage, they know what they're doing.

[ferbfletcher]

Bad idea to run this snippet.

It is not reducing the memory used by the program, it is reducing the physical memory used, and dumping it onto the hard drive into your page file. Your page file is probably only as large as the amount of memory in the computer, eventually you will run out of page file as well. Not to mention, hard drives are much much slower than memory chips, even SSD drives can't compete, so anything that you will use that data for is going to take longer to access.

This snippet was a better idea when people used XP, and didn't know what they were doing, and thought having tons of free memory meant something. Since Vista and above, the OS is going to use as much of your free memory as possible instead of a page file, because memory is so much faster. Let Windows handle your memory usage, they know what they're doing.

Doesn't Windows Task Manager show the total amount of memory that a process is using?  Regardless of whether that memory is physical or pagefile?  If the Windows Task Manager is showing that my program is using 10MB of memory, that is the total of the amount of physical + pagefile that my program is using, right?

If so, then Task Manager should still show a huge amount of memory being used, if that snippet was just dumping to page file as you say.  However, task manager does NOT show the large amount of memory being used after running this snippet.  It shows only a tiny amount of memory being used after running this snippet.

That leads me to believe that it is truly freeing that memory and not just pushing it to pagefile.

Please tell me if I am misunderstanding this.

 

[ferbfletcher]

Ok, I just read up on MSDN about this function.  It DOES, in fact, REMOVE unused memory and does NOT push it to hard drive.

You might be mixing it up with another function that does do what you mentioned.

"SetProcessWorkingSetSize" will push memory to hard drive, because it limits the amount of memory usage.  That is not this function listed above.

"EmptyWorkingSet" is an API  from Microsoft that ACTUALLY REMOVES unused memory, not push it to hard drive.



 

[BrewManNH]

I may have mistaken it for the other API function call, but I'm betting you're still going to see performance issues.

Leave it to Windows to decide what memory is needed, you're not fixing the problem you're just putting a coat of paint over it and pretending it's not there.

[kylomas]

furbfletcher,

Let me see if I understand this.  You are starting a program.  Then reading the output stream every 10 miliseconds until it no longer produces output.  Then you are terminating it.  You've indicated that it runs several hours so presumably it is producing output the entire time.  So it is reasonable to deduce that your purpose is solely to determine when the said program stops producing output.  Is all this true?

I agree with BrewmanNH that "EmptyWorkingSet" is just a stop gag for a larger problem.  If you could create a reproducer in the context of your "larger" program it would assist problem determination.

kylomas 

 

[ferbfletcher]

furbfletcher,

Let me see if I understand this.  You are starting a program.  Then reading the output stream every 10 miliseconds until it no longer produces output.  Then you are terminating it.  You've indicated that it runs several hours so presumably it is producing output the entire time.  So it is reasonable to deduce that your purpose is solely to determine when the said program stops producing output.  Is all this true?

I agree with BrewmanNH that "EmptyWorkingSet" is just a stop gag for a larger problem.  If you could create a reproducer in the context of your "larger" program it would assist problem determination.

kylomas 

 

You are correct, however, it turns out that I found an additional source of memory leak while doing some testing of this one.

While I was able to eliminate the leak from the first issue (without using the "EmptyWorkingSet"), I have so far been completely unable to eliminate the leak from the other issue I ran across without using EmptyWorkingSet.

If you run the following code exactly as-is, and watch Task Manager, you will see that the memory increases every second.  Some times it goes down, but it always goes up more than it goes down.  Again, this is unrelated to the first problem, but this thread has kindof evolved from the initial problem to now whether or not the EmptyWorkingSet snippet is good or not.

This code opens an exe with stdio redirection, then it closes the stdio, then it closes the exe.  That causes the memory to increase above what it was before the exe was ran.  Everytime this loops, memory increases more.
 

#include <Constants.au3> ;This allows the use of the $STDERR_MERGED variable
HotKeySet("{ESC}","MyExit")  ; This lets you easily get out of the While loop by using the ESC key

While 1
    $pid = Run("notepad.exe","",@SW_SHOW,$STDERR_MERGED)  ;It doesn't matter what program you run here.
    ProcessWait($pid) ;Wait until the exe starts
    StdioClose($pid) ;Close the StdIO to the exe to release the handles and memory <-- This appears to have a memory leak.  It doesn't release memory back to original amount.
    ProcessClose($pid) ;Close the exe
    ProcessWaitClose($pid) ;Wait until the exe is actually closed
    Sleep(1000) ;Wait 1 second, it doesn't really matter how long you wait here.
WEnd
Func MyExit() ;Exit the program
    Exit
EndFunc

I

Edited October 3, 2015 by ferbfletcher

[Jos]

This seems to be much better and doesn't increase memory like your example:

#include <Constants.au3> ;This allows the use of the $STDERR_MERGED variable
HotKeySet("{ESC}","MyExit")  ; This lets you easily get out of the While loop by using the ESC key

While 1
    $pid = Run("notepad.exe","",@SW_SHOW,$STDERR_MERGED)  ;It doesn't matter what program you run here.
    ProcessWait($pid) ;Wait until the exe starts
    ProcessClose($pid) ;Close the exe
    ProcessWaitClose($pid) ;Wait until the exe is actually closed
    StdioClose($pid) ;Close the StdIO to the exe to release the handles and memory 
    Sleep(1000) ;Wait 1 second, it doesn't really matter how long you wait here.
WEnd
Func MyExit() ;Exit the program
    Exit
EndFunc

 Jos

Edited October 3, 2015 by Jos

[Gianni]

<snip>
While I was able to eliminate the leak from the first issue (without using the "EmptyWorkingSet"), .....
<snip>

How did you do?