Function priority, AdLib, reentrance, and mutexes

[toasterking]

This will be a two-part quest for sanity.

We know that AutoIt's execution environment is single-threaded and only one command can be processed at any given time. We have event-based tricks (GUIOnEvents, DLL callbacks, AdLibRegister, etc.) that can interrupt the currently executing code to execute a specified function instead, then return control to the original procedure, which helps create the illusion of parallel threads. My issue is that the order and priority with which these functions are executed can be unpredictable in some circumstances (which is an issue in true multi-threader environments as well) but I cannot find any documentation on how it is supposed to be handled in AutoIt. This is what I can surmise so far by my own experimentation:

• Execution of any user-related function, or the main loop, can be interrupted to run any other user-related function. Control is returned to the first user-related function when the second finishes.
• Execution of a user-related function can also be interrupted to permit itself to be executed (i.e. user-related functions are reentrant). AutoIt seems to have built-in rules to prevent AdLib from calling a function that hasn't returned from a previous AdLib call, or an OnEvent from doing the same, but the calls can be mixed, i.e. function _DoStuff() can be called from the main loop, then AdLib can call a second iteration of function _DoStuff() (possibly with different parameters) which interrupts the first instance. When the second instance finishes, the first instance continues from where it was interrupted.
• Sometimes a function called by a DLL callback or ObjectEvent can interrupt an AdLib-called function (or vice versa), and sometimes it waits until the first function finishes. I cannot determine any consistency to the system of priorities that are enforced here, if indeed there are any.
• User-related functions invoked by DLL callbacks, AdLib, or other events can interrupt execution of some built-in AutoIt functions (e.g. Sleep()) but not others (e.g. FileCopy(), MsgBox()). I assume that this depends on whether the built-in function actively blocks, but I haven't found documentation on which ones are blocking functions either.

Please feel free correct and/or fill in the details as best you can, expecially the third point about how when one event can interrupt the function called from another.

Secondly, if I want to ensure that one particular function does not get interrupted by a second function until it is complete, how would I go about doing that in AutoIt? The best I can come up with is by faking an execution mutex for the function. Since the application is probably not all that important, I'll use some pseudocode and a real-world analogy.

So here is my hypothetical script. It is missing some functions and oversimplified for demonstration purposes.

expandcollapse popup

; ----- Setup -----
 
Global $fNewMail = False, $fGarageIsOpen = False
 
; Create COM object for fictitious WeatherEvents class which provides events for sunup and sundown.
$oWeather = ObjCreate("WeatherTool.WeatherEvents")
ObjEvent($oWeather, "_SunlightEvent_")
; Create COM object for fictitious OutdoorEvents class which provides events for mailbox tamper.
$oWeather = ObjCreate("SpyTool.OutdoorEvents")
ObjEvent($oWeather, "_OutdoorEvent_")
 
AdlibRegister("_CheckMail",10000)
 
; ----- Main loop -----
 
While 1
   ; Plenty of other hypothetical stuff to do here...
   ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
   Sleep(100)
WEnd
 
; ----- Functions -----
 
; This opens the garage door when the SunUp event is received from the WeatherTool.WeatherEvents COM class.
Func _SunlightEvent_SunUp()
   _GarageDoorOpen() ; Pretend that this is a non-blocking function that takes about 20 seconds to return. (It's a slow-moving door.)
   If Not @error Then $fGarageIsOpen = True ; Check for success before setting the flag.
EndFunc
 
; This closes the garage door when the SunDown event is received from the WeatherTool.WeatherEvents COM class.
Func _SunlightEvent_SunDown()
   _GarageDoorClose() ; Pretend that this is a non-blocking function that takes about 20 seconds to return. (It's a slow-moving door.)
   If Not @error Then $fGarageIsOpen = False ; Check for success before setting the flag.
EndFunc
 
; This executes when someone tampers with the door of the mailbox and sets the $fNewMail flag.
Func _OutdoorEvent_Tamper_Mailbox()
   $fNewMail = True
EndFunc
 
; AdLib calls this every 10 seconds
Func _CheckMail()
   If $fGarageIsOpen = True And $fNewMail = True Then ; If garage door is open and there is new mail, the robot should go get it.
     _Robot_GoOutside() ; Instruct robot to go outside.
     _Robot_GetMail() ; Instruct robot to get mail.
     If Not @error Then $fNewMail = False ; Mail was retrieved successfully; clear new mail flag.
     _Robot_GoInside() ; Instruct robot to go inside.
   EndIf
EndFunc

This looks like it should work in most cases, but what happens if mail is received and _OutdoorEvent_Tamper_Mailbox() is called, then it gets dark outside and _SunlightEvent_SunDown() is called soon afterward? We're waiting for GarageDoorClose() to finish. Meanwhile, AdLib fires and interrupts execution to call _CheckMail(). $fGarageIsOpen is still true until the door finishes closing and $fNewMail is true because the robot hasn't fetched the mail. So here is what happens: _Robot_GoOutside() runs and the robot tries to go outside and runs into the garage door. There is a brief screeching of twisting metal and the universe explodes, and I won't stand for that. The best way I know to prevent this in AutoIt is to fake an execution mutex using a global variable. Here is the revised code:

expandcollapse popup

; ----- Setup -----
 
Global $fNewMail = False, $fGarageIsOpen = False, $fMutexGarageAction = False
 
; Create COM object for fictitious WeatherEvents class which provides events for sunup and sundown.
$oWeather = ObjCreate("WeatherTool.WeatherEvents")
ObjEvent($oWeather, "_SunlightEvent_")
; Create COM object for fictitious OutdoorEvents class which provides events for mailbox tamper.
$oWeather = ObjCreate("SpyTool.OutdoorEvents")
ObjEvent($oWeather, "_OutdoorEvent_")
 
AdlibRegister("_CheckMail",10000)
 
; ----- Main loop -----
 
While 1
   ; Plenty of other hypothetical stuff to do here...
   ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
   Sleep(100)
WEnd
 
; ----- Functions -----
 
; This opens the garage door when the SunUp event is received from the WeatherTool.WeatherEvents COM class.
Func _SunlightEvent_SunUp()
   If $fMutexGarageAction = True Then Return
   $fMutexGarageAction = True
   _GarageDoorOpen() ; Pretend that this is a non-blocking function that takes about 20 seconds to return. (It's a slow-moving door.)
   If Not @error Then $fGarageIsOpen = True ; Check for success before setting the flag.
   $fMutexGarageAction = False
EndFunc
 
; This closes the garage door when the SunDown event is received from the WeatherTool.WeatherEvents COM class.
Func _SunlightEvent_SunDown()
   If $fMutexGarageAction = True Then Return
   $fMutexGarageAction = True
   _GarageDoorClose() ; Pretend that this is a non-blocking function that takes about 20 seconds to return. (It's a slow-moving door.)
   If Not @error Then $fGarageIsOpen = False ; Check for success before setting the flag.
   $fMutexGarageAction = False
EndFunc
 
; This executes when someone tampers with the door of the mailbox and sets the $fNewMail flag.
Func _OutdoorEvent_Tamper_Mailbox()
   $fNewMail = True
EndFunc
 
; AdLib calls this every 10 seconds
Func _CheckMail()
   If $fMutexGarageAction = True Then Return
   If $fGarageIsOpen = True And $fNewMail = True Then ; If garage door is open and there is new mail, the robot should go get it.
     _Robot_GoOutside() ; Instruct robot to go outside.
     _Robot_GetMail() ; Instruct robot to get mail.
     If Not @error Then $fNewMail = False ; Mail was retrieved successfully; clear new mail flag.
     _Robot_GoInside() ; Instruct robot to go inside.
   EndIf
EndFunc

Theoretically this should prevent any two actions or checks involving the garage door from happening at once. As you can see, _SunlightEvent_SunUp() and _SunlightEvent_SunDown() also check the variable before running to prevent harm occuring from function reentry. There are at least two problems with this design. Perhaps most obvious is that it can only be implemented in user-defined functions. But it also doesn't account for the few milliseconds of time in which _SunlightEvent_SunDown() has just been invoked but $fMutexGarageAction hasn't been set true yet.

So, is there a proper way to implement true function mutexes in AutoIt, or a way to make a user-defined function blocking, or does anyone have a better suggestion on how to handle this? (And please don't say to just always assume the door is closed unless known otherwise in my example. It's the execution order I'm concerned with.)

Edit: Added line breaks to enhance readability.

Edited October 4, 2012 by toasterking

[BrewManNH]

You could always disable the AdLib function on the entrance to any function you don't want interrupted, especially if it's a long running function, then reenable it after that function has finished.

[toasterking]

You could always disable the AdLib function on the entrance to any function you don't want interrupted, especially if it's a long running function, then reenable it after that function has finished.

True, that would work in this case, but often it's a non-AdLib function called by an event that interrupts my other function. I could unregister/reregister that as well, but then it starts getting really messy.

I'm hoping that this will catch the eye of a developer who can explain how AutoIt's scheduling engine works. I was surprised not to have seen more chatter on the subject.

Edited October 4, 2012 by toasterking

[BrewManNH]

Well, most of us don't have robots running out to get our mail.

[toasterking]

Well, most of us don't have robots running out to get our mail.

The PO doesn't have door service in my area, so what am I to do? That robot sucks at cooking and massages. I had to find something to keep it busy.

[toasterking]

I really hate to use this site as a reference, but the "Thread Priority" feature here is almost exactly what I am looking for in AutoIt:

http://www.autohotkey.com/docs/misc/Threads.htm

To that effect, note the "Priority" parameter here for the SetTimer command, which is roughly analogous to AdLibRegister() in AutoIt:

http://www.autohotkey.com/docs/commands/SetTimer.htm

And the "Critical" command which prevents the current "thread" from being interrupted by all other threads, essentially making it blocking:

http://www.autohotkey.com/docs/commands/Critical.htm

The execution environment should be a pretty close analog to AutoIt, as "threads" in AHK are conceptual (despite the author's incorrect use of the terminology) since the interpreter is also single-threaded.

I haven't found any documentation for AutoIt that explains how to set a "thread" priority or make it "critical". I suspect that the functionality is not exposed to the user on this level in AutoIt. But if I have to choose between using an ugly kluge in AutoIt or switching to AutoHotkey, I'll take the ugly kluge in AutoIt. :-) I'm still not sure if there is a kluge less ugly than the one I suggested above with the global variable, however.

Thanks in advance for anyone who ventures to invest their valuable time in consideration of my problem.

[BrewManNH]

Have you attempted any tests to see how things work? Put some consolewrites as a function is called and just before it exits the functions and see what order they're called in.

[toasterking]

Have you attempted any tests to see how things work? Put some consolewrites as a function is called and just before it exits the functions and see what order they're called in.

Yes, I have observed on many occasions in my production scripts the points I summarized in my first post. Did you want me to post examples illustrating all of them? (The third will be pretty tough since I can't make sense of it myself.)

[toasterking]

Okay, here are demonstrations of my first two points:

This demonstrates how an AdLib-called function interrupts a function called from the main loop.

AdLibRegister("_B",1500)
_A()
Exit

Func _A()
   ConsoleWrite("Func A: Checkpoint 1" & @CRLF)
   Sleep(1000)
   ConsoleWrite("Func A: Checkpoint 2" & @CRLF)
   Sleep(1000)
   ConsoleWrite("Func A: Checkpoint 3" & @CRLF)
EndFunc

Func _B()
   ConsoleWrite("Func B: Checkpoint 1" & @CRLF)
   Sleep(100)
   ConsoleWrite("Func B: Checkpoint 2" & @CRLF)
   Sleep(100)
   ConsoleWrite("Func B: Checkpoint 3" & @CRLF)
EndFunc

And this demonstrates how function _A() is reentrant. When it sends a WM_CLOSE message to the window, that fires an event which runs a second instance of _A() which interrupts the first. Notice how the second instance interrupts the first, but it doesn't get infinitely recursive as it looks like it should from the code. This is because AutoIt seems to keep track of how the function is called, and the GUI event cannot call another instance of the function until the existing instance has returned.

#include <GUIConstantsEx.au3>
Opt("GUIOnEventMode",1)
Global $nLevels = 0

$hWin = GuiCreate("Test")
GUISetOnEvent($GUI_EVENT_CLOSE,"_A")
_A()
Exit

Func _A()
   $nLevels += 1
   ConsoleWrite("Func A (Instance " & $nLevels & "): Checkpoint 1" & @CRLF)
   Sleep(1000)
   WinClose($hWin)
   ConsoleWrite("Func A (Instance " & $nLevels & "): Checkpoint 2" & @CRLF)
   Sleep(1000)
   ConsoleWrite("Func A (Instance " & $nLevels & "): Checkpoint 3" & @CRLF)
   $nLevels -= 1
EndFunc

[toasterking]

Now, check this one out. Here a single function "_A()" is called from the main loop, from 2 different events, and from AdLib.

expandcollapse popup

#include <GUIConstantsEx.au3>
Opt("GUIOnEventMode",1)
Global $nLevels = 0

$hWin = GuiCreate("Test",100,100)
GUISetOnEvent($GUI_EVENT_CLOSE,"_GuiCloseA")
$hButton = GUICtrlCreateButton("Button",10,10)
GUICtrlSetOnEvent(-1,"_ButtonA")
GUISetState(@SW_SHOW)
AdlibRegister("_AdlibCallA",400)
While 1
    Sleep(Random(15,75,1))
    _A("Main loop")
    Sleep(Random(15,75,1))
    WinClose($hWin)
    Sleep(Random(15,75,1))
    ControlClick($hWin,"",$hButton)
WEnd


Func _A($sSource)
    $nLevels += 1
    ConsoleWrite("Func A (Instance " & $nLevels & "): Checkpoint 1; called from " & $sSource & @CRLF)
    Sleep(100)
    ConsoleWrite("Func A (Instance " & $nLevels & "): Checkpoint 2; called from " & $sSource & @CRLF)
    Sleep(100)
    ConsoleWrite("Func A (Instance " & $nLevels & "): Checkpoint 3; called from " & $sSource & @CRLF)
    $nLevels -= 1
EndFunc

Func _GuiCloseA()
    _A("WinClose")
EndFunc

Func _ButtonA()
    _A("Button click")
EndFunc

Func _AdlibCallA()
    _A("AdLib")
EndFunc

Let that run for a while and see what it does. From my observation of its behavior as far as the source of the function call, the rules in this case seem to be:

• Anything can interrupt the main loop.
• Nothing can interrupt AdLib.
• Nothing can interrupt the call from GuiSetOnEvent (window close).
• The call from GuiCtrlSetOnEvent (button click) can be interrupted by AdLib, but not by anything else.

And what I am asking is, "WTF?!?" In this example the behavior is pretty consistent but completely unpredictable!

[toasterking]

Then it's decided: I will use my ugly kluges for now. But if anyone has a better solution, please feel free to resurrect this thread months or even years later, as I will probably still want to know. BrewManNH, thanks for helping me to ask the right questions.

[wolflake]

^{I'd like to say thank you for "documenting" this because I have an adlib calling a function that might not finish before the adlib triggered to call it again.} 

^{Now I know that it will be ok because it won't be interupted by the next adlib call.}

[batworx]

I realise this reply is over 2 years late, but I've sort of found a way to prevent reentrancy in functions. Here's an example:

; Script starts here.

$NoExecute = False

Func Test ()

If ($NoExecute = False) Then

$NoExecute = True

Else

Return

EndIf

; Insert code we want to run here:

MsgBox (64, "Alert", "Testing reentrancy in AutoIt.")

$NoExecute = False

EndFunc

Hope this helps.

[TheSaint]

I have too much in my head to go through your code right now and all the discussion, but I thought I'd add this little offering, which may not have occurred to you and might be useful.

When I have timing etc problems, I nearly always end up having to run a small companion executable (hidden from Taskbar). Both executables share some ini file or Registry entries, that both read and write. I have a few instances myself, where a floating STOP button is the better option. This lets a multi-core Windows environment do some of the work for you.

Call it the Watcher watching the Watcher, if you like.

Edited December 13, 2014 by TheSaint

[toasterking]

I realise this reply is over 2 years late, but I've sort of found a way to prevent reentrancy in functions. Here's an example:

[...]

 

Thanks, batworx!  I just saw your reply.  Apparently, I was no longer following the topic.

This is the same method that I used in the second bit of code in my first post (I used $fMutexGarageAction in place of $NoExecute).  It doesn't technically prevent the function from being reentered, but it does prevent the important part of the code from being executed concurrently IF the caller is the same.  But consider if you had only one function being called from two different callers, each competing for priority.  The function could get called once from the main loop, then before it finishes, interrupted by an event callback and executed again.  Because of the value of the global variable $NoExecute, it doesn't run any useful code inside the function the second time.  If it's working with data relevant to the event callback then this is a big problem because that data never gets operated on and it's like the event never gets properly serviced.  This is why you might need to prevent true reentrancy.  If there was a way to make that event callback wait until the function is finished before calling it a second time, then reentrancy is prevented and everything works, but I cannot find a way to do that.

Also, this method doesn't do anything to prevent a a function from being interrupted by a different function.

[toasterking]

[...] When I have timing etc problems, I nearly always end up having to run a small companion executable (hidden from Taskbar). Both executables share some ini file or Registry entries, that both read and write. I have a few instances myself, where a floating STOP button is the better option. [...]

 

Thanks for your idea.  There's not really enough detail there to tell if this addresses any of my problems.  I have run multiple concurrent AutoIt scripts in separate processes as well that communicate with each other as a way to get around the single-thread limit in AutoIt by leveraging the preemptive multitasking in the OS, but that really doesn't address the concerns I have with AutoIt's scheduling within a single thread.  Maybe you can provide more detail on your approach.  I have learned that the registry is the better option of the two you suggest if both scripts will be writing to the same values.  Each registry operation is atomic because it is serialized by the OS, while writing an INI file involves many smaller non-atomic operations, so it would be possible for one script to read the INI file when the other has only partially written it.  If the value that was read from the INI is then written back to it, the original value is corrupt at that point.