Leaderboard

UI Automation Events In standard Windows GUI code, you can respond to Windows messages by registering and implementing a message handler function. A WM_NOTIFY message handler is registered and implemented in this way: ; Registration GUIRegisterMsg( $WM_NOTIFY, "WM_NOTIFY" ) ; Implementation Func WM_NOTIFY( $hWnd, $iMsg, $wParam, $lParam ) ; Code EndFunc The WM_NOTIFY function is a callback function. In UI Automation code, you can respond to UI Automation events by registering and implementing an event handler object. A FocusChangedEventHandler object is registered and implemented in this way: ; Registration UIAEH_FocusChangedEventHandlerCreate() $oUIAutomation.AddFocusChangedEventHandler( 0, $oUIAEH_FocusChangedEventHandler ) ; Implementation ; This is the function that receives events Func UIAEH_FocusChangedEventHandler_HandleFocusChangedEvent( $pSelf, $pSender ) ; Code EndFunc $oUIAEH_FocusChangedEventHandler is the event handler object. It's a callback object. This thread is about how to respond to UI Automation events by registering, implementing and using event handler objects. User Defined Functions Registering and implementing the event handler object in the code above is very simple. To make it that simple, it's necessary to include a few functions in a UDF. The zip-file at bottom of post contains a UIAEH_FocusChangedEventHandler.au3 UDF that allows the registration and implementation of the event handler object as simple as shown above. UIAEH_FocusChangedEventHandlerCreate() is a function in this UDF. The function creates the $oUIAEH_FocusChangedEventHandler object. Because this is a callback object it must be implemented through ObjectFromTag() (coded in ObjectFromTag.au3 in zip-file) function by trancexx and not through the usual ObjCreateInterface() which is used to create other UI Automation objects. (The warning at top of the help text is much more an error in the help text than an error in the function. Thousands of examples show that ObjCreateInterface() works flawlessly.) You don't need to use ObjectFromTag() in your own code. This function is completely handled in the UDFs. UIAEH_FocusChangedEventHandler_HandleFocusChangedEvent() is the function that receives events. This function is coded but commented out in UIAEH_FocusChangedEventHandler.au3. Copy this function to your own code. See FocusChangedEventHandler example below. UIA Event Handlers The first version of Microsoft's UI Automation code (the Windows 7 version that works on Windows XP, Windows Vista, Windows 7 and all later versions) implements four different event handler objects. In addition to the FocusChangedEventHandler object, it's the AutomationEventHandler, PropertyChangedEventHandler and StructureChangedEventHandler objects. The zip-file at the bottom contains the corresponding three event handler UDFs. By the way, these four event handlers are implemented in AutoIt several years ago in this and the following posts. FocusChangedEventHandler exposes a method to handle events that are raised when the keyboard focus moves to another UI Automation element. AutomationEventHandler exposes a method to handle Microsoft UI Automation events. These events include MenuOpened ($UIA_MenuOpenedEventId) and Window_WindowOpened ($UIA_Window_WindowOpenedEventId) events. See module UIA_EventIds in UIA_Constants.au3 for a complete list. Note that EventIds that belong to Windows 8 or later cannot be used in this version of the event handler. But they can be used after code updates over the coming weeks. 2019-12-15 If you select Desktop, AutomationEvent and all Events (Event Ids) in UI Automation Event Monitor below and click Start, you'll see the following output in SciTE console: $oUIAEH_AutomationEventHandler OK Desktop: AddAutomationEventHandler() = AsyncContentLoaded OK Desktop: AddAutomationEventHandler() = AutomationFocusChanged ERR Desktop: AddAutomationEventHandler() = AutomationPropertyChanged OK Desktop: AddAutomationEventHandler() = InputDiscarded OK Desktop: AddAutomationEventHandler() = InputReachedOtherElement OK Desktop: AddAutomationEventHandler() = InputReachedTarget OK Desktop: AddAutomationEventHandler() = Invoke_Invoked OK Desktop: AddAutomationEventHandler() = LayoutInvalidated OK Desktop: AddAutomationEventHandler() = MenuClosed OK Desktop: AddAutomationEventHandler() = MenuModeEnd OK Desktop: AddAutomationEventHandler() = MenuModeStart OK Desktop: AddAutomationEventHandler() = MenuOpened OK Desktop: AddAutomationEventHandler() = Selection_Invalidated OK Desktop: AddAutomationEventHandler() = SelectionItem_ElementAddedToSelection OK Desktop: AddAutomationEventHandler() = SelectionItem_ElementRemovedFromSelection OK Desktop: AddAutomationEventHandler() = SelectionItem_ElementSelected OK Desktop: AddAutomationEventHandler() = StructureChanged OK Desktop: AddAutomationEventHandler() = Text_TextChanged OK Desktop: AddAutomationEventHandler() = Text_TextSelectionChanged OK Desktop: AddAutomationEventHandler() = ToolTipClosed OK Desktop: AddAutomationEventHandler() = ToolTipOpened OK Desktop: AddAutomationEventHandler() = Window_WindowClosed OK Desktop: AddAutomationEventHandler() = Window_WindowOpened OK Note the error for the AutomationFocusChanged event handler (at the top of the list). I've found that there is an error, but so far have no explanation for it. PropertyChangedEventHandler exposes a method to handle Microsoft UI Automation events that occur when a property is changed. These events include Name ($UIA_NamePropertyId) and ValueValue ($UIA_ValueValuePropertyId) changes. See module UIA_PropertyIds in UIA_Constants.au3 for a complete list. Note that PropertyIds that belong to Windows 8 or later cannot be used in this version of the event handler. But they can be used after code updates over the coming weeks. StructureChangedEventHandler exposes a method to handle events that occur when the Microsoft UI Automation tree structure is changed. This event handler seems to be more valuable in eg. screen readers than in usual UI Automation code. NotificationEventHandler exposes a method to handle Microsoft UI Automation notification events. See this post. FocusChangedEventHandler example This is full code in Examples\FocusChangedEventHandlerEx.au3. Run the code in SciTE with F5. #AutoIt3Wrapper_Au3Check_Parameters=-d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6 -w 7 ;#AutoIt3Wrapper_UseX64=y Opt( "MustDeclareVars", 1 ) #include "..\Includes\UIAEH_FocusChangedEventHandler.au3" Example() Func Example() UIAEH_FocusChangedEventHandlerCreate() If Not IsObj( $oUIAEH_FocusChangedEventHandler ) Then Return ConsoleWrite( "$oUIAEH_FocusChangedEventHandler ERR" & @CRLF ) ConsoleWrite( "$oUIAEH_FocusChangedEventHandler OK" & @CRLF ) Local $oUIAutomation = ObjCreateInterface( $sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtag_IUIAutomation ) If Not IsObj( $oUIAutomation ) Then Return ConsoleWrite( "$oUIAutomation ERR" & @CRLF ) ConsoleWrite( "$oUIAutomation OK" & @CRLF ) Local $iError = $oUIAutomation.AddFocusChangedEventHandler( 0, $oUIAEH_FocusChangedEventHandler ) If $iError Then Return ConsoleWrite( "AddFocusChangedEventHandler() ERR" & @CRLF ) ConsoleWrite( "AddFocusChangedEventHandler() OK" & @CRLF ) HotKeySet( "{ESC}", "Quit" ) While Sleep(10) WEnd EndFunc Func Quit() UIAEH_FocusChangedEventHandlerDelete() Exit EndFunc ; This is the function that receives events Func UIAEH_FocusChangedEventHandler_HandleFocusChangedEvent( $pSelf, $pSender ) ; Ret: long Par: ptr ConsoleWrite( @CRLF & "UIAEH_FocusChangedEventHandler_HandleFocusChangedEvent: " & $pSender & @CRLF ) Local $oSender = ObjCreateInterface( $pSender, $sIID_IUIAutomationElement, $dtag_IUIAutomationElement ) $oSender.AddRef() ConsoleWrite( "Title = " & UIAEH_GetCurrentPropertyValue( $oSender, $UIA_NamePropertyId ) & @CRLF & _ "Class = " & UIAEH_GetCurrentPropertyValue( $oSender, $UIA_ClassNamePropertyId ) & @CRLF & _ "Ctrl type = " & UIAEH_GetCurrentPropertyValue( $oSender, $UIA_ControlTypePropertyId ) & @CRLF & _ "Ctrl name = " & UIAEH_GetCurrentPropertyValue( $oSender, $UIA_LocalizedControlTypePropertyId ) & @CRLF & _ "Handle = " & "0x" & Hex( UIAEH_GetCurrentPropertyValue( $oSender, $UIA_NativeWindowHandlePropertyId ) ) & @CRLF & _ "Value = " & UIAEH_GetCurrentPropertyValue( $oSender, $UIA_ValueValuePropertyId ) & @CRLF ) Return 0x00000000 ; $S_OK #forceref $pSelf EndFunc ; Auxiliary function (for simple properties only) ; There must be only one instance of this function Func UIAEH_GetCurrentPropertyValue( $oSender, $iPropertyId ) Local $vPropertyValue $oSender.GetCurrentPropertyValue( $iPropertyId, $vPropertyValue ) Return $vPropertyValue EndFunc Test the code by opening the Desktop context menu and move the mouse over the menu items. When the mouse hovers over a menu item it gets focus. Exit with Esc. AutomationEventHandlerEx.au3 Test the example by opening some windows, menus and tooltips. PropertyChangedEventHandlerEx.au3 Open Windows Task Manager (Ctrl+Shift+Esc) and you'll get a whole bunch of property changed events. Note that to access Windows Task Manager (and other programs) on Windows 10, use #RequireAdmin in the script and run SciTE as administrator. StructureChangedEventHandlerEx.au3 Clicking on a folder in File/Windows Explorer treeview generates a lot of structure changed events. New examples will be added over the coming weeks. UI Automation Event Monitor UI Automation Event Monitor is a tool to detect UI Automation events. The first group of toolbar buttons is used to set up the UI Automation event handler. The images have a PropertyChangedEventHandler set up to monitor Name ($UIA_NamePropertyId) and ValueValue ($UIA_ValueValuePropertyId) property changes in Windows Task Manager. Note that to access Windows Task Manager (and other programs) on Windows 10, use #RequireAdmin in the script (UIAEHEvents.au3) and run SciTE as administrator. These four toolbar buttons are hot-track enabled with the technique used in this example. The buttons open four listviews to set up the event handler. All listviews have multiple selections enabled. Select multiple items at once and check a single item in/out to check all selected items in/out. Use mouse or Space key to check an item in/out. In the first image, the mouse hovers over the Top Windows button so that it's selected. In the second image, the Top Windows button is clicked so that the listview is opened. Moving the mouse over the next three buttons automatically opens the corresponding listviews due to hot-tracking. Click the toolbar button again to close the listview or move the mouse above the title bar. Windows Task Manager is checked in the Top Windows listview. The Event Handlers listview is enabled when a Top Window is checked. The Event Ids listview is enabled when the AutomationEvent event handler is checked. The Property Ids listview is enabled when the PropertyChanged event handler is checked. In Event Ids and Property Ids listview you can select an item by pressing the first letter on the keyboard. The second group of toolbar buttons is used to Start/Stop the event handler (or multiple event handlers), Pause/Continue updating the Events listview, and Clear the listview. Note that the Pause button only stops updating the listview. Events are still detected and stored in $aEvents array. All new events are added at the bottom of the listview when you click Continue or Stop. The Pause Main Loop button is used to stop the GUI main loop, which significantly increases the performance of the Events listview. When the GUI main loop is paused you can only click the Continue Main Loop button (same button as Pause Main Loop) and you can close the GUI. Events listview The Events listview is a virtual listview because of performance and to make it easy to constantly add new rows to the bottom of the listview. Nevertheless, there can still be performance issues when a lot of events are generated. To remedy these issues, it's possible to stop the GUI main loop with the Pause Main Loop button. This significantly increases the performance of the Events listview. When there are a large number of events (when many new rows are added quickly at the bottom of the listview), this procedure must be followed to stop the event handler (or multiple event handlers): Make sure the GUI main loop isn't paused. Click the Pause button (second toolbar group) to pause Events listview updates. Click the Stop button to stop the event handler. When there are a small number of events you can immediately click the Stop button. To test the UI Automation Event Monitor, try performing the four examples (in Examples\, briefly described above) through the monitor GUI. There'll probably be some GUI updates in the coming weeks. UIA Menu, Tooltip and Window Events Monitor is a simpler version of UI Automation Event Monitor that detects only MenuOpened, ToolTipOpened and Window_WindowOpened events (AutomationEvents). Updates Windows 8, Windows 8.1 and Windows 10 updates Threads UI Automation UDFs contains all include files. UIASpy - UI Automation Spy Tool is a GUI tool that provides information about windows and controls and their interconnection and provides functionality to generate sample code. UIASpy is essential for creating UI Automation code. In Using UI Automation Code in AutoIt you can find and download examples and read information about using UIA code. IUIAutomation MS framework automate chrome, FF, IE, .... created by junkew August 2013 is the first AutoIt thread on UIA code. Zip-file The zip contains source files for GUIs and examples. Note that UI Automation UDFs must be installed in the Includes folder. You need AutoIt 3.3.12 or later. Tested on Windows 7 and Windows 10. Note that under Windows 7 and earlier versions, only the original four event handlers (AutomationEventHandler, FocusChangedEventHandler, PropertyChangedEventHandler, StructureChangedEventHandler) introduced in Windows 7 can be used. Comments are welcome. Let me know if there are any issues. UIAEHEvents.7z

Dam, you have almost 1.4k post and we are still there, asking you to provide the code. Every time, same pattern

👍you covered the whole uia model with excellent examples in AutoIt coding. Not much left to give examples on.

No... _WD_ElementAction will read an element's attributes or properties, but it doesn't offer a mechanism to update them (because it's not in the W3C specs). You should be able to do it with _WD_ExecuteScript. Figure out how to do it using Javascript in the browser's Developer Tools console. Once you have the correct syntax, then translate that for use in _WD_ExecuteScript.

You'll need to switch to the correct frame before the element can be located / accessed. See DemoFrames function in wd_demo.au3 for an example of the available functions for accessing frames.

Notepad context menu Automating a context menu consists of opening the menu, detecting the open menu and clicking a menu item. Here, the technique is demonstrated with Notepad as an example and the "Right to left Reading order" menu item in the context menu of the Edit control. The inspiration for the example is found in this thread. Opening the menu A small two-button GUI is used to open the context menu by generating a secondary mouse click: ; Create AutoIt GUI Local $hGui = GUICreate( "Automate Notepad Context Menu", 330, 40, 100, 100 ) Local $idRight = GUICtrlCreateButton( "Right to left Reading order", 10, 10, 150, 20 ) Local $idLeft = GUICtrlCreateButton( "Left to right Reading order", 170, 10, 150, 20 ) GUICtrlSetState( $idLeft, $GUI_DISABLE ) ; Show GUI GUISetState( @SW_SHOW ) ; Main loop Local $iMsg While 1 $iMsg = GUIGetMsg() Switch $iMsg Case $idRight, $idLeft ; Secondary mouse click UIA_WinActivate( $oNotepad ) UIA_WinWaitActive( $oNotepad ) UIA_MouseClick( $oEdit1, True ) GUICtrlSetState( $iMsg = $idRight ? $idRight : $idLeft, $GUI_DISABLE ) GUICtrlSetState( $iMsg = $idRight ? $idLeft : $idRight, $GUI_ENABLE ) ConsoleWrite( @CRLF & "Secondary mouse click" & @CRLF ) Detecting the open menu The event handler to detect menu open events is created this way: ; Create UI Automation event handler UIAEH_AutomationEventHandlerCreate() If Not IsObj( $oUIAEH_AutomationEventHandler ) Then Return ConsoleWrite( "$oUIAEH_AutomationEventHandler ERR" & @CRLF ) ConsoleWrite( "$oUIAEH_AutomationEventHandler OK" & @CRLF ) ; Detect menu open events Local $iError = $oUIAutomation.AddAutomationEventHandler( $UIA_MenuOpenedEventId, $pDesktop, $TreeScope_Subtree, 0, $oUIAEH_AutomationEventHandler ) If $iError Then Return ConsoleWrite( "AddAutomationEventHandler() ERR" & @CRLF ) ConsoleWrite( "AddAutomationEventHandler() OK" & @CRLF ) This is the function that receives events. The event is sent to the AutoIt main loop using a dummy control, $idContextMenu: ; This is the function that receives events Func UIAEH_AutomationEventHandler_HandleAutomationEvent( $pSelf, $pSender, $iEventId ) ; Ret: long Par: ptr;int ConsoleWrite( "UIAEH_AutomationEventHandler_HandleAutomationEvent: $iEventId = " & $iEventId & @CRLF ) $oContextMenu = ObjCreateInterface( $pSender, $sIID_IUIAutomationElement, $dtag_IUIAutomationElement ) $oContextMenu.AddRef() If Not IsObj( $oContextMenu ) Then Return ConsoleWrite( "$oContextMenu ERR" & @CRLF ) ConsoleWrite( "$oContextMenu OK" & @CRLF ) GUICtrlSendToDummy( $idContextMenu ) Return 0x00000000 ; $S_OK #forceref $pSelf EndFunc Clicking a menu item And this is code that clicks the "Right to left Reading order" menu item: Case $idContextMenu ; Context menu item Local $pAndCondition1 $oUIAutomation.CreatePropertyCondition( $UIA_ControlTypePropertyId, $UIA_MenuItemControlTypeId, $pCondition0 ) $oUIAutomation.CreatePropertyCondition( $UIA_NamePropertyId, "Right to left Reading order", $pCondition1 ) $oUIAutomation.CreateAndCondition( $pCondition0, $pCondition1, $pAndCondition1 ) If Not $pAndCondition1 Then Return ConsoleWrite( "$pAndCondition1 ERR" & @CRLF ) ConsoleWrite( "$pAndCondition1 OK" & @CRLF ) ; Find Context menu item Local $pMenuItem1, $oMenuItem1 $oContextMenu.FindFirst( $TreeScope_Descendants, $pAndCondition1, $pMenuItem1 ) $oMenuItem1 = ObjCreateInterface( $pMenuItem1, $sIID_IUIAutomationElement, $dtag_IUIAutomationElement ) If Not IsObj( $oMenuItem1 ) Then Return ConsoleWrite( "$oMenuItem1 ERR" & @CRLF ) ConsoleWrite( "$oMenuItem1 OK" & @CRLF ) ; Click Context menu item Local $pInvokePattern1, $oInvokePattern1 $oMenuItem1.GetCurrentPattern( $UIA_InvokePatternId, $pInvokePattern1 ) $oInvokePattern1 = ObjCreateInterface( $pInvokePattern1, $sIID_IUIAutomationInvokePattern, $dtag_IUIAutomationInvokePattern ) If Not IsObj( $oInvokePattern1 ) Then Return ConsoleWrite( "$oInvokePattern1 ERR" & @CRLF ) ConsoleWrite( "$oInvokePattern1 OK" & @CRLF ) $oInvokePattern1.Invoke() ConsoleWrite( "$oInvokePattern1.Invoke()" & @CRLF ) Full code in Examples\Notepad context menu\ContextMenu.au3. Note that because of the UIAEH_AutomationEventHandler.au3 UDF, the code in the example here is much simpler than the code in the link above. To run the example open Notepad and enter or paste some text into the Edit control. Then run ContextMenu.au3 in SciTE with F5. New zip-file at bottom of first post.

Then make an autoit program that automates notepad.exe compile it and copy and start it on the vps server. Just add some send and mousemove commands. Example code you can find in help Look at commands run send controlsend winactivate mousemove etc.