IUIAutomation MS framework automate chrome, FF, IE, ....

[junkew]

Differs due to many different factors like platform version. As inspect.exe is the main ms tool that's the best default tool to check with. And then you also have different tools using different ways of identifying. The uia is a good base from Microsoft but commercial tools like ranorex or testcomplete can give different insights and some helpfull tools look with ocr or ai imaging basically a smart way of bitmap analyzing.

[dev_automation_call]

Hi,

Okay thats really useful. Sooo after trying everything above i still have the issue.  Inspect.exe does show me different trees based on the different OS versions. Not sure what to try next since it seems on windows server more elements get found(I dont want to use any other libs e.g ranorex or img search). I tried swapping out OLEACC.dll and UIAutomationCore.dll (UIA docs references these as core to the functionality of the lib) in system 32 to use the windows server dll's as a last ditch attempt. But this only broke chrome in the way that no tabs load anymore and u get the aw,snap message everytime 😂.

[dev_automation_call]

Actually i did find the item in the tree, the tree as mentioned is slightly different. This is the major difference 

Provider Description: MSAA Proxy (IAccessible2) (unmanaged:UIAutomationCore.DLL)]" - Windows Pro (Not working)

Provider Description: MSAA Proxy (unmanaged:UIAutomationCore.DLL)]" - Windows server (Working)

Can i get rid of IAccessible2 somehow haha

[junkew]

Why do you want the trees to be the same? Finding elements on different platforms or versions needs a certain strategy.

[NassauSky]

Hi @junkew ,

In version V0-70, has anyone questioned the output of ex1_Treewalker? Is it common for the clipping of the first letter or more of the Window in the console output?  ie  <askbar>, <eamViewer Panel>, <ite [6 of 6]> 

Would you recommend I make any code adjustments to accommodate for this?

Thanks!

 

**** Deal with browser windows ****
location: 0<askbar><Taskbar><Shell_TrayWnd><000100AE>
location: 0<eamViewer Panel><TeamViewer Panel (minimized)><TV_ControlWinMinimized><000303DA>
location: 46<ite [6 of 6]><Z:\OfficeCheck\Alt\ex1_Treewalker.au3 - SciTE-Lite [6 of 6]><SciTEWindow><00090034>
location: 8<Google Chrome><Google - Google Chrome><Chrome_WidgetWin_1><000C06A8>

**** Continue with finding 2 browser windows ****
location: 0<askbar><Taskbar><Shell_TrayWnd><000100AE>
location: 0<eamViewer Panel><TeamViewer Panel (minimized)><TV_ControlWinMinimized><000303DA>
location: 46<ite [6 of 6]><Z:\OfficeCheck\Alt\ex1_Treewalker.au3 - SciTE-Lite [6 of 6]><SciTEWindow><00090034>
location: 8<Google Chrome><Google - Google Chrome><Chrome_WidgetWin_1><000C06A8>
Yes found first browser 

**** Deal with 2nd browser window****
location: 0<askbar><Taskbar><Shell_TrayWnd><000100AE>
location: 0<eamViewer Panel><TeamViewer Panel (minimized)><TV_ControlWinMinimized><000303DA>
location: 46<ite [6 of 6]><Z:\OfficeCheck\Alt\ex1_Treewalker.au3 - SciTE-Lite [6 of 6]><SciTEWindow><00090034>
location: 8<Google Chrome><Google - Google Chrome><Chrome_WidgetWin_1><000C06A8>
location: 121<Microsoft? Edge><IUIAutomation MS framework automate chrome, FF, IE, .... - Page 45 - AutoIt Example Scripts - AutoIt Forums - Profile 1 - Microsoft? Edge><Chrome_WidgetWin_1><001201FC>
Yes found second browser 
>Exit code: 0

 

[junkew]

No it was never questioned but if you see a fix please post Just the example code so others can benefit.

[Viszna]

I'm using UIAWrapper.au3 and sometimes I get this error:

"C:\AUTOMATY\UIAWrappers.au3" (1674) : ==> Array variable has incorrect number of subscripts or subscript dimension range exceeded.:
$x = Int($t[1] + ($t[3] / 2))
$x = Int($t[1] + (^ ERROR
->14:05:04 AutoIt3.exe ended.rc:1

The biggest problem is catching the error, the script runs ~10 hours every day.
The function is called dozens of times a day.
The error occurs 2 a month, but it is irritating because it interrupts the script.

Excerpt from UIAWrappers.au3

1668        Local $t
1669        $t = StringSplit(_UIA_getPropertyValue($obj2ActOn, $UIA_BoundingRectanglePropertyId), ";")
1670    ;~  If no parameter detect the middle
1671        If $p1 = 0 Then
1672    ;~  consolewrite(_UIA_getPropertyValue($obj, $UIA_BoundingRectanglePropertyId) & $t[1] & ";" & $t[2] & ";" & $t[3] & ";" & $t[4] & @crlf)
1673    ;~  _winapi_mouse_event($MOUSEEVENTF_ABSOLUTE + $MOUSEEVENTF_MOVE,$t[1],$t[2])
1674            $x = Int($t[1] + ($t[3] / 2))
1675            $y = Int($t[2] + $t[4] / 2)
1676        Else
1677    ;~      calculate location relative in the element found
1678            $x = Int($t[1] + $p1)
1679            $y = Int($t[2] + $p2)
1680        EndIf

Maybe it's worth adding a verification of the $t variable on line 1670 before further operations on the $t array?

I added an error trap - now I'm waiting for it to appear and check what goes to $t.

[junkew]

Most likely $obj2acton is not on the screen or not found and as such you do not get a rectangle. 

So check if the object is there besides that the uiwrappers could be improved regarding errors handling.

[mLipok]

Recently I was working on simplifiaction in usage with UIA

And thus I have almostly completed new UDF to simplify UIA usage.

 

Example:

Func _Example()

    Local $hWND_UIAFound = 0
    _log(@ScriptLineNumber & ' ' & WinExists($_sExampleGuiTitle, "")

    _UIASimple_WinExist($_sExampleGuiTitle, "", "Some text to find", $hWND_UIAFound)

    _UIASimple_ButtonClick($hWND_UIAFound, "", "Some text to find", "Orange")

    _UIASimple_ButtonClick($hWND_UIAFound, "", "", "&Fake button")

    _UIASimple_ButtonClick($_sExampleGuiTitle, "OK", "Some text to find", "Banana")

EndFunc   ;==>_Example

it uses my UIASimple.au3 which I will publish when it will be finished.

And part of UIASimple.au3 UDF:

....
....
....
Global $_UIASimple_TEXT_MATCH_MODE = $PropertyConditionFlags_IgnoreCase ; https://learn.microsoft.com/en-us/windows/win32/api/uiautomationclient/ne-uiautomationclient-propertyconditionflags
....
....
....
#Region - _UIASimple_*** Helpers
Func _UIASimple_ButtonClick($sWinTitle, $sWinText, $sUIAText, $sButton)
    If Not WinExists($sWinTitle, $sWinText) Then Return SetError(1, 0, False)

    Local $hWND_UIAFound = 0 ; Reset
    Local $oWindow = _UIASimple_FindWindow($sWinTitle, $sWinText, $sUIAText, $hWND_UIAFound)
    If @error Then Return SetError(2, @extended, False)

    Local $oButton = _UIASimple_FindSubObject_byName($oWindow, $sButton)
    If @error Then Return SetError(3, @extended, False)

    _UIASimple_ObjectInvoke($oButton)
    If @error Then Return SetError(4, @extended, False)

    Return True
EndFunc   ;==>_UIASimple_ButtonClick
....
....
....

 

My general rule was to map the Standard AutoIt functions to give possibility to do things in a similar way but with UIA, so that it would be easy to switch to UIA (when it is needed).

But currently I have one hm.... dilemma.

As so far I used: UIA_Constants.au3
I also saw CUIAutomation2.au3
both files have defined the same values in a different convention.
For example:

$dtag_IUIAutomationInvokePattern
$dtagIUIAutomationInvokePattern

Some other const are defined with the same names so I can't use both together.

So my dilema is:
Which file should I use and why ? CUIAutomation2.au3 or UIA_Constants.au3 ?

 

Edited November 9, 2023 by mLipok

[queensoft]

I'm using UIA_V0_70.zip and UIASpy to read controls from an external program.

Program has 4 ComboBoxes. They all have almost identical properties: no title, same class type, same ctrl type, same ctrl name, no handle:

Title     = 
Class     = ComboBox
Ctrl type = 50003
Ctrl name = combo box
Selected  = False
Handle    = 00000000

The only difference I can find, using UIASpy, is the RuntimeId (something like 7;18760;23387474, 7;18760;15337269, 7;18760;12405375, 7;18760;44539515) and Parent to child index (7, 8, ....).

How do I use those to click different comboboxes?

I tried this and it did'nt work:

_UIA_Action('title:=;classname:=ComboBox;RuntimeId:='&$combo000[0], "click")   ; $combo000 = array of all 4 different RuntimeIds

If I use this, it clicks the first ComboBox just fine:

_UIA_Action("title:=;classname:=ComboBox;ControlType:=50003","click")

If I use UIASpy > right click each ComboBox > Show UI Automation element = it works perfectly, each different ComboBox is highlighted.

But I don't understant what $iIdx is, in this context (first ComboBox is 20, for instance).

Thank you.

[junkew]

Search in uia examples for index and indexrelative property.

Easiest is to use a findall that finds the 4 items in an array and then you just pick the right one.

[queensoft]

Cool, so simple!!!

;index:=2

_UIA_Action("title:=;classname:=ComboBox;ControlType:=50003;index:=2","click")

 

[SEKOMD]

Please tell me how to disable event focus in _UIA_action?

For example command:

_UIA_action($oUIElement,"setValue", "test text")

activates the window, how to disable it?

[SEKOMD]

This is most likely not possible, but I will ask - is it possible to insert html content into a page in Chrome using UIAutomation (for example, by setting an h2 tag to a blank about:blank page)?

[junkew]

It's possible with bookmarklet or in edge with reading list executing javascript.

So if you master javascript you can manipulate the whole dom tree.

Type in chrome browser addressbar

javascript:alert(‘Hello World!‘);

Edited May 21 by junkew

[NassauSky]

OK I got some UIA working in the past but stepped away from it for a while.  Now working on a new project and I'm trying to get the list of desktop icons using UIA with no luck at the moment:

 

expandcollapse popup

;Instructions How to Create This Code From UIAspy
;  1) Create the initial base code Menu->Sample Code->Initial Code->Complete Code
;  2) Click one layer at a time the correct object in the left pane
;  3) At each child branch right click a $variable then choose Create Sample Code
;  4) ?
#AutoIt3Wrapper_Au3Check_Parameters=-d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6 -w 7
#AutoIt3Wrapper_UseX64=y ; If target application is running as 64 bit code or =n if 32bit
#include "Includes\UIA_Constants.au3" ; Can be copied from UIASpy Includes folder

Opt( "MustDeclareVars", 1 )

; Start UIA detection
    Example()

    SplashTextOn("","Done locating UIA objects", 500, 100, -1, -1, -3, "Comic Sans MS", 18)
        Sleep(1500)
    SplashOff()

Func Example()
    ; Create UI Automation object
    Local $oUIAutomation = ObjCreateInterface( $sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtag_IUIAutomation )
    If Not IsObj( $oUIAutomation ) Then Return ConsoleWrite( "$oUIAutomation ERR" & @CRLF )
    ConsoleWrite( "$oUIAutomation OK" & @CRLF )

    ; Get Desktop element
    Local $pDesktop, $oDesktop
    $oUIAutomation.GetRootElement( $pDesktop )
    $oDesktop = ObjCreateInterface( $pDesktop, $sIID_IUIAutomationElement, $dtag_IUIAutomationElement )
    If Not IsObj( $oDesktop ) Then Return ConsoleWrite( "$oDesktop ERR" & @CRLF )
    ConsoleWrite( "$oDesktop OK" & @CRLF )



    ; --- Find window/control ---

    ConsoleWrite( "--- Find window/control --- Progman" & @CRLF )

    Local $pCondition0
    $oUIAutomation.CreatePropertyCondition( $UIA_ClassNamePropertyId, "Progman", $pCondition0 )
    If Not $pCondition0 Then Return ConsoleWrite( "$pCondition0 ERR" & @CRLF )
    ConsoleWrite( "$pCondition0 OK" & @CRLF )

    Local $pPane1, $oPane1
    $oDesktop.FindFirst( $TreeScope_Children, $pCondition0, $pPane1 )
    $oPane1 = ObjCreateInterface( $pPane1, $sIID_IUIAutomationElement, $dtag_IUIAutomationElement )
    If Not IsObj( $oPane1 ) Then Return ConsoleWrite( "$oPane1 ERR" & @CRLF )
    ConsoleWrite( "$oPane1 OK" & @CRLF )



    ; --- Find window/control ---

    ConsoleWrite( "--- Find window/control --- SHELLDLL_DefView" & @CRLF )

    Local $pCondition1
    $oUIAutomation.CreatePropertyCondition( $UIA_ClassNamePropertyId, "SHELLDLL_DefView", $pCondition1 )
    If Not $pCondition1 Then Return ConsoleWrite( "$pCondition1 ERR" & @CRLF )
    ConsoleWrite( "$pCondition1 OK" & @CRLF )

    Local $pPane2, $oPane2
    $oPane1.FindFirst( $TreeScope_Descendants, $pCondition1, $pPane2 )
    $oPane2 = ObjCreateInterface( $pPane2, $sIID_IUIAutomationElement, $dtag_IUIAutomationElement )
    If Not IsObj( $oPane2 ) Then Return ConsoleWrite( "$oPane2 ERR" & @CRLF )
    ConsoleWrite( "$oPane2 OK" & @CRLF )



    ; --- Find window/control ---

    ConsoleWrite( "--- Find window/control --- SysListView32" & @CRLF )

    Local $pCondition2
    $oUIAutomation.CreatePropertyCondition( $UIA_ClassNamePropertyId, "SysListView32", $pCondition2 )
    If Not $pCondition2 Then Return ConsoleWrite( "$pCondition2 ERR" & @CRLF )
    ConsoleWrite( "$pCondition2 OK" & @CRLF )

    Local $pList1, $oList1
    $oPane2.FindFirst( $TreeScope_Descendants, $pCondition2, $pList1 )
    $oList1 = ObjCreateInterface( $pList1, $sIID_IUIAutomationElement, $dtag_IUIAutomationElement )
    If Not IsObj( $oList1 ) Then Return ConsoleWrite( "$oList1 ERR" & @CRLF )
    ConsoleWrite( "$oList1 OK" & @CRLF )



    ; --- Find window/control ---

    ConsoleWrite( "--- Find window/control --- Chrome" & @CRLF )

    Local $pCondition3
    $oUIAutomation.CreatePropertyCondition( $UIA_NamePropertyId, "Chrome", $pCondition3 )
    If Not $pCondition3 Then Return ConsoleWrite( "$pCondition3 ERR" & @CRLF )
    ConsoleWrite( "$pCondition3 OK" & @CRLF )

    Local $pListItem1, $oListItem1
    $oPane2.FindFirst( $TreeScope_Descendants, $pCondition3, $pListItem1 )
    $oListItem1 = ObjCreateInterface( $pListItem1, $sIID_IUIAutomationElement, $dtag_IUIAutomationElement )
    If Not IsObj( $oListItem1 ) Then Return ConsoleWrite( "$oListItem1 ERR" & @CRLF )
    ConsoleWrite( "$oListItem1 OK" & @CRLF )


EndFunc

 

I'm getting an $oPane2 error so maybe I'm traversing the tree incorrectly or missing something else.

[topten]

Thank you very much for such useful library- it does magic!

I am automating ff and it works very well - I can rich most of elements, click there and send text.

But how can I handle possible errors? For example, if some input is not visible yet and I would need to scroll the screen. How can I understand, that the input already visible?

Thanx in advance @junkew

[junkew]

Automating a GUI is not easy certainly detecting if an element is not yet visible as its allways unclear how long to wait.

Easiest is to make a loop for 10 times with a sleep(250) or something you tweak that test(s) for existence/visibility and if tried for n times then fail.