IUIAutomation MS framework automate chrome, FF, IE, ....

[junkew]

one impediment at a time.  

UIAutomation is recognizing your object

• If its slow then it means properties are not good enough and UIAwrappers try to find in a dynamic way which can mean scnanning 100's of objects and take the best guess.
• Advice is to look at the treewalker examples as the wrappers are only creating an algorithm around that (which can have issues)
  #32770 windows is normally child of the desktop
• So walk the tree and highlight all elements.

I will check later for you if i can make a small coding example (but those are based on the treewalker examples anyway so please try yourself)

 

[junkew]

Your screenshots are not reachable
Another test
Start chrome, menu tools, save as. Then you have a #32770 on screen
In my W10 system its a child of Chrome_WidgetWin_1 and then below works

_UIA_setVar("oP1","title:=Nieuw tabblad;class:=Chrome_WidgetWin_1")
_UIA_Action("oP1","highlight")
_UIA_Action("oP1","setfocus")

_UIA_setVar("oP2","class:=#32770")
_UIA_Action("oP2","highlight")
_UIA_Action("oP2","setfocus")

If I do not have the #32770 with right parent structure I see 

• deep find in subtree class:=#32770;RTI.SEARCHCONTEXT

This deep finding you can prevent by adding a return in uiawrappers just after (but offcourse then it will not try to search thru all objects of the system)

If Not _UIA_IsElement($oElement) Then

                consolewrite("  deep find in subtree " & $tPhysical & ";" & $strStartElement & @CRLF)
                _UIA_LOG("  deep find in subtree " & $tPhysical & @CRLF, $UIA_Log_Wrapper)
;~              return
                $oElement = _UIA_getObjectByFindAll($oStart, $tPhysical, $treescope_subtree)
            EndIf

Try this with some trial/error and maybe even remove title property if you are sure its the only Qt application
As in each step it will just search childs you can start at highest level (below desktop) for first QWidget

_UIA_setVar("oP1","title:=CLO.*;class:=QWidget")
_UIA_Action("oP1","highlight")
_UIA_Action("oP1","setfocus")

_UIA_setVar("oP2","class:=QWidget")
_UIA_Action("oP2","highlight")
_UIA_Action("oP2","setfocus")

by counting / index number you can also reach your window but all a little more work

_UIA_setVar("oP1","class:=QWidget;index:=1")

Run and try to understand these

• ex1_Treewalker.au3
• Dump childs to a log file (see LOGS folder thats created in folder of your script) and inspect the details 

• _UIA_DumpThemAll($UIA_oDesktop,$treescope_children)

   

 

[Yoav]

Hi...

This is a screenshot of the window I'm trying to control: https://www.dropbox.com/s/xmnarbil99ddtqw/screenshot.png?dl=0

There's a big button there (with the label "Scan Sample") which I can control just fine, if I do "UIA_Action" - "Click" on it, it clicks properly.

My problem is with the radio buttons in the middle. If I use simplespy, I get the following code. I can see that it finds the radio button properly, highlights it and everything, but then the click doesn't change the selection of the radio button...  (of course that manually clicking on this UI element does work). There's no errors reported in the logs, it just says that it's clicking on the UI element.

This is the code I got from simplespy. Any tips would be highly appreciated. Thanks.

expandcollapse popup

;~ *** Standard code maintainable ***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

_UIA_setVar("oP1","Title:=NIRscan Nano GUI v2.0.2;controltype:=UIA_WindowControlTypeId;class:=Qt5QWindowIcon;instance:=1")  ;NIRscan Nano GUI v2.0.2
_UIA_setVar("oP2","Title:=;controltype:=UIA_CustomControlTypeId;class:=")   ;
_UIA_setVar("oP3","Title:=;controltype:=UIA_PaneControlTypeId;class:=") ;
_UIA_setVar("oP4","Title:=;controltype:=UIA_CustomControlTypeId;class:=")   ;
_UIA_setVar("oP5","Title:=;controltype:=UIA_CustomControlTypeId;class:=")   ;
_UIA_setVar("oP6","Title:=;controltype:=UIA_CustomControlTypeId;class:=")   ;
_UIA_setVar("oP7","Title:=;controltype:=UIA_PaneControlTypeId;class:=") ;
_UIA_setVar("oP8","Title:=;controltype:=UIA_CustomControlTypeId;class:=")   ;
_UIA_setVar("oP9","Title:=Scan Config Select;controltype:=UIA_CustomControlTypeId;class:=") ;Scan Config Select
_UIA_setVar("oP10","Title:=Scan Config Select;controltype:=UIA_CustomControlTypeId;class:=")    ;Scan Config Select
_UIA_setVar("oP11","Title:=Scan Reference Select;controltype:=UIA_CustomControlTypeId;class:=") ;Scan Reference Select

;~ $oUIElement=_UIA_getObjectByFindAll("Previous.mainwindow", "title:=Previous;ControlType:=UIA_RadioButtonControlTypeId", $treescope_subtree)
_UIA_setVar("oUIElement","Title:=Previous;controltype:=UIA_RadioButtonControlTypeId;class:=") ;ControlType:=UIA_RadioButtonControlTypeId;classname:=")

;~ Actions split away from logical/technical definition above can come from configfiles

;~_UIA_Action("oP1","highlight")
_UIA_Action("oP1","setfocus")
;~_UIA_Action("oP2","highlight")
_UIA_Action("oP2","setfocus")
;~_UIA_Action("oP3","highlight")
_UIA_Action("oP3","setfocus")
;~_UIA_Action("oP4","highlight")
_UIA_Action("oP4","setfocus")
;~_UIA_Action("oP5","highlight")
_UIA_Action("oP5","setfocus")
;~_UIA_Action("oP6","highlight")
_UIA_Action("oP6","setfocus")
;~_UIA_Action("oP7","highlight")
_UIA_Action("oP7","setfocus")
;~_UIA_Action("oP8","highlight")
_UIA_Action("oP8","setfocus")
;~_UIA_Action("oP9","highlight")
_UIA_Action("oP9","setfocus")
;~_UIA_Action("oP10","highlight")
_UIA_Action("oP10","setfocus")
;~_UIA_Action("oP11","highlight")
_UIA_Action("oP11","setfocus")

;~_UIA_action($oUIElement","highlight")
;~_UIA_action($oUIElement,"click")
;~_UIA_action("oUIElement","highlight")
_UIA_action("oUIElement","click")

 

[junkew]

To hard to say from a distance. Most likely you only need reference to p1 p11 and uielement. Check also findall treescope descendants to search all levels.

The algorithm i made is not perfect so could be you have to fallback to treewalker examples to get to right element. Spy suggests you are 10 or more levels deep but some you can skip. Try first highligt p1 then p2 and so onwards. Things not highlighting you leave out.

You could try to play with index or instance property to be 1 or 2 or 3 as sometimes same element exists hidden.

[Yoav]

I have a button that I need to click on from time to time, it makes no sense to search for it every time. 
what I want to do is to have a script that finds the oUIElement object, stores it somewhere, and then I'll have my main script that I run, which will directly access that object.
So I want my pseudo code to look like:

$obj1 = fetchObject([object 1])
_UIA_action($obj1,"click")

$obj2 = fetchObject([object 2])
_UIA_action($obj2,"click")

 

Can that be done?

[junkew]

yes, simplespy shows you the examples (there are multiple ways)

With the wrappers

_UIA_setVar("oP1","Title:=<yourtitle>;controltype:=UIA_WindowControlTypeId;class:=<yourclass>)  

$myObject=_UIA_action("oP1","object")

Or

with the findall / treewalker examples as given in the zip of post#1

[HudsonKane]

We are using the most recent version of UIAWrappers.au3 (6.3 10/07/2016)

In the function _UIA_getObjectByFindAll() line 1058

;- If not an object given directly then
    If $iMatch = 0 Then
;~ Get the exceptional properties with special meaning
        For $i = 1 To $arrSize
            $propName = $properties2Match[$i][3]

Should $propName be set to $properties2Match[$i][0] instead?

It appears to be the case and fixes some issues for us.

Thank you

[junkew]

I think you are right as the definition of those 4 is as below. Which behavior did you have

$asProperties2Match[$i][0] = $propName
            $asProperties2Match[$i][1] = $propValue
            $asProperties2Match[$i][2] = $iMatch
            $asProperties2Match[$i][3] = $bSkipSpecialProperty

 

and array is incorrect dimension  change 2nd dimension from 2 to 4 but unsure why that does not give compile or runtime errors

Local $properties2Match[1][2] ;~ All properties of the expression to match in a normalized form

 

[HudsonKane]

Our script would find the item it was looking for, but we could not get any properties of the object returned.

[junkew]

Updated first post with latest version V0.64 (many small essential tweaks and fixes)

 

@HudsonKane 

1. Fixed the issues you gave also in post 1
and
2 Not sure what you mean but  
This gives all properties

_UIA_getAllPropertyValues($UIA_oUIElement)

and this to retrieve a single property

$UIA_oUIElement.GetCurrentPropertyValue($id, $tmpValue)

and you can only get all Javascript/html by using bookmarklets thru addressbar (type javascript:......  in your addressbar and sky will be the limit)

[HudsonKane]

Thank you very much, we will start working with this version right away and let you know what we find out.

 

[Yoav]

On 9/6/2017 at 1:32 PM, junkew said:

yes, simplespy shows you the examples (there are multiple ways)

With the wrappers

_UIA_setVar("oP1","Title:=<yourtitle>;controltype:=UIA_WindowControlTypeId;class:=<yourclass>)  

$myObject=_UIA_action("oP1","object")

Or

with the findall / treewalker examples as given in the zip of post#1

Thank you, but I guess my problem is not exactly that. If I to a "setVar" and store the control in a var that's all well and good, but I then need to be able to call a script, a while later, and this script is supposed to be able to access this var (after the first script that found the control already exited). So I need this setvar to be stored somewhere so I can retrieve it later. Is that possible? Or do I need to keep the first script running all the time?

 

[junkew]

@Yoavi do not understand why refinding it would be an issue in the 2nd script. First script could remind the automationid which 2nd script uses to find it.

[Yoav]

20 hours ago, junkew said:

@Yoavi do not understand why refinding it would be an issue in the 2nd script. First script could remind the automationid which 2nd script uses to find it.

I will try to better explain myself.

I have an "initialization" script, which first finds the button. That script can successfully use:

_UIA_setVar("oP1","Title:=<yourtitle>;controltype:=UIA_WindowControlTypeId;class:=<yourclass>)  
$myObject=_UIA_action("oP1","object")

That same initialization script can even do (just for the sake of testing), either

_UIA_action($myObject,"highlight")

or

_UIA_action("oP1","highlight")

And both would work.

That initialization script then terminates. At a later time, I would like to call a new script. The sole purpose of this new script is to press that button I previously found.

That second script needs to somehow restore $myObject from somewhere. Is the value stored somewhere when I'm using _UIA_action("oP1","object") ? How do I retrieve it then?

 

I appreciate the patience. I realize it's probably something very basic here that I'm missing.

 

Thank you.

 

[junkew]

No, the found object in script1 does not stay in memory ready for scrip2.

• When first script ends nothing is there to refer to in 2nd script so 
  you have to refind it. you can find it either with same description as used in script1 or
  you make sure you save the automationid property (Global Const $UIA_AutomationIdPropertyId=30011) in script 1 in an ini file and read that value back in script 2 and use then 

_UIA_SetVar("oP1","AutomationId=" & $retrievedIniValue)

• Both ways leads to a refind where I am not sure if the speeddifference justifies finding by automationid.
• Only when you are in need of n scripts you could build an advanced mainscript that keeps references in memory and the other scripts communicate with first script doing all the actions (although I do not see a need to make it that complicated). Why you are not handling it iall n 1 script?

 

 

 

[Yoav]

15 hours ago, junkew said:

No, the found object in script1 does not stay in memory ready for scrip2.

• When first script ends nothing is there to refer to in 2nd script so 
  you have to refind it. you can find it either with same description as used in script1 or
  you make sure you save the automationid property (Global Const $UIA_AutomationIdPropertyId=30011) in script 1 in an ini file and read that value back in script 2 and use then 

_UIA_SetVar("oP1","AutomationId=" & $retrievedIniValue)

• Both ways leads to a refind where I am not sure if the speeddifference justifies finding by automationid.
• Only when you are in need of n scripts you could build an advanced mainscript that keeps references in memory and the other scripts communicate with first script doing all the actions (although I do not see a need to make it that complicated). Why you are not handling it iall n 1 script?

 

Thank you, that's what I was afraid of... I would do it all in one script, but I don't know if it's possible in autoit to have a script running and then waiting to be called again without creating another instance and in such a way that the same instance is simply invoked with a possibility to access the previously defined and populated variables. We're getting a little off-topic, perhaps I'll proceed in a more appropriate forum category...

 

Thanks.

[masCh]

Do you have something for Windows 10 that will look for Steam icon on the hidden part of system tray, right click it and select Exit, without using ProcessClose()?

[junkew]

I do not know you (as i do not have steam) should just try with Simplespy and inspect.exe if you can see the elements highlight. If you can you can scan them with the treewalkers. Within examples there are examples on finding and clicking clock in the systemtray.

[TheDcoder]

Hello again. I am playing around with UI Automation again. Maybe I found a potential bug in this UDF

When I use LegacyIAccessibleName with _UIA_getObjectByFindAll it is not working, but when I try to iterate through the objects I can find the correct element using the Legacy Name.

Here is an example:

#include "UIA\UIAWrappers.au3"

Local $oChrome=_UIA_getObjectByFindAll($UIA_oDesktop, "controltype:=UIA_WindowControlTypeId;class:=Chrome_WidgetWin_1", $treescope_children)
Local $oUIElement=_UIA_getObjectByFindAll($oChrome, "title:=Play;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)
_UIA_action($oUIElement,"highlight")

$oMuteButton = _UIA_getObjectByFindAll($oChrome, 'LegacyIAccessibleName:=Mute;ControlType:=UIA_ButtonControlTypeId')
MsgBox(0, "Highlight", "Using _UIA_getObjectByFindAll to find object and highlight")
_UIA_action($oMuteButton,"highlight")
MsgBox(0, "Highlight", "Using GetSiblingObjectByLegacyName to find object and highlight")
$oMuteButton = GetSiblingObjectByLegacyName($oUIElement, "Mute")
_UIA_action($oMuteButton,"highlight")

Func GetSiblingObjectByLegacyName($oSiblingObject, $sName)
    $UIA_oUIAutomation.RawViewWalker($UIA_pTW)

    $oTW = ObjCreateInterface($UIA_pTW, $sIID_IUIAutomationTreeWalker, $dtagIUIAutomationTreeWalker)

    While IsObj($oSiblingObject) = True
        $oTW.GetNextSiblingElement($oSiblingObject, $UIA_pUIElement)
        $oSiblingObject = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
        If _UIA_getPropertyValue($oSiblingObject,  $UIA_LegacyIAccessibleNamePropertyId) = $sName Then Return $oSiblingObject
    WEnd
EndFunc

In this example I made a custom function which iterates through all of the sibling objects for a given object and finds the correct object by the Legacy Name... and guess what? _UIA_getObjectByFindAll cannot find it, I wonder why? You can run this example by opening a youtube video in chrome, it should highlight the play button and the mute button.

Thanks for the help in advance, TD

[junkew]

Seems not to be in my wrappers. Seems to be a difference between findall and treewalkers and implementation of Google Chrome.

Interesting to study later how this goes for IE and FireFox on YouTube

• This works also for the play button using LegacyIAccessibleName but not for the mute button

;~ Local $oUIElement=_UIA_getObjectByFindAll($oChrome, "title:=Play;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)
Local $oUIElement=_UIA_getObjectByFindAll($oChrome, "LegacyIAccessibleName:=Play;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)

• When I try with simplespy
  • on calc.exe all seems to function fine also with LegacyIAccessibleName
  • When I try with youtube and simplespy the LegacyIAccessibleName is NOT coming for mute but it is filled for play button
  • When I try your example with the treewalker (all 3 different ones tried) it finds it
• Debugging in uiaWrappers with some consolewrite does not give me a clue either

  $tmpStr = _UIA_getBasePropertyInfo($itUIA_oUIElement)
              _UIA_LOG(" Element: " & $i & " " & $tmpStr & @CRLF, $UIA_Log_Wrapper)
              consolewrite(" Element: " & $i & " " & $tmpStr & @CRLF, $UIA_Log_Wrapper)

  extending this function to show the legacy name

  Func _UIA_getBasePropertyInfo($oUIElement)
      Local $title = _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId)
      Local $class = _UIA_getPropertyValue($oUIElement, $uia_classnamepropertyid)
      Local $controltypeName = _UIA_getControlName(_UIA_getPropertyValue($oUIElement, $UIA_ControlTypePropertyId))
      Local $controltypeId = _UIA_getPropertyValue($oUIElement, $UIA_ControlTypePropertyId)
      Local $nativeWindow = _UIA_getPropertyValue($oUIElement, $UIA_NativeWindowHandlePropertyId)
      Local $controlRect = _UIA_getPropertyValue($oUIElement, $UIA_BoundingRectanglePropertyId)
      Local $acceleratorkey = _UIA_getPropertyValue($oUIElement, $UIA_AcceleratorKeyPropertyId)
      Local $automationid = _UIA_getPropertyValue($oUIElement, $UIA_AutomationIdPropertyId)
      Local $LegacyIAccessibleName= _UIA_getPropertyValue($oUIElement,  $UIA_LegacyIAccessibleNamePropertyId)
      Return "Title is: <" & $title & ">" & @TAB _
               & "Class   := <" & $class & ">" & @TAB _
               & "controltype:= " & "<" & $controltypeName & ">" & @TAB _
               & ",<" & $controltypeId & ">" & @TAB _
               & ", (" & Hex($controltypeId) & ")" & @TAB _
               & "rect := < " & $controlRect & ">" & @TAB _
               & "hwnd := < " & $nativeWindow & ">" & @TAB _
               & "acceleratorkey := < " & $acceleratorkey & ">" & @TAB _
               & "LegacyIAccessibleName := <" & $LegacyIAccessibleName & ">" & @TAB _
               & "automationid := <" & $automationid & ">" & @CRLF
  
  EndFunc   ;==>_UIA_getBasePropertyInfo