_UIA_getObjectByFindAll does not find $oP0 / simplespy does not disambiguate objects with similar properties [Bug]&[Workaround]

matt55777 · April 29, 2019

Hi,
I am automating a powerpoint addon. Runs fine so far except at some point I need to click an object but the simplespy code fails to find the $oP0 object. The code below

Local $oP4=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=Présentation1 - PowerPoint;controltype:=UIA_WindowControlTypeId;class:=PPTFrameClass", $treescope_children)
_UIA_Action($oP4,"setfocus")
Local $oP3=_UIA_getObjectByFindAll($oP4, "Title:=Office Timeline;controltype:=UIA_WindowControlTypeId;class:=Window", $treescope_children)
_UIA_Action($oP3,"setfocus")
Local $oP2=_UIA_getObjectByFindAll($oP3, "Title:=;controltype:=UIA_ListControlTypeId;class:=ListView", $treescope_children)
Local $oP1=_UIA_getObjectByFindAll($oP2, "Title:=Otl.Views.CustomControls.WizardPage;controltype:=UIA_ListItemControlTypeId;class:=ListBoxItem", $treescope_children)
Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=Data;controltype:=UIA_GroupControlTypeId;class:=Expander", $treescope_children)
;~ First find the object in the parent before you can do something
;~$oUIElement=_UIA_getObjectByFindAll("Data.mainwindow", "title:=Data;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)
Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=Data;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)

Consolewrite("$oUIElement obj ? " & isobj($oUIElement) & " P0 " & isobj($oP0) & " P1 " & isobj($oP1) & _
              " P2 " & isobj($oP2) & " P3 " & isobj($oP3) & " P4 " & isobj($oP4))

returns:

Console:
  >>> $oUIElement obj ? 0 P0 0 P1 1 P2 1 P3 1 P4 1

So $oP4 - $oP1 are ok but $oP0 is not found by _UIA_getObjectByFindAll.

The inspect tool from Microsoft gives me the following for object $oP0:

How found:  Selected from tree...
Name:   "Data"
ControlType:    UIA_GroupControlTypeId (0xC36A)
LocalizedControlType:   "group"
BoundingRectangle:  {l:1129 t:84 r:1165 b:684}
IsEnabled:  true
IsOffscreen:    false
IsKeyboardFocusable:    true
HasKeyboardFocus:   false
AcceleratorKey: ""
AccessKey:  ""
ProcessId:  11580
RuntimeId:  [7.2D3C.16F6631]
AutomationId:   "PART_Expander"
FrameworkId:    "WPF"
ClassName:  "Expander"
IsControlElement:   true
IsContentElement:   true
ProviderDescription:    "[pid:11580,hwnd:0x0 Main(parent link):Unidentified Provider (managed:MS.Internal.Automation.ElementProxy, PresentationCore, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)]"
IsPeripheral:   [Not supported]
LiveSettingProperty:    [Not supported]
IsPassword: false
ItemStatus: ""
ItemType:   ""
IsRequiredForForm:  false
HelpText:   ""
FlowsFrom:  [Not supported]
ClickablePoint: {x:1147 y:384}
Orientation:    0
OptimizeForVisualContent:   [Not supported]
PositionInSet:  [Not supported]
SizeOfSet:  [Not supported]
Level:  [Not supported]
AnnotationTypes:    [Not supported]
AnnotationObjects:  [Not supported]
LandmarkType:   [Not supported]
LocalizedLandmarkType:  [Not supported]
FullDescription:    [Not supported]
Annotation.AnnotationTypeName:  [Not supported]
Annotation.AnnotationTypeId:    [Not supported]
Annotation.Author:  [Not supported]
Annotation.DateTime:    [Not supported]
Annotation.Target:  [Not supported]
Drag.DropEffect:    [Not supported]
Drag.DropEffects:   [Not supported]
Drag.GrabbedItems:  [Not supported]
Drag.IsGrabbed: [Not supported]
DropTarget.DropTargetEffect:    [Not supported]
DropTarget.DropTargetEffects:   [Not supported]
ExpandCollapse.ExpandCollapseState: Collapsed (0)
LegacyIAccessible.ChildId:  0
LegacyIAccessible.DefaultAction:    "Développer"
LegacyIAccessible.Description:  ""
LegacyIAccessible.Help: ""
LegacyIAccessible.KeyboardShortcut: ""
LegacyIAccessible.Name: "Data"
LegacyIAccessible.Role: groupement (0x14)
LegacyIAccessible.State:    réduit,pouvant être actif (0x100400)
LegacyIAccessible.Value:    ""
ObjectModel.UnderlyingObjectModel:  [Error: calling getter for this property: hr=0xFFFFFFFF80070057 - Paramètre incorrect.]
SpreadsheetItem.AnnotationObjects:  [Not supported]
SpreadsheetItem.AnnotationTypes:    [Not supported]
SpreadsheetItem.Formula:    [Not supported]
Style.ExtendedProperties:   [Not supported]
Style.FillColor:    [Not supported]
Style.FillPatternColor: [Not supported]
Style.FillPatternStyle: [Not supported]
Style.Shape:    [Not supported]
Style.StyleId:  [Not supported]
Style.StyleName:    [Not supported]
Transform2.CanZoom: [Not supported]
Transform2.ZoomLevel:   [Not supported]
Transform2.ZoomMinimum: [Not supported]
Transform2.ZoomMaximum: [Not supported]
IsAnnotationPatternAvailable:   [Not supported]
IsDragPatternAvailable: [Not supported]
IsDockPatternAvailable: false
IsDropTargetPatternAvailable:   [Not supported]
IsExpandCollapsePatternAvailable:   true
IsGridItemPatternAvailable: false
IsGridPatternAvailable: false
IsInvokePatternAvailable:   false
IsItemContainerPatternAvailable:    false
IsLegacyIAccessiblePatternAvailable:    true
IsMultipleViewPatternAvailable: false
IsObjectModelPatternAvailable:  [Not supported]
IsRangeValuePatternAvailable:   false
IsScrollItemPatternAvailable:   false
IsScrollPatternAvailable:   false
IsSelectionItemPatternAvailable:    false
IsSelectionPatternAvailable:    false
IsSpreadsheetItemPatternAvailable:  [Not supported]
IsSpreadsheetPatternAvailable:  [Not supported]
IsStylesPatternAvailable:   [Not supported]
IsSynchronizedInputPatternAvailable:    true
IsTableItemPatternAvailable:    false
IsTablePatternAvailable:    false
IsTextChildPatternAvailable:    [Not supported]
IsTextEditPatternAvailable: [Not supported]
IsTextPatternAvailable: false
IsTextPattern2Available:    [Not supported]
IsTogglePatternAvailable:   false
IsTransformPatternAvailable:    false
IsTransform2PatternAvailable:   [Not supported]
IsValuePatternAvailable:    false
IsVirtualizedItemPatternAvailable:  false
IsWindowPatternAvailable:   false
IsCustomNavigationPatternAvailable: [Not supported]
FirstChild: "Data" button
LastChild:  "" button
Next:   [null]
Previous:   [null]
Other Props:    Object has no additional properties
Children:   "Data" button
    "Import from Excel" text
    "" custom
    "Data" text
    "" button
    "" button
    "" button
    "" button
    "" button
Ancestors:  "Otl.Views.CustomControls.WizardPage" list item
    "" list view
    "Office Timeline" window
    "Présentation1 - PowerPoint" fenêtre

~~How can I troubleshoot this?~~

EDIT :

It looks like the problem is that simplespy does not check for parents having the same name in the tree.

In my case, i had:

I want to press the "Data" button but as you can see there are 2 Otl.Views.CustomControls.WizardPage objects and _UIA_getObjectByFindAll selects the wrong one which explains why $oP0 is not found.

As a workaround, I added

ConsoleWrite(_UIA_getBasePropertyInfo($oUIElement))

to the findThemAll function to try to find a disambiguation variable. In my case, automationid := <DataSelectionPage> worked when I added it in _UIA_getObjectByFindAll

Local $oP1=_UIA_getObjectByFindAll($oP2, "AutomationId:=DataSelectionPage;Title:=Otl.Views.CustomControls.WizardPage;controltype:=UIA_ListItemControlTypeId;class:=ListBoxItem", $treescope_children)

Edited April 29, 2019 by matt55777
new information

junkew · April 29, 2019

Use index or indexrelative properties. See examples.zip how to disambuguate by index position. Thats one of the reasons for calling it a simplespy.

extended with example

Made with simplespy on calc
Added the run and winwaitactive
Extended the concept of oUIElement to oUIElement1,2,3 with index:=nn
Extended the highlghlighting of the buttons by addiing UIA_Action on the 3 elements

;~  *** Standard code maintainable ***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

Local $iPID = Run("calc.exe")
winwaitactive("[REGEXPTITLE:Calc*]")

_UIA_setVar("oP1","Title:=Calculator;controltype:=UIA_WindowControlTypeId;class:=ApplicationFrameWindow")   ;Calculator
_UIA_setVar("oP2","Title:=Calculator;controltype:=UIA_WindowControlTypeId;class:=Windows.UI.Core.CoreWindow")   ;Calculator
_UIA_setVar("oP3","Title:=;controltype:=UIA_GroupControlTypeId;class:=LandmarkTarget")  ;
_UIA_setVar("oP4","Title:=Number pad;controltype:=UIA_GroupControlTypeId;class:=NamedContainerAutomationPeer")  ;Number pad

;~ $oUIElement=_UIA_getObjectByFindAll("Seven.mainwindow", "title:=Seven;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)
_UIA_setVar("oUIElement1","controltype:=UIA_ButtonControlTypeId;class:=Button;index:=1") ;ControlType:=UIA_ButtonControlTypeId;classname:=Button")
_UIA_setVar("oUIElement2","controltype:=UIA_ButtonControlTypeId;class:=Button;index:=2") ;ControlType:=UIA_ButtonControlTypeId;classname:=Button")
_UIA_setVar("oUIElement3","controltype:=UIA_ButtonControlTypeId;class:=Button;index:=3") ;ControlType:=UIA_ButtonControlTypeId;classname:=Button")

;~ Actions split away from logical/technical definition above can come from configfiles

;~_UIA_Action("oP1","highlight")
_UIA_Action("oP1","setfocus")
;~_UIA_Action("oP2","highlight")
_UIA_Action("oP2","setfocus")
;~_UIA_Action("oP3","highlight")
_UIA_Action("oP3","setfocus")
;~_UIA_Action("oP4","highlight")
_UIA_Action("oP4","setfocus")

_UIA_action("oUIElement1","highlight")
;~_UIA_action("oUIElement","click")
_UIA_action("oUIElement2","highlight")

_UIA_action("oUIElement3","highlight")

Edited April 29, 2019 by junkew

matt55777 · April 30, 2019

Thanks for your patience, I have been using your framework only for a couple of days so I am still learning! I take note that index & indexrelative can also be used to disambiguate.

I think I remember from Microsoft site that automationID is guaranteed to be unique amongst siblings. It means that if it is defined then automationid is enough to choose the right node all the time, correct? If that is the case, maybe simplespy could be enhanced to include this in the code generator when automationid exists?!

junkew · April 30, 2019

I have never used automationid as i think its only unique for the running instance and other reason is that i prefer name or title stuff in it so i know myself which element it is. Future version i will put on todo to have it configurable per classtype

LarsJ · April 30, 2019

junkew, You should definitely read the documentation.

junkew · May 1, 2019

I did

Quote

AutomationIdProperty does not guarantee a unique identity throughout the tree; it typically needs container and scope information to be useful.

matt55777 · May 1, 2019

Another quote

Quote

When it is available the AutomationIdProperty of an element is expected to always be the same in any instance of the application regardless of the local language. The value should be unique among sibling elements but not necessarily unique across the entire desktop.

junkew · May 1, 2019

in same article you can read this is true for 1 build but not neccesarily for next (nightly) build. Close collaboration with developers is needed to have accessibility identification with any property. If you treat the app as a black box then in one case title,class is unique and in others you need more.

Feel free to modify simplespy to your need.

junkew · May 5, 2019

I looked thru my W10 system for AutomationId and see its not allways having a value but however I think there are some usefull purposes in automation to make use of this when its filled (I make then the assumption developers fill this properly)

Rules I will apply in future spy

If there iis an automation id use that and put in comments the alternative description (as is used now)

It will help a little in

Multilanguage application's
- Calc.exe:
  - name := ((Calculator)|(Rekenmachine)) as its not having an automationid
  - AutomationId:=equalsButton (instead of a regex)
- Windows itself is inconsistent directly on start and taskbar many are not having an automationid

Sign In

_UIA_getObjectByFindAll does not find $oP0 / simplespy does not disambiguate objects with similar properties [Bug]&[Workaround]

Recommended Posts

matt55777

Link to comment

Share on other sites

junkew

Link to comment

Share on other sites

matt55777

Link to comment

Share on other sites

junkew

Link to comment

Share on other sites

LarsJ

Link to comment

Share on other sites

junkew

Link to comment

Share on other sites

matt55777

Link to comment

Share on other sites

junkew

Link to comment

Share on other sites

junkew

Link to comment

Share on other sites

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members

Browse

AutoIt Resources

Release

Beta