Question on Controls / Targeting with AutoIT Info Finder

[zenocon]

Hello,

I'm trying to write a script to automate part of what appears to be a Winforms app.  At least when I use AutoIT Info and drop the finder on it, it lists the Window's class as: WindowsForms10.Windows.8.app.0.3551b1b_r14_ad2.

The bizarre thing is some of the controls don't seem to be targetable.  See the screen shot attached with what appears to show a WinForms GroupBox with the title "Command Window".  Inside the group box it appears to have a Label, and TextBox, and a Button.  Using the AutoIT Info Finder, I'm not able to individually address the inner components.  I can only seem to target the GroupBox. 

Attached are the results of the Finder/Control window.  Notice that it shows the control has class SWHGRP, in Advanced Mode: [CLASS:SWHGRP;INSTANCE:1].  My assumption here, though is that this is the GroupBox control.  What I want to do is automate entering some text into the TextBox in there and then clicking the button.  

If this is just a standard Winforms app, it seems odd to me that these inner controls would not be targetable, but I'm pretty new to AutoIT (although certainly not new to software development).  Using a standard mouse, I can certainly click in the TextBox, enter some text and click the button, so they are legit controls.

• Can someone help me understand why some controls like this appear to not be able to be targeted by the Finder? 
• If I write my own script can I just enumerate all the children and see if I can find it that way?  Is it worth trying to do this, or if the AutoIT Info Finder cannot address them, then that's likely definitive and I'm out of luck?
• Is it possible they may have some kind of 3rd party control embedded in there like a webview -- and this might be what is causing the issue?

 

 

[zenocon]

So, I've done a lot of reading.  And I've started playing with the IUIAotmation scripts.  I will try the `simplespy` script and see if I can target those controls using the UIAutomation integration.  If there are any other tools that would be useful for introspecting windows apps / components, I'd love to hear about them -- most of the links posted in here are broken or dead, so it is a long haul to try to get updated information.

[zenocon]

Just now, zenocon said:

So, I've done a lot of reading.  And I've started playing with the IUIAotmation scripts.  I will try the `simplespy` script and see if I can target those controls using the UIAutomation integration.  If there are any other tools that would be useful for introspecting windows apps / components, I'd love to hear about them -- most of the links posted in here are broken or dead, so it is a long haul to try to get updated information.

Correction: I've tried the simplespy script locally and it works fine, but I don't have easy access to the target app/controls I want to get at.  I have to physically drive to another location and try it on-site, so I'm trying to do as much homework as I can to ensure the trips are worth it, if that makes sense.

[LarsJ]

Hello Mr. zenocon and welcome to our forums.

The "AutoIt Window Info" tool generally detects standard Windows controls in the Control Library, but has only very limited support for Windows Forms controls in .NET Framework.

The idea of enumerating the child windows will not really help. The controls are still Windows Forms controls and "AutoIt Window Info" tool will probably not give much information.

That some kind of 3rd party code is the reason why you cannot identify the controls is unlikely. The reason is simply the "AutoIt Window Info" tool's lack of support for these controls.

The best chance to identify the controls is the UI Automation framework and the "Simple spy tool". Microsoft's answer to this tool is Inspect.exe which is included in the Windows SDK. You can also download a copy in this post.

[zenocon]

Hi LarsJ -- Thanks much for the response.  I have made some further progress.  I have tried the Simple Spy Tool, but it also was not seemingly able to target the text box control I am hoping to interact with.  So, I tried Microsoft's inspect.exe that I found in the forums here and that can identify it.  The information is below from that tool.

As you can see, I know it *can* be automated since it is showing that is how I found it.  I did it through the Actions menu on that tool, and executed the Value.setValue action (which worked).  So I thought I had it easy b/c it seems to have an AutomationId, and I thought well maybe I can use that to directly address it.  However, my attempts to do so below were not successful, so I'm hoping someone else might be able to show me the err of my ways (code below)

expandcollapse popup

How found:  Value.SetValue...
Name:   "Tag:"
ControlType:    UIA_EditControlTypeId (0xC354)
LocalizedControlType:   "edit"
BoundingRectangle:  {l:595 t:521 r:699 b:541}
IsEnabled:  true
IsOffscreen:    false
IsKeyboardFocusable:    true
HasKeyboardFocus:   false
AccessKey:  ""
ProcessId:  6216
RuntimeId:  [2A.440654]
AutomationId:   "27"
FrameworkId:    "Win32"
ClassName:  "Edit"
NativeWindowHandle: 0x440654
IsContentElement:   true
ProviderDescription:    "[pid:7432,hwnd:0x440654 Main:Nested [pid:6216,hwnd:0x440654 Annotation(parent link):Microsoft: Annotation Proxy (unmanaged:uiautomationcore.dll); Main:Microsoft: MSAA Proxy (unmanaged:uiautomationcore.dll)]; Hwnd(parent link):Microsoft: HWND Proxy (unmanaged:uiautomationcore.dll)]"
IsPeripheral:   [Not supported]
LiveSettingProperty:    [Not supported]
IsPassword: false
HelpText:   ""
FlowsFrom:  [Not supported]
OptimizeForVisualContent:   [Not supported]
PositionInSet:  [Not supported]
SizeOfSet:  [Not supported]
Level:  [Not supported]
AnnotationTypes:    [Not supported]
AnnotationObjects:  [Not supported]
LandmarkType:   [Not supported]
LocalizedLandmarkType:  [Not supported]
FullDescription:    [Not supported]
Annotation.AnnotationTypeName:  [Not supported]
Annotation.AnnotationTypeId:    [Not supported]
Annotation.Author:  [Not supported]
Annotation.DateTime:    [Not supported]
Annotation.Target:  [Not supported]
Drag.DropEffect:    [Not supported]
Drag.DropEffects:   [Not supported]
Drag.GrabbedItems:  [Not supported]
Drag.IsGrabbed: [Not supported]
DropTarget.DropTargetEffect:    [Not supported]
DropTarget.DropTargetEffects:   [Not supported]
LegacyIAccessible.ChildId:  0
LegacyIAccessible.DefaultAction:    ""
LegacyIAccessible.Description:  ""
LegacyIAccessible.Help: ""
LegacyIAccessible.KeyboardShortcut: ""
LegacyIAccessible.Name: "Tag:"
LegacyIAccessible.Role: editable text (0x2A)
LegacyIAccessible.State:    focusable (0x100000)
LegacyIAccessible.Value:    "dv"
ObjectModel.UnderlyingObjectModel:  [Error: calling getter for this property: hr=0xFFFFFFFF80070057 - The parameter is incorrect.]
Scroll.HorizontallyScrollable:  false
Scroll.HorizontalScrollPercent: 0.000000
Scroll.HorizontalViewSize:  100.000000
Scroll.VerticallyScrollable:    false
Scroll.VerticalScrollPercent:   0.000000
Scroll.VerticalViewSize:    100.000000
SpreadsheetItem.AnnotationObjects:  [Not supported]
SpreadsheetItem.AnnotationTypes:    [Not supported]
SpreadsheetItem.Formula:    [Not supported]
Style.ExtendedProperties:   [Not supported]
Style.FillColor:    [Not supported]
Style.FillPatternColor: [Not supported]
Style.FillPatternStyle: [Not supported]
Style.Shape:    [Not supported]
Style.StyleId:  [Not supported]
Style.StyleName:    [Not supported]
Value.IsReadOnly:   false
Value.Value:    "dv"
Transform2.CanZoom: [Not supported]
Transform2.ZoomLevel:   [Not supported]
Transform2.ZoomMinimum: [Not supported]
Transform2.ZoomMaximum: [Not supported]
IsAnnotationPatternAvailable:   [Not supported]
IsDragPatternAvailable: [Not supported]
IsDockPatternAvailable: false
IsDropTargetPatternAvailable:   [Not supported]
IsExpandCollapsePatternAvailable:   false
IsGridItemPatternAvailable: false
IsGridPatternAvailable: false
IsInvokePatternAvailable:   false
IsItemContainerPatternAvailable:    false
IsLegacyIAccessiblePatternAvailable:    true
IsMultipleViewPatternAvailable: false
IsObjectModelPatternAvailable:  [Not supported]
IsRangeValuePatternAvailable:   false
IsScrollItemPatternAvailable:   false
IsScrollPatternAvailable:   true
IsSelectionItemPatternAvailable:    false
IsSelectionPatternAvailable:    false
IsSpreadsheetItemPatternAvailable:  [Not supported]
IsSpreadsheetPatternAvailable:  [Not supported]
IsStylesPatternAvailable:   [Not supported]
IsSynchronizedInputPatternAvailable:    false
IsTableItemPatternAvailable:    false
IsTablePatternAvailable:    false
IsTextChildPatternAvailable:    [Not supported]
IsTextEditPatternAvailable: [Not supported]
IsTextPatternAvailable: false
IsTextPattern2Available:    [Not supported]
IsTogglePatternAvailable:   false
IsTransformPatternAvailable:    false
IsTransform2PatternAvailable:   [Not supported]
IsValuePatternAvailable:    true
IsVirtualizedItemPatternAvailable:  false
IsWindowPatternAvailable:   false
IsCustomNavigationPatternAvailable: [Not supported]
FirstChild: [null]
LastChild:  [null]
Next:   "COMMAND:" text
Previous:   "Tag:" text
Other Props:    Object has no additional properties
Children:   Container has no children
Ancestors:  "Dispatch (DSP)" pane
    "" pane
    "http://redacted/
    "" pane
    "" pane
    "" pane
    "" pane
    "" pane
    "" pane
    "" pane
    "" pane
    "" pane
    "" pane
    "" tab
    "" pane
    "" pane
    "" pane
    "" pane
    "" pane
    "" pane
    "" pane
    "" pane
    "" pane
    "" pane
    "" pane
    "Service/STG - Dispatching - redacted - redacted Drive" window
    "Desktop" pane
    [ No Parent ]

Here is some simple code I have tried:

 

#include "../uia/UIA_V0_64/UIAWrappers.au3"
#include "../uia/UIA_V0_64/CUIAutomation2.au3"
AutoItSetOption("MustDeclareVars", 1)

Example()

Func Example()
    Local $mainApp = _UIA_getObjectByFindAll($UIA_oDesktop, "AutomationId:=Portal", $treescope_children)
    If Not IsObj($mainApp) Then _WriteErrorLog( "$mainApp ERR" & @CRLF )
    _WriteErrorLog( "$mainApp OK" & @CRLF )

    Local $editbox = _UIA_getObjectByFindAll($mainApp, "AutomationId:=27", $treescope_subtree)
    If Not IsObj($editbox) Then _WriteErrorLog( "$editbox ERR" & @CRLF )
    _UIA_Action($editbox,"setfocus")
    _UIA_Action($editbox,"highlight")
    Sleep(1000)
    _UIA_Action($editbox,"sendkeys","DV")
    _UIA_Action($editbox,"sendkeys","{ENTER}")
EndFunc

Func _WriteErrorLog($ErrorMessage)
    ConsoleWrite($ErrorMessage)
    FileWriteLine(@ScriptDir & "\" & @ScriptName & ".log", @HOUR & ":" & @MIN & ":" & @SEC & ": " & $ErrorMessage)
EndFunc   ;==>WriteErrorLog

It did not work as expected -- strangely enough it showed a completely different highlighted portion of the app, and the error log also indicated that the $editbox was not an object, which is just confusing.  $mainApp prints ok, though.  I'm using the AutomationId:=Portal on that which I gleaned through inspect.ext tool.  It seems to be an OK selector for the target app.

But here, I'm wondering why I'm not able to successfully target the text box with it's listed AutomationId:=27, or somehow it is confusing it with another?

Given the dump above of the element I want to automate, can anyone recommend a more robust selector that will allow me to interact with the control.

Thank you in advance

[junkew]

Use name property to be Tag: 

Or use another object together with indexrelative property

See examples.zip with different examples of name:=value pair examples including indexrelative property. Best to calculate with inspect.exe

Post simplespy output and tell what highlights. Most likely z-order of elements is incorrect and as such object from point gives incorrect result.