IUIAutomation MS framework automate chrome, FF, IE, ....

[junkew]

Little hard to answer. See first post and read the given references. Google is your friend for other languages. Here for example on how to do from vba.

https://social.msdn.microsoft.com/Forums/en-US/21d7b351-b003-42a7-97b3-8d95cc298b2c/uiautomation-on-vbaexcel-macro?forum=isvvba

 

[Proph]

Thanks for all the hard work junkew!  I've tried a couple of times to wrap my brain around how this works.  But it is really confusing.  I'm sure once I understand it a bit more it will click and I'll be able to figure it out. LOL!

Basically what I am testing with now is Malwarebytes 3.0.6.  When it's first opened up a box pops up with a "Learn More" button and a "Close" button.

 

I tried SimpleSpy and it pulled some info from the window... but I don't see any info about the "Close" button which is what I had highlighted when I clicked CTRL + W.  I'll post the info it grabbed below.

When I used Inspect.exe I was able to see the "Close" info.  I'll post that info below as well.  Utalizing the info from SimpleSpy and Inspect... what autoit code would I utalize to left click on the "Close" button?

 

SimpleSpy:

expandcollapse popup

Mouse position is retrieved 738-585
At least we have an element [Dashboard Page][]
Having the following values for all properties: 
Title is: <Dashboard Page>    Class   := <>    controltype:= <UIA_PaneControlTypeId>    ,<50033>    , (0000C371)    
*** Parent Information ***
Title is: <Malwarebytes Premium Trial 3.0.6 Malwarebytes Premium Trial 3.0.6>    Class   := <Qt5QWindowIcon>    controltype:= <UIA_WindowControlTypeId>    ,<50032>    , (0000C370)    
*** Detailed properties of the highlighted element ***
UIA_AcceleratorKeyPropertyId :=
UIA_AccessKeyPropertyId :=
UIA_AriaPropertiesPropertyId :=
UIA_AriaRolePropertyId :=
UIA_AutomationIdPropertyId :=
UIA_BoundingRectanglePropertyId :=474;218;586;431
UIA_ClassNamePropertyId :=
UIA_ClickablePointPropertyId :=
UIA_ControllerForPropertyId :=
UIA_ControlTypePropertyId :=50033
UIA_CulturePropertyId :=0
UIA_DescribedByPropertyId :=
UIA_DockDockPositionPropertyId :=5
UIA_ExpandCollapseExpandCollapseStatePropertyId :=3
UIA_FlowsToPropertyId :=
UIA_FrameworkIdPropertyId :=
UIA_GridColumnCountPropertyId :=0
UIA_GridItemColumnPropertyId :=0
UIA_GridItemColumnSpanPropertyId :=1
UIA_GridItemContainingGridPropertyId :=
UIA_GridItemRowPropertyId :=0
UIA_GridItemRowSpanPropertyId :=1
UIA_GridRowCountPropertyId :=0
UIA_HasKeyboardFocusPropertyId :=False
UIA_HelpTextPropertyId :=
UIA_IsContentElementPropertyId :=True
UIA_IsControlElementPropertyId :=True
UIA_IsDataValidForFormPropertyId :=False
UIA_IsDockPatternAvailablePropertyId :=False
UIA_IsEnabledPropertyId :=True
UIA_IsExpandCollapsePatternAvailablePropertyId :=False
UIA_IsGridItemPatternAvailablePropertyId :=False
UIA_IsGridPatternAvailablePropertyId :=False
UIA_IsInvokePatternAvailablePropertyId :=True
UIA_IsItemContainerPatternAvailablePropertyId :=False
UIA_IsKeyboardFocusablePropertyId :=True
UIA_IsLegacyIAccessiblePatternAvailablePropertyId :=True
UIA_IsMultipleViewPatternAvailablePropertyId :=False
UIA_IsOffscreenPropertyId :=False
UIA_IsPasswordPropertyId :=False
UIA_IsRangeValuePatternAvailablePropertyId :=False
UIA_IsRequiredForFormPropertyId :=False
UIA_IsScrollItemPatternAvailablePropertyId :=False
UIA_IsScrollPatternAvailablePropertyId :=False
UIA_IsSelectionItemPatternAvailablePropertyId :=False
UIA_IsSelectionPatternAvailablePropertyId :=False
UIA_IsSynchronizedInputPatternAvailablePropertyId :=False
UIA_IsTableItemPatternAvailablePropertyId :=False
UIA_IsTablePatternAvailablePropertyId :=False
UIA_IsTextPatternAvailablePropertyId :=False
UIA_IsTogglePatternAvailablePropertyId :=False
UIA_IsTransformPatternAvailablePropertyId :=False
UIA_IsValuePatternAvailablePropertyId :=False
UIA_IsVirtualizedItemPatternAvailablePropertyId :=False
UIA_IsWindowPatternAvailablePropertyId :=False
UIA_ItemStatusPropertyId :=
UIA_ItemTypePropertyId :=
UIA_LabeledByPropertyId :=
UIA_LegacyIAccessibleChildIdPropertyId :=0
UIA_LegacyIAccessibleDefaultActionPropertyId :=SetFocus
UIA_LegacyIAccessibleDescriptionPropertyId :=
UIA_LegacyIAccessibleHelpPropertyId :=
UIA_LegacyIAccessibleKeyboardShortcutPropertyId :=
UIA_LegacyIAccessibleNamePropertyId :=Dashboard Page
UIA_LegacyIAccessibleRolePropertyId :=16
UIA_LegacyIAccessibleSelectionPropertyId :=
UIA_LegacyIAccessibleStatePropertyId :=1048576
UIA_LegacyIAccessibleValuePropertyId :=
UIA_LocalizedControlTypePropertyId :=pane
UIA_MultipleViewCurrentViewPropertyId :=0
UIA_MultipleViewSupportedViewsPropertyId :=
UIA_NamePropertyId :=Dashboard Page
UIA_NativeWindowHandlePropertyId :=0
UIA_OrientationPropertyId :=0
UIA_ProcessIdPropertyId :=6764
UIA_ProviderDescriptionPropertyId :=[pid:6764,hwnd:0x0 Main(parent link):Microsoft: MSAA Proxy (unmanaged:uiautomationcore.dll)]
UIA_RangeValueIsReadOnlyPropertyId :=True
UIA_RangeValueLargeChangePropertyId :=0
UIA_RangeValueMaximumPropertyId :=0
UIA_RangeValueMinimumPropertyId :=0
UIA_RangeValueSmallChangePropertyId :=0
UIA_RangeValueValuePropertyId :=0
UIA_RuntimeIdPropertyId :=
UIA_ScrollHorizontallyScrollablePropertyId :=False
UIA_ScrollHorizontalScrollPercentPropertyId :=0
UIA_ScrollHorizontalViewSizePropertyId :=100
UIA_ScrollVerticallyScrollablePropertyId :=False
UIA_ScrollVerticalScrollPercentPropertyId :=0
UIA_ScrollVerticalViewSizePropertyId :=100
UIA_SelectionCanSelectMultiplePropertyId :=False
UIA_SelectionIsSelectionRequiredPropertyId :=False
UIA_SelectionselectionPropertyId :=
UIA_SelectionItemIsSelectedPropertyId :=False
UIA_SelectionItemSelectionContainerPropertyId :=
UIA_TableColumnHeadersPropertyId :=
UIA_TableItemColumnHeaderItemsPropertyId :=
UIA_TableRowHeadersPropertyId :=
UIA_TableRowOrColumnMajorPropertyId :=2
UIA_TableItemRowHeaderItemsPropertyId :=
UIA_ToggleToggleStatePropertyId :=2
UIA_TransformCanMovePropertyId :=False
UIA_TransformCanResizePropertyId :=False
UIA_TransformCanRotatePropertyId :=False
UIA_ValueIsReadOnlyPropertyId :=True
UIA_ValueValuePropertyId :=
UIA_WindowCanMaximizePropertyId :=False
UIA_WindowCanMinimizePropertyId :=False
UIA_WindowIsModalPropertyId :=False
UIA_WindowIsTopmostPropertyId :=False
UIA_WindowWindowInteractionStatePropertyId :=0
UIA_WindowWindowVisualStatePropertyId :=0

[Proph]

Inspect:

expandcollapse popup

How found:    Selected from tree...
Name:    "Close"
ControlType:    UIA_ButtonControlTypeId (0xC350)
LocalizedControlType:    "button"
BoundingRectangle:    {l:887 t:708 r:997 b:746}
IsEnabled:    true
IsOffscreen:    false
IsKeyboardFocusable:    true
HasKeyboardFocus:    false
AccessKey:    ""
ProcessId:    6764
ProviderDescription:    "[pid:6764,hwnd:0x0 Main(parent link):Microsoft: MSAA Proxy (unmanaged:uiautomationcore.dll)]"
IsPassword:    false
HelpText:    ""
LegacyIAccessible.ChildId:    0
LegacyIAccessible.DefaultAction:    "Press"
LegacyIAccessible.Description:    ""
LegacyIAccessible.Help:    ""
LegacyIAccessible.KeyboardShortcut:    ""
LegacyIAccessible.Name:    "Close"
LegacyIAccessible.Role:    push button (0x2B)
LegacyIAccessible.State:    focusable (0x100000)
LegacyIAccessible.Value:    ""
IsAnnotationPatternAvailable:    false
IsDragPatternAvailable:    false
IsDockPatternAvailable:    false
IsDropTargetPatternAvailable:    false
IsExpandCollapsePatternAvailable:    false
IsGridItemPatternAvailable:    false
IsGridPatternAvailable:    false
IsInvokePatternAvailable:    true
IsItemContainerPatternAvailable:    false
IsLegacyIAccessiblePatternAvailable:    true
IsMultipleViewPatternAvailable:    false
IsObjectModelPatternAvailable:    false
IsRangeValuePatternAvailable:    false
IsScrollItemPatternAvailable:    false
IsScrollPatternAvailable:    false
IsSelectionItemPatternAvailable:    false
IsSelectionPatternAvailable:    false
IsSpreadsheetItemPatternAvailable:    false
IsSpreadsheetPatternAvailable:    false
IsStylesPatternAvailable:    false
IsSynchronizedInputPatternAvailable:    false
IsTableItemPatternAvailable:    false
IsTablePatternAvailable:    false
IsTextChildPatternAvailable:    false
IsTextEditPatternAvailable:    false
IsTextPatternAvailable:    false
IsTextPattern2Available:    false
IsTogglePatternAvailable:    false
IsTransformPatternAvailable:    false
IsTransform2PatternAvailable:    false
IsValuePatternAvailable:    false
IsVirtualizedItemPatternAvailable:    false
IsWindowPatternAvailable:    false
IsCustomNavigationPatternAvailable:    false
FirstChild:    [null]
LastChild:    [null]
Next:    "Malwarebytes" text
Previous:    "Learn More" button
Other Props:    Object has no additional properties
Children:    Container has no children
Ancestors:    "Malwarebytes Premium Trial 3.0.6 Malwarebytes Premium Trial 3.0.6" window
    "Desktop" pane
    [ No Parent ]

[junkew]

looks like you have not used latest simplespy.

It will give you a source code example and also important your whole tree hierarchy of all (grand)parents

The basic concept is

1. Setfocus to desktop, mainwindow
2. Optional setfocus/activate then depending on tree (sub)elements, 
3. Now your element should be visible and only when visible you can click/gettext/settext/sendkeys etc. actions whereas the starter is action highlight so you can at least see you reached your element

Frequently its confusing hierarchy that messes up finding the element you want to act on or elements with similar id's to differenticate from.

I also see you spy on QtWidgets and then developers should use QT Accessibility which is mainly iaccessible and iaccessible2. The examples for that are also in the thread made by LarsJ on how to get to the iaccessible/iaccessible2 interfaces. But as you try an QT App not developed by you I am not sure how accessible it will be.

maybe try ranorex spy: http://www.ranorex.com/test-automation-tools/ranorex-spy.html

If ranorex spy reveals the information you can do it also with the iuiautomation stuff but most likely then the iaccessible examples you should study

Edited February 11, 2017 by junkew

[junkew]

See post 142 and that source should work I tried it just a moment ago with Malware. QTWidgets are recognizable with the iaccessible/iaccessible2 interface.

Checked with Ranorex spy which nicely recognizes directly. I will enhance simplespy in future with iaccessible.

Tool testcomplete is not recognizing the objects out of the box

 

[InnI]

Hi all.

In the post #139 LarsJ said: With IUIAutomationLegacyIAccessiblePattern.GetIAccessible you can get a pointer for IAccessible. I have tested this. It works.

I tried that

#include "CUIAutomation2.au3"

$hHandle = ControlGetHandle("Program Manager", "", "SysListView321") ; desktop Win7

Global $pElement, $oElement, $pIAccess, $pIAccess, $oIAccess, $oAcc, $iCount

$oUIAutomation = ObjCreateInterface($sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation)
$oUIAutomation.ElementFromHandle($hHandle, $pElement)
$oElement = ObjCreateInterface($pElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)

$oElement.GetCurrentPattern($UIA_LegacyIAccessiblePatternId, $pIAccess)
$oIAccess = ObjCreateInterface($pIAccess, $sIID_IUIAutomationLegacyIAccessiblePattern, $dtagIUIAutomationLegacyIAccessiblePattern)

$oIAccess.GetIAccessible($oAcc)
ConsoleWrite(VarGetType($oAcc) & ":" & IsObj($oAcc) & @CRLF)

$oAcc.get_accChildCount($iCount) ; The requested action with this object has failed

$oAcc looks like object (VarGetType), but not is object (IsObj). And I get an error.

Then I changed description in CUIAutomation2.au3 to "GetIAccessible hresult(ptr*);" and tried that

$pAcc = 0
$oIAccess.GetIAccessible($pAcc)
$oAcc = ObjCreateInterface($pAcc, $sIID_IAccessible, $dtagIAccessible)
$oAcc.get_accChildCount($iCount) ; Variable must be of type "Object"

And get an error again.

How can I get access to IAccessible methods from UIAutomation element?

Thanks.

[LarsJ]

The easiest way to create an IAccessible object is to use the IAccessible functions:

#include "CUIAutomation2.au3"
#include "MSAccessibility.au3"

Opt( "MustDeclareVars", 1 )

Example()

Func Example()
  Local $hHandle = ControlGetHandle("Program Manager", "", "SysListView321")
  If Not $hHandle Then Return ConsoleWrite( "$hHandle ERR" & @CRLF )
  ConsoleWrite( "$hHandle OK" & @CRLF )

  Local $pAccess
  AccessibleObjectFromWindow( $hHandle, $OBJID_CLIENT, $tIID_IAccessible, $pAccess )
  If Not $pAccess Then Return ConsoleWrite( "$pAccess ERR" & @CRLF )
  ConsoleWrite( "$pAccess OK" & @CRLF )

  Local $oAccess
  $oAccess = ObjCreateInterface( $pAccess, $sIID_IAccessible, $dtagIAccessible )
  If Not IsObj( $oAccess ) Then Return ConsoleWrite( "$oAccess ERR" & @CRLF )
  ConsoleWrite( "$oAccess OK" & @CRLF )

  Local $iChildCount
  $oAccess.get_accChildCount( $iChildCount )
  If Not $iChildCount Then Return ConsoleWrite( "$iChildCount ERR" & @CRLF )
  ConsoleWrite( "$iChildCount = " & $iChildCount & @CRLF )
EndFunc

Note that it's usually much easier to use UIAutomation objects than IAccessible objects.

In post 139 I was testing with Chrome which supports the IAccessible interface very well.

MSAccessibility.7z

Edited February 25, 2017 by LarsJ

[InnI]

@LarsJ

I know about functions from MSAccessibility UDF. And usually I use LegacyIAccessiblePattern methods because it really is easier. But I can't understand how to use GetIAccessible method for create IAccessible object. Can you explain me?

Thanks.

[LarsJ]

You do it this way:

; LegacyIAccessible object
Local $pLegacyIAccessible, $oLegacyIAccessible
$oUIAutomationElement.GetCurrentPattern( $UIA_LegacyIAccessiblePatternId, $pLegacyIAccessible )
$oLegacyIAccessible = ObjCreateInterface( $pLegacyIAccessible, $sIID_IUIAutomationLegacyIAccessiblePattern, $dtagIUIAutomationLegacyIAccessiblePattern )
If Not IsObj( $oLegacyIAccessible ) Then Return ConsoleWrite( "$oLegacyIAccessible ERR" & @CRLF )
ConsoleWrite( "$oLegacyIAccessible OK" & @CRLF )

; IAccessible object
Local $pIAccessible, $oIAccessible
$oLegacyIAccessible.GetIAccessible( $pIAccessible )
$oIAccessible = ObjCreateInterface( $pIAccessible, $sIID_IAccessible, $dtagIAccessible )
If Not IsObj( $oIAccessible ) Then Return ConsoleWrite( "$oIAccessible ERR" & @CRLF )
ConsoleWrite( "$oIAccessible OK" & @CRLF )

Here is an example with Chrome:

Open Chrome
Enter this in address bar: chrome://accessibility
Click off in this line: Global accessibility mode: off
off should switch to on: Global accessibility mode: on

Run this code:

expandcollapse popup

#include "CUIAutomation2.au3"

Opt( "MustDeclareVars", 1 )

Example()

Func Example()
  ; Chrome window handle
  Local $hChrome = WinGetHandle( "[CLASS:Chrome_WidgetWin_1]", "" )  ; Chrome
  If Not $hChrome Then Return ConsoleWrite( "$hChrome ERR" & @CRLF )
  ConsoleWrite( "$hChrome OK" & @CRLF )

  ; UI Automation object
  Local $oUIAutomation
  $oUIAutomation = ObjCreateInterface( $sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation )
  If Not IsObj( $oUIAutomation ) Then Return ConsoleWrite( "$oUIAutomation ERR" & @CRLF )
  ConsoleWrite( "$oUIAutomation OK" & @CRLF )

  ; UI Automation element
  Local $pChrome, $oChrome
  $oUIAutomation.ElementFromHandle( $hChrome, $pChrome )
  $oChrome = ObjCreateInterface( $pChrome, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
  If Not IsObj( $oChrome ) Then Return ConsoleWrite( "$oChrome ERR" & @CRLF )
  ConsoleWrite( "$oChrome OK" & @CRLF )

  ; LegacyIAccessible object
  Local $pLegacyIAccessible, $oLegacyIAccessible
  $oChrome.GetCurrentPattern( $UIA_LegacyIAccessiblePatternId, $pLegacyIAccessible )
  $oLegacyIAccessible = ObjCreateInterface( $pLegacyIAccessible, $sIID_IUIAutomationLegacyIAccessiblePattern, $dtagIUIAutomationLegacyIAccessiblePattern )
  If Not IsObj( $oLegacyIAccessible ) Then Return ConsoleWrite( "$oLegacyIAccessible ERR" & @CRLF )
  ConsoleWrite( "$oLegacyIAccessible OK" & @CRLF )

  ; IAccessible object
  Local $pIAccessible, $oIAccessible
  $oLegacyIAccessible.GetIAccessible( $pIAccessible )
  $oIAccessible = ObjCreateInterface( $pIAccessible, $sIID_IAccessible, $dtagIAccessible )
  If Not IsObj( $oIAccessible ) Then Return ConsoleWrite( "$oIAccessible ERR" & @CRLF )
  ConsoleWrite( "$oIAccessible OK" & @CRLF )
EndFunc

Output in SciTE:

$hChrome OK
$oUIAutomation OK
$oChrome OK
$oLegacyIAccessible OK
$oIAccessible OK

The code works for Chrome. But there is no guarantee that it works for other windows/controls. If it doesn't work, there is probably not much to do.

[InnI]

4 hours ago, LarsJ said:

The code works for Chrome. But there is no guarantee that it works for other windows/controls.

Yes, it works for Chrome. And for Firefox. Strange, but I couldn't find any Microsoft applications that work with this code. Thank you for explanation.

[junkew]

Nice background article on the iAccessible history (but I am so far mainly using IUIAutomation and Java Accessibility Bridge and not the MSAA part)

http://www.codemag.com/article/0810042

https://www.codeproject.com/articles/18290/introduction-to-microsoft-active-accessibility

and how chrome implements it 

https://chromium.googlesource.com/chromium/chromium/+/trunk/content/browser/accessibility

and NVDA helped me a lot in finding starting points to google further on
https://github.com/nvaccess/nvda/blob/master/source/IAccessibleHandler.py

 

[natalia]

Hello please can you kindly help me with one UI script as I am struggling . I have one need urgently for a project . Please help me kindly . Sequence is like this as mentioned below .

1) you need to select the Pricing Tab from upper bar left
2) Then Select a option from the page ( upper left hand side )
3) Then Click the Search button from right button
4) Choose the correct Pricing ( Various Currency option 13 USD , EUR,YEN,etc..)option which is with hyperlink ( Left hand side )
5) Click hyperlink from the left hand pane
6) Then type the Product name and click search
7) Once found simply choose
8) Click " Add Selected Item " option box from middle upper bar .

That all . I need script help for that can be run with 64 bit windows machine

[junkew]

you should ask that in general help and support and provide more details. Advice is to first read frequently asked questions question number 31.

Without information from Au3Inf or simplespy people will not be able to guide you to a solution.

[TheDcoder]

Hi, first of all, great framework!

I am in a scenario where I have multiple buttons with the "same properties" (atleast the same "title" property). They all look the same... but I have to click on a specific button. The simplespy.au3 script's generated code clicks on the first button... and I have to click on the 3rd button (let's assume). How do I accomplish this?

I can provide more information if wanted, thanks in advance!

[junkew]

property indexrelative                   just goes from a found element n positions further in the elementarray (usefull for textbox without a name after a known label)

property index or instance           just finds the n-th element with that specific match (similar behavior as AutoIt has)

[TheDcoder]

Thank you very much! . I just appended ;indexrelative:=3 to the _UIA_getObjectByFindAll's $str parameter on $oUIElement's declaration, it worked like a charm! Now it clicks on the 3rd button instead of the first . simplespy.au3 should contain functionality to add indexrelative to the $str parameter

[eqjtlm]

Hello, my English is not very good, can only use google to translate, my question is to use your code to get the object.

Can you control the object in this way?

$oExcel = ObjCreate("Excel.Application")

$oExcel.Visible = 1 
$oExcel.WorkBooks.Add

Local $aArray[16][16] 
For $i = 0 To 15 
    For $j = 0 To 15 
        $aArray[$i][$j] = $i 
    Next ;
Next

 

It's not good to translate., please forgive me

Edited April 27, 2017 by eqjtlm

[TheDcoder]

Not exactly but UIA uses objects so you can use different methods associated with them to manipulate UI elements. This might be a good start if you are experienced: https://msdn.microsoft.com/en-us/library/ms747327(v=vs.110).aspx https://msdn.microsoft.com/en-us/library/windows/desktop/ee684009(v=vs.85).aspx

Edited April 27, 2017 by TheDcoder

[LarsJ]

eqjtlm, If you want to automate Excel spreadsheets then look at the functions in the Excel.au3 UDF. See under "Excel Management" in the AutoIt help file.

TheDcoder, This is not the correct link to documentation. As you can see in the Note box in top of the page the documentation is about UI Automation implemented in the .NET framework. But this is not what this thread is about.

This thread is about UI Automation implemented in the Windows API. And you can find the documentation here.

Maybe the chit-chatting documentation is more or less the same. But the Reference documentation (the important documentation) is definitely not the same.

Edited April 27, 2017 by LarsJ

[eqjtlm]

31 minutes ago, LarsJ said:

eqjtlm，如果要自动化Excel电子表格，请查看Excel.au3 UDF中的函数。请参阅AutoIt帮助文件中的“Excel管理”。

TheDcoder，这不是文档的正确链接。正如您可以在页面顶部的“注释”框中看到的，文档是关于UI Automation实现的。.NET框架。但这不是这个线程。

这个线程是关于在Windows API中实现的UI自动化。您可以在这里找到文档。

也许聊天聊天文档或多或少是相同的。但参考文件（重要文件）绝对不一样。

Hello, my question is an example, I want to automate the two controls of "vsflex" and "ttf16", and I want to manipulate him like my example

 

 

We have the time here is 23:30, and I want to sleep, see you tomorrow

Edited April 27, 2017 by eqjtlm