IUIAutomation MS framework automate chrome, FF, IE, ....

junkew · April 11, 2018

Yes should be possible. Can you reproduce same problem on www.google.com or any other public reachable url?

junkew · April 11, 2018

released in first post as attachment version 0.66 + examplesV0.66

@rvicta download in first post v0.66 and see example 6 where in last step 10 I just reference a piece of text by name only.

Example could be

;~ Find the chrome window
local $oChrome = _UIA_getFirstObjectOfElement($UIA_oDesktop, "class:=Chrome_WidgetWin_1", $treescope_children)
If Not IsObj($oChrome) Then
    _UIA_DumpThemAll($UIA_oDesktop, $treescope_subtree)
    Exit
EndIf
;~ Get the document reference
$oDocument = _UIA_getFirstObjectOfElement($oChrome, "controltype:=" & $UIA_DocumentControlTypeId, $treescope_subtree)
If Not IsObj($oDocument) Then
    _UIA_DumpThemAll($oDocument, $treescope_subtree)
EndIf
    
Local $oUIElement=_UIA_getObjectByFindAll($oDocument, "title:=Part Number Entry;ControlType:=Text", $treescope_subtree)

_UIA_Action($oUIElement,"highlight")
consolewrite(_UIA_action($oUIElement, "property", "title"))

otherwise search in examples for indexrelative:=(+/-)n where you can reference relative n position back or forward close to another object. You can determine n by looking into the result of _UIA_DumpThemAll($oDocument,$treescope_subtree)

rvicta · April 16, 2018

Thank you, junkew, for your suggestions. This worked great for what I was trying to do. I can now read the text controls properly.

But now, I have come upon an odd behavior. This is still the same Chrome session for a customer. I have 3 (actually there are more and they exhibit this same behavior but I'm using these three as an example) edit controls on the screen. The first and third controls have different values in them. When I write their values to the console they showed the correct values. The second edit control, if it is empty on the screen, will show me the same value as the first edit control when I write it to the console. I know the index is correct for this edit control because I commented out the code to read from all other edit controls on the page and the correct one gets highlighted when running the script.

Local $oPhasedOut = _UIA_getObjectByFindAll($oDocument, "controltype:=" & $UIA_TextControlTypeId & ";Index:=15", $treescope_subtree)
   Local $sPhasedOut = _UIA_action($oPhasedOut, "property", "title")
   ConsoleWrite("Phased Out: " & $sPhasedOut & @CRLF)

   Local $oCost = _UIA_getObjectByFindAll($oDocument, "controltype:=" & $UIA_EditControlTypeId & ";Index:=6", $treescope_subtree)
   Local $sCost = _UIA_Action($oCost, "getValue")
   ConsoleWrite("Cost: " & $sCost & @CRLF)

   Local $oTrade = _UIA_getObjectByFindAll($oDocument, "controltype:=" & $UIA_EditControlTypeId & ";Index:=7", $treescope_subtree)
   Local $sTrade = _UIA_Action($oTrade, "getValue")
   ConsoleWrite("Trade: " & $sTrade & @CRLF)

   Local $oList = _UIA_getObjectByFindAll($oDocument, "controltype:=" & $UIA_EditControlTypeId & ";Index:=8", $treescope_subtree)
   Local $sList = _UIA_Action($oList, "getValue")
   ConsoleWrite("List: " & $sList & @CRLF)

The output from SimplySpy for this empty edit control is:

Having the following values for all properties: 
Title is: <>    Class   := <>   controltype:= <UIA_EditControlTypeId>   ,<50004>    , (0000C354)    610;551;87;20
*** Parent Information top down ***
5: Title is: <BS1-My Favorites> Class   := <Chrome_WidgetWin_1> controltype:= <UIA_WindowControlTypeId> ,<50032>    , (0000C370)    14;16;1663;1056
"Title:=BS1-My Favorites;controltype:=UIA_WindowControlTypeId;class:=Chrome_WidgetWin_1""   
4: Title is: <BS1-My Favorites> Class   := <Chrome_RenderWidgetHostHWND>    controltype:= <UIA_DocumentControlTypeId>   ,<50030>    , (0000C36E)    22;101;1647;963
"Title:=BS1-My Favorites;controltype:=UIA_DocumentControlTypeId;class:=Chrome_RenderWidgetHostHWND""    
3: Title is: <> Class   := <>   controltype:= <UIA_CustomControlTypeId> ,<50025>    , (0000C369)    22;101;1647;808
"Title:=;controltype:=UIA_CustomControlTypeId;class:="" 
2: Title is: <> Class   := <>   controltype:= <UIA_CustomControlTypeId> ,<50025>    , (0000C369)    22;136;1647;773
"Title:=;controltype:=UIA_CustomControlTypeId;class:="" 
1: Title is: <> Class   := <>   controltype:= <UIA_CustomControlTypeId> ,<50025>    , (0000C369)    365;259;961;688
"Title:=;controltype:=UIA_CustomControlTypeId;class:="" 
0: Title is: <Part Number Entry GM :AD1000                     Description GEAR OIL Non-Stocked                                                                                            On Hand 0 Last Sale 10/01/13 In Invent     Stock Group* 203 Phased Out 3/07/15 Group Cost Stock Status                      Trade        Order Status                                List 27.68 Minimum Flat Price Maximum Comp 1 Pack Quantity 1 Comp 2 Low Year Update* Y High Year Prog Code   Reman Ind New             Return Code Price Guide Promo Code*  Taxable*   Y Remarks/Dsp Function* Ok Exit Cancel Bin/Shelf* >  Class   := <>   controltype:= <UIA_CustomControlTypeId> ,<50025>    , (0000C369)    375;285;1023;616
"Title:=Part Number Entry GM :AD1000                     Description GEAR OIL Non-Stocked                                                                                            On Hand 0 Last Sale 10/01/13 In Invent     Stock Group* 203 Phased Out 3/07/15 Group Cost Stock Status                      Trade        Order Status                                List 27.68 Minimum Flat Price Maximum Comp 1 Pack Quantity 1 Comp 2 Low Year Update* Y High Year Prog Code   Reman Ind New             Return Code Price Guide Promo Code*  Taxable*   Y Remarks/Dsp Function* Ok Exit Cancel Bin/Shelf* ;controltype:=UIA_CustomControlTypeId;class:=""  


;~ *** Standard code maintainable ***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

_UIA_setVar("oP1","Title:=BS1-My Favorites;controltype:=UIA_WindowControlTypeId;class:=Chrome_WidgetWin_1") ;BS1-My Favorites
_UIA_setVar("oP2","Title:=BS1-My Favorites;controltype:=UIA_DocumentControlTypeId;class:=Chrome_RenderWidgetHostHWND")  ;BS1-My Favorites
_UIA_setVar("oP3","Title:=;controltype:=UIA_CustomControlTypeId;class:=")   ;
_UIA_setVar("oP4","Title:=;controltype:=UIA_CustomControlTypeId;class:=")   ;
_UIA_setVar("oP5","Title:=;controltype:=UIA_CustomControlTypeId;class:=")   ;
_UIA_setVar("oP6","Title:=Part Number Entry GM :AD1000                     Description GEAR OIL Non-Stocked                                                                                            On Hand 0 Last Sale 10/01/13 In Invent     Stock Group* 203 Phased Out 3/07/15 Group Cost Stock Status                      Trade        Order Status                                List 27.68 Minimum Flat Price Maximum Comp 1 Pack Quantity 1 Comp 2 Low Year Update* Y High Year Prog Code   Reman Ind New             Return Code Price Guide Promo Code*  Taxable*   Y Remarks/Dsp Function* Ok Exit Cancel Bin/Shelf* ;controltype:=UIA_CustomControlTypeId;class:=")    ;Part Number Entry GM :AD1000                     Description GEAR OIL Non-Stocked                                                                                            On Hand 0 Last Sale 10/01/13 In Invent     Stock Group* 203 Phased Out 3/07/15 Group Cost Stock Status                      Trade        Order Status                                List 27.68 Minimum Flat Price Maximum Comp 1 Pack Quantity 1 Comp 2 Low Year Update* Y High Year Prog Code   Reman Ind New             Return Code Price Guide Promo Code*  Taxable*   Y Remarks/Dsp Function* Ok Exit Cancel Bin/Shelf* 

;~ $oUIElement=_UIA_getObjectByFindAll(".mainwindow", "title:=;ControlType:=UIA_EditControlTypeId", $treescope_subtree)
_UIA_setVar("oUIElement","Title:=;controltype:=UIA_EditControlTypeId;class:=") ;ControlType:=UIA_EditControlTypeId;classname:=")

;~ Actions split away from logical/technical definition above can come from configfiles 

;~_UIA_Action("oP1","highlight")
_UIA_Action("oP1","setfocus")
;~_UIA_Action("oP2","highlight")
_UIA_Action("oP2","setfocus")
;~_UIA_Action("oP3","highlight")
_UIA_Action("oP3","setfocus")
;~_UIA_Action("oP4","highlight")
_UIA_Action("oP4","setfocus")
;~_UIA_Action("oP5","highlight")
_UIA_Action("oP5","setfocus")
;~_UIA_Action("oP6","highlight")
_UIA_Action("oP6","setfocus")

_UIA_action("oUIElement","highlight")
;~_UIA_action("oUIElement","click")


;~ *** Standard code Flexible***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

Local $oP5=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=BS1-My Favorites;controltype:=UIA_WindowControlTypeId;class:=Chrome_WidgetWin_1", $treescope_children)    
_UIA_Action($oP5,"setfocus")
Local $oP4=_UIA_getObjectByFindAll($oP5, "Title:=BS1-My Favorites;controltype:=UIA_DocumentControlTypeId;class:=Chrome_RenderWidgetHostHWND", $treescope_children)  
_UIA_Action($oP4,"setfocus")
Local $oP3=_UIA_getObjectByFindAll($oP4, "Title:=;controltype:=UIA_CustomControlTypeId;class:=", $treescope_children)   
Local $oP2=_UIA_getObjectByFindAll($oP3, "Title:=;controltype:=UIA_CustomControlTypeId;class:=", $treescope_children)   
Local $oP1=_UIA_getObjectByFindAll($oP2, "Title:=;controltype:=UIA_CustomControlTypeId;class:=", $treescope_children)   
Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=Part Number Entry GM :AD1000                     Description GEAR OIL Non-Stocked                                                                                            On Hand 0 Last Sale 10/01/13 In Invent     Stock Group* 203 Phased Out 3/07/15 Group Cost Stock Status                      Trade        Order Status                                List 27.68 Minimum Flat Price Maximum Comp 1 Pack Quantity 1 Comp 2 Low Year Update* Y High Year Prog Code   Reman Ind New             Return Code Price Guide Promo Code*  Taxable*   Y Remarks/Dsp Function* Ok Exit Cancel Bin/Shelf* ;controltype:=UIA_CustomControlTypeId;class:=", $treescope_children)    
;~ First find the object in the parent before you can do something
;~$oUIElement=_UIA_getObjectByFindAll(".mainwindow", "title:=;ControlType:=UIA_EditControlTypeId", $treescope_subtree)
Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=;ControlType:=UIA_EditControlTypeId", $treescope_subtree)
;~_UIA_action($oUIElement,"highlight")
_UIA_action($oUIElement,"click")


*** Detailed properties of the highlighted element ***
UIA_iaccessiblechildId:= <0>
UIA_handle:= <0>
UIA_BoundingRectangle:= <610;551;87;20>
UIA_ProcessId:= <2964>
UIA_ControlType:= <50004>
UIA_LocalizedControlType:= <edit>
UIA_HasKeyboardFocus:= <False>
UIA_IsKeyboardFocusable:= <True>
UIA_IsEnabled:= <True>
UIA_Culture:= <0>
UIA_IsControlElement:= <True>
UIA_IsContentElement:= <True>
UIA_IsPassword:= <False>
UIA_NativeWindowHandle:= <0>
UIA_IsOffscreen:= <False>
UIA_Orientation:= <0>
UIA_IsRequiredForForm:= <False>
UIA_IsDockPatternAvailable:= <False>
UIA_IsExpandCollapsePatternAvailable:= <False>
UIA_IsGridItemPatternAvailable:= <False>
UIA_IsGridPatternAvailable:= <False>
UIA_IsInvokePatternAvailable:= <True>
UIA_IsMultipleViewPatternAvailable:= <False>
UIA_IsRangeValuePatternAvailable:= <False>
UIA_IsScrollPatternAvailable:= <False>
UIA_IsScrollItemPatternAvailable:= <False>
UIA_IsSelectionItemPatternAvailable:= <False>
UIA_IsSelectionPatternAvailable:= <False>
UIA_IsTablePatternAvailable:= <False>
UIA_IsTableItemPatternAvailable:= <False>
UIA_IsTextPatternAvailable:= <True>
UIA_IsTogglePatternAvailable:= <False>
UIA_IsTransformPatternAvailable:= <False>
UIA_IsValuePatternAvailable:= <True>
UIA_IsWindowPatternAvailable:= <False>
UIA_ValueIsReadOnly:= <False>
UIA_RangeValueValue:= <0>
UIA_RangeValueIsReadOnly:= <True>
UIA_RangeValueMinimum:= <0>
UIA_RangeValueMaximum:= <0>
UIA_RangeValueLargeChange:= <0>
UIA_RangeValueSmallChange:= <0>
UIA_ScrollHorizontalScrollPercent:= <0>
UIA_ScrollHorizontalViewSize:= <100>
UIA_ScrollVerticalScrollPercent:= <0>
UIA_ScrollVerticalViewSize:= <100>
UIA_ScrollHorizontallyScrollable:= <False>
UIA_ScrollVerticallyScrollable:= <False>
UIA_SelectionCanSelectMultiple:= <False>
UIA_SelectionIsSelectionRequired:= <False>
UIA_GridRowCount:= <0>
UIA_GridColumnCount:= <0>
UIA_GridItemRow:= <0>
UIA_GridItemColumn:= <0>
UIA_GridItemRowSpan:= <1>
UIA_GridItemColumnSpan:= <1>
UIA_DockDockPosition:= <5>
UIA_ExpandCollapseExpandCollapseState:= <3>
UIA_MultipleViewCurrentView:= <0>
UIA_WindowCanMaximize:= <False>
UIA_WindowCanMinimize:= <False>
UIA_WindowWindowVisualState:= <0>
UIA_WindowWindowInteractionState:= <0>
UIA_WindowIsModal:= <False>
UIA_WindowIsTopmost:= <False>
UIA_SelectionItemIsSelected:= <False>
UIA_TableRowOrColumnMajor:= <2>
UIA_ToggleToggleState:= <2>
UIA_TransformCanMove:= <False>
UIA_TransformCanResize:= <False>
UIA_TransformCanRotate:= <False>
UIA_IsLegacyIAccessiblePatternAvailable:= <True>
UIA_LegacyIAccessibleChildId:= <0>
UIA_LegacyIAccessibleRole:= <42>
UIA_LegacyIAccessibleState:= <1048576>
UIA_LegacyIAccessibleDefaultAction:= <activate>
UIA_IsDataValidForForm:= <False>
UIA_ProviderDescription:= <[pid:2964,providerId:0x0 Main(parent link):Microsoft: MSAA Proxy (unmanaged:uiautomationcore.dll)]>
UIA_IsItemContainerPatternAvailable:= <False>
UIA_IsVirtualizedItemPatternAvailable:= <False>
UIA_IsSynchronizedInputPatternAvailable:= <False>

I noticed that UIA_iaccessiblevalue is not present for empty edit controls. Is there a way to check for this? Then I would probably know that there is no value.

Edited April 16, 2018 by rvicta

junkew · April 17, 2018

Within simplespy or uiawrappers I filter out empty values.
To be 100% sure my advice use

inspect as part of windows SDK
VisualUIAVerifyNative.exe as part of windows SDK (8.1)

I see you use index (which is probably fine for your purpose) but understand clearly also the indexrelative property as that could be usefull to get the textlabel normally nn positions before your editbox if your developers program nicely its the same distance for all textlabel controls.

rvicta · April 17, 2018

Thank you for the help, junkew.

Using $UIA_LegacyIAccessibleValuePropertyId for edit controls I am able to determine if it is empty. Using Inspect.exe I was able to see that ValuePropertId would either contain the value or an empty string.

youtuber · April 27, 2018

Can you write me a sample code here is very mixed
Getting the active window url address chrome

junkew · April 28, 2018

Did yoy try simplespy on the addressbar? You will get an example with simplespy. In examples.zip in first ten examples you see how to get addressbar object. All major browsers are in examples.zip.

GrahamFewster · June 12, 2018

I am playing around with UI Automation. Any update?

Earthshine · June 12, 2018

why?

junkew · June 13, 2018

Many ideas to rewrite with dot syntax and more wrapped away to deal consistently with different browsers in 1 syntax using autoitobject internal udf. But as time is limited not on short term.

But if you miss something let me know so maybe easy to add.

r0ash · June 20, 2018

Skype 8 (on Windows 8.1) use Chrome (Chrome_WidgetWin_1)

Some time its able to select the Sign In button and sometime it dont. The following screenshot depicts when it do not select the individual item Not Sign in nor Create Account etc.

I suspect Skype is somehow changing the DOM, but still, its visible, manually its possible to click the Sign in button, when it do not highlight the control (and do not display the code), the following code do not work

#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

Local $oP2=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=Skype;controltype:=UIA_WindowControlTypeId;class:=Chrome_WidgetWin_1", $treescope_children)
_UIA_Action($oP2,"setfocus")
Local $oP1=_UIA_getObjectByFindAll($oP2, "Title:=Skype Preview;controltype:=UIA_DocumentControlTypeId;class:=Chrome_RenderWidgetHostHWND", $treescope_children)
_UIA_Action($oP1,"setfocus")
Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=;controltype:=UIA_CustomControlTypeId;class:=", $treescope_children)
;~ First find the object in the parent before you can do something
;~$oUIElement=_UIA_getObjectByFindAll("SigninwithMicrosoft.mainwindow", "title:=Sign in with Microsoft;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)
Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=Sign in;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)
;_UIA_action($oUIElement,"highlight")
_UIA_action($oUIElement,"click")

Is there any way to find the exact reason and solution?

Thanks.

Update

I realized that whenever I am running simplespy in parallel, it identifies the control, therefore I copied _EventProc() from simplespy.au3 and added into my script, tested couple of times, its working everytime. Trying to understand why Chrome Accessibility has its effect on finding the control. Would still appreciate expert opinion here.

Edited June 20, 2018 by r0ash
adding more information

junkew · June 20, 2018

I have to check but when chromium starts it sends an alert to see if there are screenreaders around. Simplespy answers the alert and chrome turns on the accessibility tree.

InnI · June 21, 2018

@junkew @r0ash

It seems like no need to wait until chromium sends EVENT_SYSTEM_ALERT message. Just to send a message to RenderWidget control and chromium turns on the accessibility tree. Tested on Google Chrome 67 portable without --force-renderer-accessibility parameter

$hWnd = ControlGetHandle("[RegexpTitle: - Google Chrome]", "", "Chrome_RenderWidgetHostHWND1")
_SendMessage($hWnd, $WM_GETOBJECT, 0, 1)

milkmoron · July 17, 2018

Hi I'm looking to automate a third party application called ams360 for data entry. Is it possible with this UDF? I don't think the Control Functions work correctly for me. Is there an easier alternative, it looks very complicated and I don't have alot of scripting skill. What should I be doing to learn more about how to use this UDF.

TheDcoder · July 18, 2018

@milkmoron I am not aware of any other easier alternative, but you can starting by reading the linked posts and examples in the first post:

Good luck

xtcislove · July 18, 2018

nice

milkmoron · July 18, 2018

So I'm using IUIAutomation and I'm not sure if I'm doing this correctly but heres my code to select the description text area. Store the value for later and send it to another text area.

#include "UIAWrappers.au3"
ControlFocus ( "View Activity", "", "")
Local $oViewActivity = ControlGetFocus ( "View Activity" )

_UIA_SETVAR("Description","classname:=WindowsForms10.EDIT.app.0.30495d1_r9_ad118")
Local $oPolicy = _UIA_Action("Description","getvalue")

MsgBox($MB_SYSTEMMODAL, "", "The text in Name1 is: " & $oPolicy)

ControlFocus ( "Activity / Suspense", "", "")
WinActivate ( "Activity / Suspense" )

_UIA_SETVAR("FIADescription","title:=Description:")
;_UIA_SETVAR("FIADescription","classname:=WindowsForms10.EDIT.app.0.30495d1_r9_ad14
_UIA_Action("FIADescription","sendkeys", $oPolicy)

It pauses during Local $oPolicy = _UIA_Action("Description","getvalue")

junkew · July 18, 2018

Start with simplespy and post output of that. Better done in general help and support area with new thread as the answers can be lengthy.

For beginners frequently its hard to understand gui hierarchy and z order of elements. For that simplespy and inspect are tools to help you.

First step is to have action highlight working on your element of interest from then onwards it should become easier. In the background some log*.xml is made giving you all details on how udf is trying to find your element.

milkmoron · July 19, 2018

On 7/18/2018 at 11:56 AM, junkew said:

Start with simplespy and post output of that. Better done in general help and support area with new thread as the answers can be lengthy.

For beginners frequently its hard to understand gui hierarchy and z order of elements. For that simplespy and inspect are tools to help you.

First step is to have action highlight working on your element of interest from then onwards it should become easier. In the background some log*.xml is made giving you all details on how udf is trying to find your element.

Which parts need to be outputted. It has some confidential information in it.

careca · July 19, 2018

Really no need to quote the previous post.

The parts to post the output of are the parts that show the controls in question.

IUIAutomation MS framework automate chrome, FF, IE, ....

Recommended Posts

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

junkew

junkew

junkew

Posted Images

IUIAutomation - Topic with framework and examples

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members

Similar Content