Is there a way to specify a UIAutomation control in a way that yields faster actions?

[bobmcrae]

I am experimenting with UIAWrappers.au3 from junkew to complete an application which presents absolutely no control information using AutoIT Window Info.  While I am able to complete the form successfully, I am not happy with the speed.  As a benchmark, the simple Send method occurs in far less than 1-second, but the UIAutomation approach takes 3-seconds.  I am wondering whether performance gains may be achieved by specifying the controls more precisely; but I am unsure how to do that.  I was able to speed things up a bit by setting $UIA_DefaultWaitTime=0.  The controls of interest are 5-levels deep, as show in the simplespy output below.  It seems I do get faster response by specifying the target/top-level window, as show in the code below.  Any ideas?

#include "UIAWrappers.au3"

_UIA_setVar("Global.Debug", False)
_UIA_setVar("Global.Debug.File", False)
_UIA_setVar("Global.Highlight", False)


_UIA_setVar("DPN","Title:=NC-stat DPNCheck Communicator;controltype:=UIA_WindowControlTypeId;class:=Window")
_UIA_action('DPN','setFocus')

_UIA_setVar("DPN.firstName","AutomationId:=txtFirstName")
_UIA_setVar("DPN.lastName", "AutomationId:=txtLastName")
_UIA_Action('DPN.lastName','setvalue','last name')
_UIA_setVar("DPN.ID",       "AutomationId:=txtSubjectId")
_UIA_setVar("DPN.DOB",      "AutomationId:=PART_TextBox")
_UIA_setVar("DPN.Ft",       "AutomationId:=txtSubjectHeight")
_UIA_setVar("DPN.In",       "AutomationId:=txtSubjectHeight2")

_UIA_Action('DPN.firstName','setvalue','first name')
_UIA_Action('DPN.ID','setvalue','ID012345')
_UIA_Action('DPN.DOB','setvalue','1/31/1932')
_UIA_Action('DPN.Ft','setvalue','6')
_UIA_Action('DPN.In','setvalue','1')

SimpleSpy output:

;~ *** Standard code ***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

Local $oP4=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=NC-stat DPNCheck Communicator;controltype:=UIA_WindowControlTypeId;class:=Window", $treescope_children)   
_UIA_Action($oP4,"setfocus")
Local $oP3=_UIA_getObjectByFindAll($oP4, "Title:=;controltype:=UIA_PaneControlTypeId;class:=Frame", $treescope_children)    
Local $oP2=_UIA_getObjectByFindAll($oP3, "Title:=;controltype:=UIA_TabControlTypeId;class:=TabControl", $treescope_children)    
Local $oP1=_UIA_getObjectByFindAll($oP2, "Title:=Patient;controltype:=UIA_TabItemControlTypeId;class:=TabItem", $treescope_children)    
Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=;controltype:=UIA_PaneControlTypeId;class:=Frame", $treescope_children)    
;~ First find the object in the parent before you can do something
;~$oUIElement=_UIA_getObjectByFindAll(".mainwindow", "title:=;ControlType:=UIA_EditControlTypeId", $treescope_subtree)
Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=;ControlType:=UIA_EditControlTypeId", $treescope_subtree)
_UIA_action($oUIElement,"click")

 

[LarsJ]

The advantage of using the wrapper functions is that they are easy to use. The disadvantage is that they execute a large number of code lines. Especially _UIA_Action. The only thing you can do to increase performance is to avoid the wrapper functions and call the interface methods in CUIAutomation2.au3 directly.

You can look at some of my examples in the UI Automation thread to see how to do it.

[junkew]

Next version I hope to have better speed of the uiaWrappers but I am not sure If I cause 2 seconds delay.

I know the logic in the wrappers is also if no treehierarchy is specified fallback to desktop and start searching the whole tree which can be huge and timeconsuming (when you turn logging on you can see what happens in detail and how many objects are traversed before finding the element)

Func _UIA_getFirstObjectOfElement($obj, $str, $treeScope) is probably quicker as it does not do a findall

I will put on my TODO some logic like

if in UIA_Action 1 property is used for finding an element directly call directly something like below

;- Change Func _UIA_getFirstObjectOfElement

$propertyID = $UIA_NamePropertyId ;- Or automation id property or .....
$tVal= ..... ;- property value

$UIA_oUIAutomation.createPropertyCondition($propertyID, $tval, $pCondition)
$oCondition = ObjCreateInterface($pCondition, $sIID_IUIAutomationPropertyCondition, $dtagIUIAutomationPropertyCondition)
$t = $obj.Findfirst($treeScope, $oCondition, $UIA_pUIElement)
$UIA_oUIElement = ObjCreateInterface($UIA_pUIElement, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)

 

[bobmcrae]

Thanks guys.  I was really struggling to follow even the simple examples, but after writing the main functionality of populating a form in C#, I was able to understand the Automation class a bit further -- without the abstraction of AutoIT.  Then, I turned my attention back to AutoIT and with your guidance above and the C# experience, I wrote a simple UDF for my specific case.  Not only do I appreciate the more AutoIT native syntax, the speed is now fully reasonable.  The _UAI_SetVar and _UAI_Action methods (using the unaltered UIAWrappers.au3) takes ~4-seconds to enter a user name into a text field.  The revised code below populates that field in ~20-milliseconds.  Yeah!  Thanks again for the guidance.

Sample Code:

#include "UIAutomation.au3"

; find parent element
$hWindow = WinGetHandle('Sunlight MiniOmni')
$aeParent = _getAeFromHandle($hWindow)

; find target element
$aeTarget = _getAeFromCondition($aeParent, $UIA_AutomationIdPropertyId, 'txtUserName')

; set target element value
_setAeValue($aeTarget, 'username here')

My Simple UDF (UIAutomation.au3):

expandcollapse popup

#include "CUIAutomation2.au3"

Global $aInterfaceObj

_UIAutomationInit()


Func _UIAutomationInit()
    $aInterfaceObj  = ObjCreateInterface( $sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation )
    Return IsObj($aInterfaceObj) ? $aInterfaceObj : False
EndFunc ;  _UIAutomationInit()


Func _setAeValue($aeTarget, $valueStr)
    Local $patternPointer

    If Not IsObj($aeTarget) Then Return False

    $aeTarget.GetCurrentPattern($UIA_ValuePatternId, $patternPointer)
    $aePattern = ObjCreateInterface( $patternPointer, $sIID_IUIAutomationValuePattern, $dtagIUIAutomationValuePattern)
    $aePattern.SetValue($valueStr)

EndFunc ;_setAeValue()


Func _getAeFromCondition($aeParent, $propType, $propStr)
    Local $targetPointer, $propCondInterfaceObj

    If Not IsObj($aInterfaceObj) Then Return False
    If Not IsObj($aInterfaceObj) Then Return False
    If Not IsObj($aeParent) Then Return False

    $aInterfaceObj.createPropertyCondition( $propType, $propStr, $propCondInterfaceObj )
    $aeParent.FindFirst( $TreeScope_Descendants, $propCondInterfaceObj, $targetPointer)
    $ae = ObjCreateInterface( $targetPointer, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )

    Return IsObj($ae) ? $ae : False

EndFunc ; _getAeFromCondition()


Func _getAeFromHandle($hWindow)
    Local $winPointer

    If Not WinExists($hWindow) Then Return False
    If Not IsObj($aInterfaceObj) Then Return False

    $aInterfaceObj.ElementFromHandle( $hWindow, $winPointer )
    $ae = ObjCreateInterface( $winPointer, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )

    Return IsObj($ae) ? $ae : False
EndFunc ; _getAeFromHandle()

 

[junkew]

Findfirst is for sure match quicker and in your scenario your developers behave nicely on automationid.

The uiawrappers give 

A. Regular expression matching

B. Matching on index or ordinal value

C. Match with multiple properties

As said I will improve on single property and exact matching using findfirst

[Chiitus]

Nice topic!

First i will learn, then i will ask forsome help later on 

Really amazing job @junkew

Thanks adv.

@UPDATE:

On 19/02/2016 at 3:06 AM, bobmcrae said:

Thanks guys.  I was really struggling to follow even the simple examples, but after writing the main functionality of populating a form in C#, I was able to understand the Automation class a bit further -- without the abstraction of AutoIT.  Then, I turned my attention back to AutoIT and with your guidance above and the C# experience, I wrote a simple UDF for my specific case.  Not only do I appreciate the more AutoIT native syntax, the speed is now fully reasonable.  The _UAI_SetVar and _UAI_Action methods (using the unaltered UIAWrappers.au3) takes ~4-seconds to enter a user name into a text field.  The revised code below populates that field in ~20-milliseconds.  Yeah!  Thanks again for the guidance.

Sample Code:

#include "UIAutomation.au3"

; find parent element
$hWindow = WinGetHandle('Sunlight MiniOmni')
$aeParent = _getAeFromHandle($hWindow)

; find target element
$aeTarget = _getAeFromCondition($aeParent, $UIA_AutomationIdPropertyId, 'txtUserName')

; set target element value
_setAeValue($aeTarget, 'username here')

My Simple UDF (UIAutomation.au3):

expandcollapse popup

#include "CUIAutomation2.au3"

Global $aInterfaceObj

_UIAutomationInit()


Func _UIAutomationInit()
    $aInterfaceObj  = ObjCreateInterface( $sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation )
    Return IsObj($aInterfaceObj) ? $aInterfaceObj : False
EndFunc ;  _UIAutomationInit()


Func _setAeValue($aeTarget, $valueStr)
    Local $patternPointer

    If Not IsObj($aeTarget) Then Return False

    $aeTarget.GetCurrentPattern($UIA_ValuePatternId, $patternPointer)
    $aePattern = ObjCreateInterface( $patternPointer, $sIID_IUIAutomationValuePattern, $dtagIUIAutomationValuePattern)
    $aePattern.SetValue($valueStr)

EndFunc ;_setAeValue()


Func _getAeFromCondition($aeParent, $propType, $propStr)
    Local $targetPointer, $propCondInterfaceObj

    If Not IsObj($aInterfaceObj) Then Return False
    If Not IsObj($aInterfaceObj) Then Return False
    If Not IsObj($aeParent) Then Return False

    $aInterfaceObj.createPropertyCondition( $propType, $propStr, $propCondInterfaceObj )
    $aeParent.FindFirst( $TreeScope_Descendants, $propCondInterfaceObj, $targetPointer)
    $ae = ObjCreateInterface( $targetPointer, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )

    Return IsObj($ae) ? $ae : False

EndFunc ; _getAeFromCondition()


Func _getAeFromHandle($hWindow)
    Local $winPointer

    If Not WinExists($hWindow) Then Return False
    If Not IsObj($aInterfaceObj) Then Return False

    $aInterfaceObj.ElementFromHandle( $hWindow, $winPointer )
    $ae = ObjCreateInterface( $winPointer, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )

    Return IsObj($ae) ? $ae : False
EndFunc ; _getAeFromHandle()

 

I'm getting:

Unknown function name.:
$aeParent = _getAeFromHandle($hWindow)
$aeParent = ^ ERROR

Some idea why?

Thanks adv.

Edited March 1, 2016 by Chiitus

[bobmcrae]

@Chiitus, it looks like perhaps you did not copy the entire  UIAutomation.au3 UDF, as the function _getAeFromHandle is at the bottom of the code above...take another look.

[maniootek]

how did you find this

'txtUserName'

(which way did you read it?)

and why did you use it together with

$UIA_AutomationIdPropertyId

Can you also give me direct link to this application

Sunlight MiniOmni

then I can download it, use "simple spy" and try to understand your code to learn UIA

Edited April 10, 2016 by maniootek

[sn1kzZ]

@junkew Thanks so much for your UDF, it makes things much faster but can you include an "action" function in your UDF?

I'm using your script to automate mstsc.exe in win2016, I can inject username + password thanks to your code, but how do i click on the ok button?

Thanks.

[sn1kzZ]

Sorry I tagged the wrong user, I meant to tag @bobmcrae. if some mod can edit my post above, ty.

[BrewManNH]

That user hasn't been here since 2016, I'm not sure he'll be responding.

[LarsJ]

sn1kzZ, Here is a list of control types and supported patterns (actions).

You should create a new thread with your own question. Add information about application, controls and actions. Then it should not be impossible to make some fast code.

 

You can also google this way:

LarsJ CUIAutomation2.au3 site:autoitscript.com

Then you'll get a list of examples with very fast code.

[junkew]

As you can put text in it with setvalue it should be easy to use action leftclick on the button. Just sinplespy on your ok button and see if it highlights. Post the output and code if its not working.