Using UI Automation Code in AutoIt

[IanN1990]

I am trying to automate an console application which doesn't support STDIN and STDOUT. Though i could use ControlSend as a replacement for STDIN, ControlGetText returns no information. I found your guide and thought i would give it a try. 

Here is the code i have below

expandcollapse popup

#NoTrayIcon
#include "CUIAutomation2.au3"

Example()

Func Example()
  ; Open CMD
  run("cmd.exe")
  
  sleep(1000)

  ; Create UI Automation object
  Local $oUIAutomation = ObjCreateInterface( $sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation )
  If Not IsObj( $oUIAutomation ) Then Return ConsoleWrite( "$oUIAutomation ERR" & @CRLF )
  ConsoleWrite( "$oUIAutomation OK" & @CRLF )

  ; Get Desktop element
  Local $pDesktop, $oDesktop
  $oUIAutomation.GetRootElement( $pDesktop )
  $oDesktop = ObjCreateInterface( $pDesktop, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
  If Not IsObj( $oDesktop ) Then Return ConsoleWrite( "$oDesktop ERR" & @CRLF )
  ConsoleWrite( "$oDesktop OK" & @CRLF )

  ConsoleWrite( "--- CMD window ---" & @CRLF )

  Local $pCondition ; Note that $UIA_ClassNamePropertyId maybe ia a CASE SENSITIVE condition
  $oUIAutomation.CreatePropertyCondition( $UIA_ClassNamePropertyId, "ConsoleWindowClass", $pCondition )
  If Not $pCondition Then Return ConsoleWrite( "$pCondition ERR" & @CRLF )
  ConsoleWrite( "$pCondition OK" & @CRLF )

  Local $pCMD, $oCMD
  $oDesktop.FindFirst( $TreeScope_Descendants, $pCondition, $pCMD )
  $oCMD = ObjCreateInterface( $pCMD, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
  If Not IsObj( $oCMD ) Then Return ConsoleWrite( "$oCMD ERR" & @CRLF )
  ConsoleWrite( "$oCMD OK" & @CRLF )

  ConsoleWrite( "--- Fill CMD element ---" & @CRLF )

  ; Note that $UIA_AutomationIdPropertyId is a STRING and maybe a CASE SENSITIVE condition
  $oUIAutomation.CreatePropertyCondition( $UIA_AutomationIdPropertyId, "Text Area", $pCondition )
  If Not $pCondition Then Return ConsoleWrite( "$pCondition ERR" & @CRLF )
  ConsoleWrite( "$pCondition OK" & @CRLF )

  Local $pConsole, $oConsole
  $oCMD.FindFirst( $TreeScope_Descendants, $pCondition, $pConsole )
  $oConsole = ObjCreateInterface( $pConsole, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
  If Not IsObj( $oConsole ) Then Return ConsoleWrite( "$oConsole ERR" & @CRLF )
  ConsoleWrite( "$oConsole OK" & @CRLF )

  Local $pValue, $oValue
  $oConsole.GetCurrentPattern( $UIA_LegacyIAccessiblePatternId, $pValue )
  $oValue = ObjCreateInterface( $pValue, $sIID_IUIAutomationLegacyIAccessiblePattern, $dtagIUIAutomationLegacyIAccessiblePattern )
  If Not IsObj( $oValue ) Then Return ConsoleWrite( "$oValue ERR" & @CRLF )

  $oValue.SetValue("Test")
EndFunc

No errors come back but nothing is printed. I also cant figure out how i would read from it. 
I used the Legacy option as the only other supported item i can see is Text Pattern which requires pointers, structures or long data types ^^

Have i gone wrong somewhere?

 

Edited April 17, 2019 by IanN1990

[junkew]

Use send or controlsend to send keys. Should in general work. Why do you think stdin out is not supported.

[LarsJ]

IanN1990, The text in a console window is usually written directly in the client area in the window with the DrawText function or the like. DrawText is a GDI (graphics) function and the text is simply colored pixels directly in the window. The text in a console window is comparable to an image and is not represented as an edit or text control and therefore cannot be automated either with UIA, MSAA or classic automation code.

The text in Notepad or in a listbox, listview or treeview is, however, represented as a real edit or text control where the text is stored in a DllStruct (char, wchar, str, wstr, bstr or similar) that is included in the control. Therefore, these text controls can be automated. But this is not the case for the text in a console window and therefore the text cannot be automated.

[LarsJ]

This is an update of some of the (old) examples that have been needed due to the Windows 8, 8.1 and 10 update a month ago. New zip-file at bottom of first post.

[Exit]

@LarsJ Is it possible to distribute as real ZIP files ?   "*.7Z" files are not supported in a raw Win10 installation and not all have 7Z software installed on there systems.

 

[LarsJ]

Sorry Exit, but you'll have to live with the 7z-files.

[Exit]

@LarsJ Can you explain why it is not possible to use ZIP standard ?

[philkryder]

I've tried the UIASpy.au3 on both  3.3.14.2 and 3.3.14.5

Both give an error at line 3787 in UIASpy_ElemInfo.au3.

This is on Windows 10 1709 at work.

on my 1809 machine at home it runs.  ? any ideas on why would be appreciated? thanks! phil

ps - I did a msgbox on $iIdx and it was ZERO here when it failed.

>"C:\Program Files (x86)\AutoIt3\SciTE\..\AutoIt3.exe" "C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.au3" /run /prod /ErrorStdOut /in "R:\Operations\Macros\MacroTools\___AutoIT\UIA\UIAExamples\UIASpy.au3" /UserParams    
+>14:31:21 Starting AutoIt3Wrapper v.15.920.938.0 SciTE v.3.6.0.0   Keyboard:00010409  OS:WIN_10/  CPU:X64 OS:X64    Environment(Language:0409)
+>         SciTEDir => C:\Program Files (x86)\AutoIt3\SciTE   UserDir => C:\Users\pkryder\AppData\Local\AutoIt v3\SciTE\AutoIt3Wrapper   SCITE_USERHOME => C:\Users\pkryder\AppData\Local\AutoIt v3\SciTE 
>Running AU3Check (3.3.14.2)  from:C:\Program Files (x86)\AutoIt3  input:R:\Operations\Macros\MacroTools\___AutoIT\UIA\UIAExamples\UIASpy.au3
+>14:31:21 AU3Check ended.rc:0
>Running:(3.3.14.2):C:\Program Files (x86)\AutoIt3\autoit3.exe "R:\Operations\Macros\MacroTools\___AutoIT\UIA\UIAExamples\UIASpy.au3"    
--> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop
"R:\Operations\Macros\MacroTools\___AutoIT\UIA\UIAExamples\Includes\UIASpy_ElemInfo.au3" (3787) : ==> Subscript used on non-accessible variable.:
Local $aElemIndexes = ($aElems[$iIdx][7])[1], $aIndex, $jCnt = 0
Local $aElemIndexes = ($aElems[$iIdx][7])^ ERROR
->14:31:22 AutoIt3.exe ended.rc:1
+>14:31:22 AutoIt3Wrapper Finished.
>Exit code: 1    Time: 1.476
 

Edited May 21, 2019 by philkryder

[IanN1990]

Back for round 2! Determined to make use of this UDF!  
 

expandcollapse popup

#NoTrayIcon
#include "CUIAutomation2.au3"

Example()

Func Example()
    ; Create UI Automation object
    Local $oUIAutomation = ObjCreateInterface( $sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation )
    If Not IsObj( $oUIAutomation ) Then Return ConsoleWrite( "$oUIAutomation ERR" & @CRLF )
    ConsoleWrite( "$oUIAutomation OK" & @CRLF )

     ; Get Desktop element
    Local $pDesktop, $oDesktop
    $oUIAutomation.GetRootElement( $pDesktop )
    $oDesktop = ObjCreateInterface( $pDesktop, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
    If Not IsObj( $oDesktop ) Then Return ConsoleWrite( "$oDesktop ERR" & @CRLF )
    ConsoleWrite( "$oDesktop OK" & @CRLF )

    ; --- Find window/control ---

    ConsoleWrite( "--- Find window/control ---" & @CRLF )

    Local $pCondition0
    $oUIAutomation.CreatePropertyCondition( $UIA_NamePropertyId, "XenDesktop 7.6.300", $pCondition0 )
    If Not $pCondition0 Then Return ConsoleWrite( "$pCondition0 ERR" & @CRLF )
    ConsoleWrite( "$pCondition0 OK" & @CRLF )

    Local $pWindow1, $oWindow1
    $oDesktop.FindFirst( $TreeScope_Descendants, $pCondition0, $pWindow1 )
    $oWindow1 = ObjCreateInterface( $pWindow1, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
    If Not IsObj( $oWindow1 ) Then Return ConsoleWrite( "$oWindow1 ERR" & @CRLF )
    ConsoleWrite( "$oWindow1 OK" & @CRLF )

    ; --- Find window/control ---

    ConsoleWrite( "--- Find window/control ---" & @CRLF )

    Local $pCondition1
    $oUIAutomation.CreatePropertyCondition( $UIA_NamePropertyId, "Next", $pCondition1 )
    If Not $pCondition1 Then Return ConsoleWrite( "$pCondition1 ERR" & @CRLF )
    ConsoleWrite( "$pCondition1 OK" & @CRLF )

    Local $pButton1, $oButton1
    $oWindow1.FindFirst( $TreeScope_Descendants, $pCondition1, $pButton1 )
    $oButton1 = ObjCreateInterface( $pButton1, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
    If Not IsObj( $oButton1 ) Then Return ConsoleWrite( "$oButton1 ERR" & @CRLF )
    ConsoleWrite( "$oButton1 OK" & @CRLF )
EndFunc

Console Output

$oUIAutomation OK
$oDesktop OK
--- Find window/control ---
$pCondition0 OK
$oWindow1 OK
--- Find window/control ---
$pCondition1 OK
$oButton1 ERR

Code was generated by the "Create Sample Code" context menu from UIASpy. Why is it unable to find the next button?

See below for UIASpy Info

[LarsJ]

Exit, Because the process of creating 7z-files (selecting files, setting timestamp on updated files, generating the 7z-file and keeping track of updates) is automated.

philkryder, You can download a version of UIASpy that works on all Windows 10 versions in this post.

IanN1990, There is probably another control in a child window, also named Next. And this is the control you have identified. Because the control does not belong to $oWindow1, the creation of the $oButton1 object fails.

In the pictures you can see that the button is a direct child of the top window, so I think you can just replace TreeScope_Descendants with TreeScope_Children, so that FindFirst() only searches for direct children.

[Exit]

47 minutes ago, LarsJ said:

Exit, Because the process of creating 7z-files (selecting files, setting timestamp on updated files, generating the 7z-file and keeping track of updates) is automated.

7Z has the option to create a real zip file. So it would only be a small change in your automated process.

This would make the installation of an additional program superfluous. Not everybody has '7Z' installed.

[philkryder]

 

 

 

7 hours ago, LarsJ said:

philkryder, You can download a version of UIASpy that works on all Windows 10 versions in this post.

 

perfect! thanks! Phil

[philkryder]

Lars - 

I've just fired up the UIASpy_Gui().
I click on a Control Element and the Red Rectangle is about a one centimeter higher than the control.
the rectangle shows:
$UIA_BoundingRectanglePropertyId                    l=11,t=186,w=38,h=16
but when I hover in the middle of the control with the Auto IT info tool it shows the mouse about 7 greater (ie lower) for the y position.
Position:    31, 193

The Red Rectangle is entirely above the actual control shown on the screen.

This is windows 1709. 

 

 

[LarsJ]

The version you have downloaded is an old version where not all errors are corrected.

I don't want to update the old version but instead correct the current UIASpy so it can be used on a non-updated Windows 10. But it probably won't be until the fall.

[IanN1990]

15 hours ago, LarsJ said:

IanN1990, There is probably another control in a child window, also named Next. And this is the control you have identified. Because the control does not belong to $oWindow1, the creation of the $oButton1 object fails.

In the pictures you can see that the button is a direct child of the top window, so I think you can just replace TreeScope_Descendants with TreeScope_Children, so that FindFirst() only searches for direct children.

Thank you that was perfect.
In case anyone else runs into a similar issue here is the working code. 

expandcollapse popup

#include "CUIAutomation2.au3"

Example()

Func Example()
    ; Create UI Automation object
    Local $oUIAutomation = ObjCreateInterface( $sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation )
    If Not IsObj( $oUIAutomation ) Then Return ConsoleWrite( "$oUIAutomation ERR" & @CRLF )
    ConsoleWrite( "$oUIAutomation OK" & @CRLF )

     ; Get Desktop element
    Local $pDesktop, $oDesktop
    $oUIAutomation.GetRootElement( $pDesktop )
    $oDesktop = ObjCreateInterface( $pDesktop, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
    If Not IsObj( $oDesktop ) Then Return ConsoleWrite( "$oDesktop ERR" & @CRLF )
    ConsoleWrite( "$oDesktop OK" & @CRLF )

    ; --- Find window ---

    ConsoleWrite( "--- Find window ---" & @CRLF )

    Local $pCondition0
    $oUIAutomation.CreatePropertyCondition( $UIA_NamePropertyId, "XenDesktop 7.6.300", $pCondition0 )
    If Not $pCondition0 Then Return ConsoleWrite( "$pCondition0 ERR" & @CRLF )
    ConsoleWrite( "$pCondition0 OK" & @CRLF )

    Local $pWindow1, $oWindow1
    $oDesktop.FindFirst( $TreeScope_Children, $pCondition0, $pWindow1 )
    $oWindow1 = ObjCreateInterface( $pWindow1, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
    If Not IsObj( $oWindow1 ) Then Return ConsoleWrite( "$oWindow1 ERR" & @CRLF )
    ConsoleWrite( "$oWindow1 OK" & @CRLF )

    ; --- Find control ---

    ConsoleWrite( "--- Find Control ---" & @CRLF )

    Local $pCondition1
    $oUIAutomation.CreatePropertyCondition( $UIA_NamePropertyId, "Next", $pCondition1 )
    If Not $pCondition1 Then Return ConsoleWrite( "$pCondition1 ERR" & @CRLF )
    ConsoleWrite( "$pCondition1 OK" & @CRLF )

    Local $pButton1, $oButton1
    $oWindow1.FindFirst( $TreeScope_Children, $pCondition1, $pButton1 )
    $oButton1 = ObjCreateInterface( $pButton1, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
    If Not IsObj( $oButton1 ) Then Return ConsoleWrite( "$oButton1 ERR" & @CRLF )
    ConsoleWrite( "$oButton1 OK" & @CRLF )

    Local $pInvoke, $oInvoke
    $oButton1.GetCurrentPattern( $UIA_InvokePatternId, $pInvoke)
    $oInvoke = ObjCreateInterface( $pInvoke, $sIID_IUIAutomationInvokePattern, $dtagIUIAutomationInvokePattern )
    If Not IsObj( $oInvoke ) Then Return ConsoleWrite( "$oInvoke ERR" & @CRLF )
    ConsoleWrite( "$oInvoke OK" & @CRLF )

    $oInvoke.Invoke()
EndFunc

 

[philkryder]

 

 

1

11 hours ago, LarsJ said:

The version you have downloaded is an old version where not all errors are corrected.

I don't want to update the old version but instead correct the current UIASpy so it can be used on a non-updated Windows 10. But it probably won't be until the fall.

Sounds good. I'll pull out that W10 1809 ISO that I've been waiting for McAfee to support. Don't spend time on the past for me.
FotF - Focus on the Future.
Your time is too valuable building NEW FUNCTION to spend it on past-tense-trivia - PTT.

Phil

[LarsJ]

philkryder, I've corrected the error with the red rectangle in the Windows 7 version of UIASpy, which you can find near the bottom of this post. Select the correct screen scaling in the Options menu.

Edited May 25, 2019 by LarsJ
Screen scaling

[IanN1990]

Is there another way of selecting items from combo boxes. I have tried everything i can think of and though the below code works it takes focus away from the user
 

expandcollapse popup

Func ComboboxSelect()
    ConsoleWrite( "--- Find ComboBox ---" & @CRLF )

    Local $pCondition0
    $oUIAutomation.CreatePropertyCondition( $UIA_AutomationIdPropertyId, "cmbAddressType", $pCondition0 )
    If Not $pCondition0 Then Return ConsoleWrite( "$pCondition0 ERR" & @CRLF )
    ConsoleWrite( "$pCondition0 OK" & @CRLF )

    Local $pComboBox1, $oComboBox1
    $oWindow1.FindFirst( $TreeScope_Descendants, $pCondition0, $pComboBox1 )
    $oComboBox1 = ObjCreateInterface( $pComboBox1, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
    If Not IsObj( $oComboBox1 ) Then Return ConsoleWrite( "$oComboBox1 ERR" & @CRLF )
    ConsoleWrite( "$oComboBox1 OK" & @CRLF )

    Local $pInvoke, $oInvoke
    $oComboBox1.GetCurrentPattern( $UIA_ExpandCollapsePatternId, $pInvoke)
    $oInvoke = ObjCreateInterface( $pInvoke, $sIID_IUIAutomationExpandCollapsePattern, $dtagIUIAutomationExpandCollapsePattern )
    If Not IsObj( $oInvoke ) Then Return ConsoleWrite( "$oInvoke ERR" & @CRLF )
    ConsoleWrite( "$oInvoke OK" & @CRLF )
    $oInvoke.Expand()
    
    ConsoleWrite( "--- Find Do it Later ---" & @CRLF )
    Local $pCondition4
    $oUIAutomation.CreatePropertyCondition( $UIA_NamePropertyId, "Do it later", $pCondition4 )
    If Not $pCondition4 Then Return ConsoleWrite( "$pCondition4 ERR" & @CRLF )
    ConsoleWrite( "$pCondition4 OK" & @CRLF )

    Local $pListItem1, $oListItem1
    $oComboBox1.FindFirst( $TreeScope_Children, $pCondition4, $pListItem1 )
    $oListItem1 = ObjCreateInterface( $pListItem1, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
    If Not IsObj( $oListItem1 ) Then Return ConsoleWrite( "$oListItem1 ERR" & @CRLF )
    ConsoleWrite( "$oListItem1 OK" & @CRLF )

    Local $pInvoke1, $oInvoke1
    $oListItem1.GetCurrentPattern( $UIA_SelectionItemPatternId, $pInvoke1)
    $oInvoke1 = ObjCreateInterface( $pInvoke1, $sIID_IUIAutomationSelectionItemPattern, $dtagIUIAutomationSelectionItemPattern )
    If Not IsObj( $oInvoke1 ) Then Return ConsoleWrite( "$oInvoke ERR" & @CRLF )
    ConsoleWrite( "$oInvoke OK" & @CRLF )
    $oInvoke1.Select()
    
    ;Close ComboBox
    $oInvoke.Collapse()
EndFunc

 

Edited May 25, 2019 by IanN1990

[IanN1990]

Update. After more troubleshooting i found a combo-box items only generate it's list after it has been opened. A workaround for this is the code below which opens / closes the combo but doesn't take focus. 

; --- Find window/control ---
    ConsoleWrite( "--- Find window/control ---" & @CRLF )

    Local $pCondition0
    $oUIAutomation.CreatePropertyCondition( $UIA_AutomationIdPropertyId, "cmbAddressType", $pCondition0 )
    If Not $pCondition0 Then Return ConsoleWrite( "$pCondition0 ERR" & @CRLF )
    ConsoleWrite( "$pCondition0 OK" & @CRLF )

    Local $pComboBox1, $oComboBox1
    $oWindow1.FindFirst( $TreeScope_Descendants, $pCondition0, $pComboBox1 )
    $oComboBox1 = ObjCreateInterface( $pComboBox1, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
    If Not IsObj( $oComboBox1 ) Then Return ConsoleWrite( "$oComboBox1 ERR" & @CRLF )
    ConsoleWrite( "$oComboBox1 OK" & @CRLF )

    Local $pInvoke, $oInvoke
    $oComboBox1.GetCurrentPattern( $UIA_LegacyIAccessiblePatternId, $pInvoke)
    $oInvoke = ObjCreateInterface( $pInvoke, $sIID_IUIAutomationLegacyIAccessiblePattern, $dtagIUIAutomationLegacyIAccessiblePattern )
    If Not IsObj( $oInvoke ) Then Return ConsoleWrite( "$oInvoke ERR" & @CRLF )
    ConsoleWrite( "$oInvoke OK" & @CRLF )
    $oInvoke.DoDefaultAction()
    $oInvoke.DoDefaultAction()

I also found using the Legacy Select lets me select my item without taking focus. I don't know why it needed the number 3 but without it didn't work ^^. 

;Find List Item
    Local $pCondition4
    $oUIAutomation.CreatePropertyCondition( $UIA_NamePropertyId, "Do it later", $pCondition4 )
    If Not $pCondition4 Then Return ConsoleWrite( "$pCondition4 ERR" & @CRLF )
    ConsoleWrite( "$pCondition4 OK" & @CRLF )

    Local $pListItem1, $oListItem1
    $oComboBox1.FindFirst( $TreeScope_Children, $pCondition4, $pListItem1 )
    $oListItem1 = ObjCreateInterface( $pListItem1, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
    If Not IsObj( $oListItem1 ) Then Return ConsoleWrite( "$oListItem1 ERR" & @CRLF )
    ConsoleWrite( "$oListItem1 OK" & @CRLF )

    Local $pInvoke, $oInvoke
    $oListItem1.GetCurrentPattern( $UIA_LegacyIAccessiblePatternId, $pInvoke)
    $oInvoke = ObjCreateInterface( $pInvoke, $sIID_IUIAutomationLegacyIAccessiblePattern, $dtagIUIAutomationLegacyIAccessiblePattern )
    If Not IsObj( $oInvoke ) Then Return ConsoleWrite( "$oInvoke ERR" & @CRLF )
    ConsoleWrite( "$oInvoke OK" & @CRLF )

    $oInvoke.Select(3)

 

One random question!  Can IUI enable/disable controls similar to ControlEnable/ControlDisable?

Edited May 25, 2019 by IanN1990

[LarsJ]

Your code in the above post is actually MSAA code. That's why you through the Select parameter can specify how the selection should take place, eg. with regard to control focus. Well done that you found out.

When you perform a selection with pure UI Automation code as in the previous post, it's done as a user would do it manually. Ie. the control gets focus.

With UI Automation code you can generally automate tasks that a user can also do manually. It usually does not include setting control styles. But if the control has a window handle then you may be able to set the control style with functions like SetWindowLongPtr and similar functions.