Using UI Automation Code in AutoIt

[mikester]

This is fantastic, great work! Any chance I could get some help w/how to locate a parent element?  Maybe just a small example? I've tried to work out the syntax for a couple of days and still cannot figure it out. I'm able to locate the element by name but that shows as the $UIA_NamePropertyId (Text) and there are 3 siblings for: the Button (to expand/collapse), Image (the folder), and "Edit: TextBox". What I need to be able to do is select the parent element so that its properties show up on the screen.

Any help would be GREATLY appreciated! This is the last piece of the puzzle before I can automate the entire project. Thanks in advance!!!

I forgot to mention I working w/a tree, if that matters.

Edited July 24, 2019 by mikester
forgot to mention it was a tree element

[mikester]

If it helps, here's the latest section of code I tried, although I think I was closer prior to trying this:

Local $pRawWalker, $oRawWalker
$oUIAutomation.RawViewWalker( $pRawWalker )
$oRawWalker = ObjCreateInterface( $pRawWalker, $sIID_IUIAutomationTreeWalker, $dtagIUIAutomationTreeWalker )

Local $pTreeNodeItem1, $oTreeNodeItem1
$oRawWalker.GetParentElement( $pText1, $pTreeNodeItem1)
$oTreeNodeItem1 = ObjCreateInterface( $pTreeNodeItem1, $sIID_IUIAutomationTreeWalker, $dtagIUIAutomationTreeWalker )
If Not IsObj( $oTreeNodeItem1 ) Then Return ConsoleWrite( "$oTreeNodeItem1 ERR" & @CRLF )
ConsoleWrite( "$oTreeNodeItem1 OK" & @CRLF )

 

Edited July 24, 2019 by mikester

[mikester]

I worked it out. I looked at ex1_Treewalker.au3 from junkew a little closer and that got me to this:

Local $pRawWalker, $oRawWalker
$oUIAutomation.RawViewWalker( $pRawWalker )

$oRawWalker = ObjCreateInterface( $pRawWalker, $sIID_IUIAutomationTreeWalker, $dtagIUIAutomationTreeWalker )

Local $pTreeNodeItem1, $oTreeNodeItem1
$oRawWalker.GetParentElement( $pText1, $pTreeNodeItem1)
$oTreeNodeItem1 = ObjCreateInterface( $pTreeNodeItem1, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
If Not IsObj( $oTreeNodeItem1 ) Then Return ConsoleWrite( "$oTreeNodeItem1 ERR" & @CRLF )
ConsoleWrite( "$oTreeNodeItem1 OK" & @CRLF )

Hope I didn't waste anyone's time.

[junkew]

Simplespy will help you how to get to grand parents just as treewalker examples.

[mikester]

Thanks junkew. My problem was that there is nothing unique about the parent, grandparent, or great grandparent. They all have the same properties so I needed a way to just get the parent directly.

[cyxstudio]

-- wrong thread

Edited September 18, 2019 by cyxstudio

[cyxstudio]

"C:\Users\hssoyr9\Documents\Outlook Files\Outlook\SeachHistory.au3" (64) : ==> Variable used without being declared.:
$oEdit1.GetCurrentPattern( $UIA_ValuePatternId, $pValue )

 

 

UIA_ValuePatternId is not recognized. did i miss something or forgot to include something?

[Earthshine]

you forgot to include your script and all the details required for proper help, such as, what application you are automating, etc..

[Musashi]

51 minutes ago, cyxstudio said:

UIA_ValuePatternId is not recognized. did i miss something or forgot to include something?

Global Const $UIA_ValuePatternId             = 10002

is declared in : UIA_Constants.au3

[LukeLe]

Big big big thanks to @LarsJ for the very details examples and nice UISpy program.

I learned so much in your coding convention. 

and a big thanks to  @junkew also, you're the first one who helped to open the gate of UIAutomation world to me.

Still, have much much more to learn 😀

By the way, I want to mention a case that I faced when trying to go through the Notepad examples: The  Click Save As... menu item  failed.

I ran the example in the path

Quote

Examples\1) Automation\Notepad\Windows 7, 10\1) Create code\NotepadAll.au3

However, below is the log of SciTE

>"C:\Program Files (x86)\AutoIt3\SciTE\..\AutoIt3.exe" "C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.au3" /run /prod /ErrorStdOut /in "E:\SelfStudy\AutoIT\Lib\UIAutomation\Larj\Examples\1) Automation\Notepad\Windows 7, 10\1) Create code\NotepadAll.au3" /UserParams    
+>14:44:19 Starting AutoIt3Wrapper v.19.102.1901.0 SciTE v.4.1.2.0   Keyboard:00000409  OS:WIN_7/Service Pack 1  CPU:X64 OS:X64  Environment(Language:0409)  CodePage:0  utf8.auto.check:4
+>         SciTEDir => C:\Program Files (x86)\AutoIt3\SciTE   UserDir => C:\Users\V0924813\AppData\Local\AutoIt v3\SciTE\AutoIt3Wrapper   SCITE_USERHOME => C:\Users\V0924813\AppData\Local\AutoIt v3\SciTE 
>Running AU3Check (3.3.14.5)  from:C:\Program Files (x86)\AutoIt3  input:E:\SelfStudy\AutoIT\Lib\UIAutomation\Larj\Examples\1) Automation\Notepad\Windows 7, 10\1) Create code\NotepadAll.au3
+>14:44:20 AU3Check ended.rc:0
>Running:(3.3.14.5):C:\Program Files (x86)\AutoIt3\autoit3.exe "E:\SelfStudy\AutoIT\Lib\UIAutomation\Larj\Examples\1) Automation\Notepad\Windows 7, 10\1) Create code\NotepadAll.au3"    
+>Setting Hotkeys...--> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop
$oUIAutomation OK
$oDesktop OK
--- Notepad window ---
$pCondition OK
$oNotepad OK
--- Fill Edit element ---
$pCondition OK
$oEdit OK
$oValue OK
--- Open File menu ---
$pCondition1 OK
$pCondition2 OK
$pCondition OK
$oFile OK
$oInvoke OK
--- Click "Save As..." menu ---
$pCondition2 OK
$pCondition OK
$oSaveAs ERR
+>14:44:21 AutoIt3.exe ended.rc:0
+>14:44:21 AutoIt3Wrapper Finished.
>Exit code: 0    Time: 2.319

The reason is $oInvokePattern.Invoke() seems to open the File menu, then close the menu right away. So there is no Save as... menu item found.

After a while, I noticed that File menu has ExpandCollapse Pattern Methods also

Then, instead of using Invoke() to open File menu, I used the Expand() method, and it worked. The example worked very well!

Below is the code I used replacing Invoke() by Expand()

Local $pInvokePattern, $oInvokePattern
    $oMenuItem.GetCurrentPattern($UIA_InvokePatternId, $pInvokePattern)
    $oInvokePattern = ObjCreateInterface($pInvokePattern, $sIID_IUIAutomationInvokePattern, $dtagIUIAutomationInvokePattern)
    If Not IsObj($oInvokePattern) Then Return ConsoleWrite("$oInvokePattern ERR" & @CRLF)
    ConsoleWrite("$oInvokePattern OK" & @CRLF)
    ;~ $oInvokePattern.Invoke()
    Sleep(1000)
    ; --- ExpandCollapse Pattern (action) Object ---

    ConsoleWrite("--- ExpandCollapse Pattern (action) Object ---" & @CRLF)

    Local $pExpandCollapsePattern1, $oExpandCollapsePattern1
    $oMenuItem.GetCurrentPattern($UIA_ExpandCollapsePatternId, $pExpandCollapsePattern1)
    $oExpandCollapsePattern1 = ObjCreateInterface($pExpandCollapsePattern1, $sIID_IUIAutomationExpandCollapsePattern, $dtagIUIAutomationExpandCollapsePattern)
    If Not IsObj($oExpandCollapsePattern1) Then Return ConsoleWrite("$oExpandCollapsePattern1 ERR" & @CRLF)
    ConsoleWrite("$oExpandCollapsePattern1 OK" & @CRLF)
    $oExpandCollapsePattern1.Expand()

    Sleep(1000)
    ; --- Click Save As... menu item ---

 

Edited October 3, 2019 by LukeLe
update a case that Notepad automation fails

[junkew]

Yes it can really be frustrating to find out if you have to use invoke click expand patterns. I frequently just identify the element and based on xyhw just send a mousemove and mouseclick instead of using the pattern methods.

[LarsJ]

LukeLe, The reason for the error is that the Sleep(100) command after the $oInvoke.Invoke() statement that opens the File menu is too short. Use Sleep(200) or Sleep(300) and the code will work. The amount of delay needed depends on how much is going on on the PC at the moment the Invoke() method is executed. But the delay should preferably be as short as possible.

A better way is to use UI Automation code to verify that the File menu is open before the code continues following the Invoke(). But it might be too much to make of a simple example.

At least with regard to Microsoft programs, it's my experience that the Invoke() command works, if available.

But good that you found an alternative solution.

Later this fall I'll update the UI Automation examples with new code in the Windows 10 1809 version.

[LukeLe]

@LarsJ Sorry, I changed to 200, and 300 but none of them work. The menu just appears like a zap then closes.

[LarsJ]

Strange. I'll add a link to your post in the example if there are others who get the same problem.

[rlvitorino]

I need Help in it.Why does this error occur?

 

expandcollapse popup

#AutoIt3Wrapper_Au3Check_Parameters=-d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6 -w 7

;#AutoIt3Wrapper_UseX64=n ; If target application is running as 32 bit code
#AutoIt3Wrapper_UseX64=y ; If target application is running as 64 bit code

#include "CUIAutomation2.au3"

Opt("MustDeclareVars", 1)

Run("Calc")
Sleep(1000)

Example()

Func Example()
    ; Cria UI Automation object
    Local $oUIAutomation = ObjCreateInterface($sCLSID_CUIAutomation, $sIID_IUIAutomation, $dtagIUIAutomation)
    If Not IsObj($oUIAutomation) Then Return ConsoleWrite("$oUIAutomation ERR" & @CRLF)
    ConsoleWrite("$oUIAutomation OK" & @CRLF)

    ; Get Desktop Element
    Local $pDesktop, $oDesktop
    $oUIAutomation.GetRootElement($pDesktop)
    $oDesktop = ObjCreateInterface($pDesktop, $sIID_IUIAutomationElement, $dtagIUIAutomationElement)
    If Not IsObj($oDesktop) Then Return ConsoleWrite("$oDesktop ERR" & @CRLF)
    ConsoleWrite("$oDesktop OK" & @CRLF)

    ;Espera Jameça Calculator
    WinWaitActive("Calculator ‎- Calculator", "",5)

    ConsoleWrite("--- Procura Calculator Window ---" & @CRLF)

    ; --- Copy element info ---

;~  ; $UIA_ClassNamePropertyId                            Button

    ; --- Find window/control ---

    ConsoleWrite( "--- Find window/control ---" & @CRLF )

    Local $pCondition0
    $oUIAutomation.CreatePropertyCondition( $UIA_ClassNamePropertyId, "Button", $pCondition0 )
    If Not $pCondition0 Then Return ConsoleWrite( "$pCondition0 ERR" & @CRLF )
    ConsoleWrite( "$pCondition0 OK" & @CRLF )

    Local $pButton1, $oButton1,$sIID_IUIAutomationElement8,$dtag_IUIAutomationElement8,$oParent
    $oParent.FindFirst( 4, $pCondition0, $pButton1 )
    $oButton1 = ObjCreateInterface( $pButton1, $sIID_IUIAutomationElement8, $dtag_IUIAutomationElement8 )
    If Not IsObj( $oButton1 ) Then Return ConsoleWrite( "$oButton1 ERR" & @CRLF )
    ConsoleWrite( "$oButton1 OK" & @CRLF )

EndFunc   ;==>Example

 

>"C:\Users\x209080\Desktop\Notepad++\AutoIt3\SciTE\..\AutoIt3.exe" "C:\Users\x209080\Desktop\Notepad++\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.au3" /run /prod /ErrorStdOut /in "C:\Users\x209080\Desktop\UIAExamples\TesteUIA.au3" /UserParams    
+>09:31:18 Starting AutoIt3Wrapper v.17.224.935.0 SciTE v.3.7.3.0   Keyboard:00010416  OS:WIN_10/  CPU:X64 OS:X64  Environment(Language:0409)  CodePage:0  utf8.auto.check:4
+>         SciTEDir => C:\Users\x209080\Desktop\Notepad++\AutoIt3\SciTE   UserDir => C:\Users\x209080\Desktop\Notepad++\AutoIt3\SciTE\AutoIt3Wrapper
>Running AU3Check (3.3.14.5)  params:-d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6 -w 7  from:C:\Users\x209080\Desktop\Notepad++\AutoIt3  input:C:\Users\x209080\Desktop\UIAExamples\TesteUIA.au3
+>09:31:19 AU3Check ended.rc:0
>Running:(3.3.14.5):C:\Users\x209080\Desktop\Notepad++\AutoIt3\autoit3_x64.exe "C:\Users\x209080\Desktop\UIAExamples\TesteUIA.au3"    
--> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop
$oUIAutomation OK
$oDesktop OK
--- Procura Calculator Window ---
--- Find window/control ---
$pCondition0 OK
"C:\Users\x209080\Desktop\UIAExamples\TesteUIA.au3" (47) : ==> Variable must be of type "Object".:
$oParent.FindFirst( 4, $pCondition0, $pButton1 )
$oParent^ ERROR
->09:31:26 AutoIt3.exe ended.rc:1
+>09:31:26 AutoIt3Wrapper Finished.
>Exit code: 1    Time: 9.801

 

[Musashi]

Quote

"C:\Users\x209080\Desktop\UIAExamples\TesteUIA.au3" (47) : ==> Variable must be of type "Object".:
$oParent.FindFirst( 4, $pCondition0, $pButton1 )
$oParent^ ERROR

(only at first glance)

Like the error message suggests, $oParent is not an object. Something like  $oParent = ObjCreateInterface(... is probably missing here.

[Exit]

use $odesktop instead of $oparent.

Or insert statement

$oparent=$odesktop

[LarsJ]

Starting a FindFirst-search for a UI element directly from the desktop is not a good idea if the only criterion is that it should be a button. Starting a search from the desktop will search through all application windows (usually 5 - 10 windows). And when it's a FindFirst-search for a button, the search will stop at the first button found in one of those windows. The probability that this is the right button is close to zero.

In UI Automation code you usually only use the desktop to find the application top window. And then you start all other searches from the application window or possibly a child window within the application.

Because all windows usually contain multiple buttons, you must at least combine the control type with the name of the button when performing a FindFirst-search to identify a button.

$oParent in rlvitorino's post should be the application top window. In my examples, I usually use $oWindow to designate this window.

[seadoggie01]

First, I want to say thank you so much for the tutorials, they are incredibly helpful!

Unfortunately, I think I threw myself in the deep end a bit. I'm trying to click a button on a window that looks like this (Import from disk - Single Mode)

The problem is that using .Invoke and .Expand both take the default action, which opens a File Open dialog and freezes the script until the file is selected. Interestingly, UIA spy doesn't see the secondary menu until the dropdown arrow is clicked

[LarsJ]

Window and control automation software cannot detect a menu until it's open.

In a toolbar menu item, the dropdown arrow is usually a standalone control. Isn't it possible to identify the dropdown arrow within the toolbar menu item and then directly click the arrow with Invoke? And then you can click "Import from disk - Single Mode" in the menu with a new Invoke.

Edited November 6, 2019 by LarsJ