AutoIt and Universal Windows Platform (UWP)

[Jibberish]

Hello all,

I searched the forums for UWP but got nothing helpful. Our team is going to move from our current code to UWP and rebuild our app.  I use AutoIt to automate long boring tests.

Does anyone know how AutoIt plays with UWP programs?

Thanks!

Jibberish

[mLipok]

I think @junkew will answer to you soon

 

[junkew]

@mLipok thx we should find out how people can find faq31 in an easy way.

@Jibberish i assume you mean universal windows platform. Try faq31 and try with simplespy what you get recognized. I assume all will be recognized.

[Jibberish]

Yes I did, and thanks!

[Jibberish]

I was having trouble finding simplespy, and was asking here where it was.

Then I found it.

Thanks

Jibberish

Edited December 8, 2017 by Jibberish
Complete rewrite

[careca]

I still think its a system that's way too confusing, i think autoit should start to integrate a system to handle those as it's increasingly more common to see questions about it.

[junkew]

Just let me know what is confusing with uiawrappers and i will try to enhance it to be more logical but bear in mind gui automation is not easy as there are to many exceptional controls hard to recognize.

[Earthshine]

Well this is why I switched to Coded UI 😇. Automated testing was never this easy.

Edited December 8, 2017 by Earthshine

[junkew]

@Earthshine Maybe then make a reference to Coded UI best practices to see what I can change in UIAwrappers.

I think you mean the developer builds the functional testcases also in his own programming language (white box) which is fine as long as the output can be understood by all parties in and outside of the team (either with a replayable video or a bunch of screenshots with subtitles to tell what functionally is happening)

The base of all (black box) GUI tools I have seen is about

• Use a bunch of 1,n properties to recognize your control and give them a logical programmatic name (object map, repository, application map)

More advanced tools

1. allow regex in the property identification values
2. Have runnable specification language (no Gherkin/BDD is not directly executable without reprogramming it in your implementation program language), your script is your documentation (or can be presented in such a way its understandable to business and IT people)
3. Support all type of UI Objects in a similar way (from mainframe telon screens up to the modern smart watch, 1 tool fits all)

AutoIt + UI Wrappers is wel suited for windows + browser application(s) and not suited for mainframe, phone, watch without a lot of work. 

Java widgets is somewhere in the middle doable from AutoIt with some Java Accessibility Bridge but not seen a full finished udf to handle stuff easily.

[Earthshine]

will do.

[careca]

16 hours ago, junkew said:

Just let me know what is confusing with uiawrappers and i will try to enhance it to be more logical but bear in mind gui automation is not easy as there are to many exceptional controls hard to recognize.

Im starting to get how it works, had to change a couple of code in the examples because of the different language for titles and such.

Now i got some code going but it takes forever between actions.

As for simplify stuff maybe simplespy could be added to autoit, and something about it in help file.

There could be examples with less code imo.

I cut down tons of lines for the example of the calculator, only needed to press 1 number to get how it works, baby steps.

Edited December 8, 2017 by careca

[Jibberish]

Thanks @careca, I followed your example with just pressing '1' in calc. However I get an error I don't understand.

#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <constants.au3>
#include <WinAPI.au3>
#include <debug.au3>
#include "CUIAutomation2.au3"
#include "UIAWrappers.au3"

#AutoIt3Wrapper_UseX64=Y  ;Should be used for stuff like tagpoint having right struct etc. when running on a 64 bits os
Local $oDesktop
run("calc.exe")

$oCalc=_UIA_getFirstObjectOfElement($oDesktop,"class:=CalcFrame", $treescope_children)
$oNotepad=_UIA_getFirstObjectOfElement($oDesktop,"class:=Notepad", $treescope_children)


$sText="1"
$oButton=_UIA_getFirstObjectOfElement($oCalc,"name:=" & $sText, $treescope_subtree)
$oInvokeP=_UIA_getpattern($oButton,$UIA_InvokePatternID)
$oInvokeP.Invoke

>Running:(3.3.14.2):C:\Program Files (x86)\AutoIt3\autoit3_x64.exe "C:\Scripts\Calc_Push_1_Test.au3"   
--> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop
"C:\Scripts\Calc_Push_1_Test.au3" (21) : ==> Variable must be of type "Object".:
$oInvokeP.Invoke
$oInvokeP^ ERROR
->15:22:02 AutoIt3.exe ended.rc:1
 

The calculator opens but no button push due to the error.

Running on Windows 10 RS3.

Calc_Push_1_Test.au3

[Jibberish]

Just noticed I had a leftover line from the original source:

$oNotepad=_UIA_getFirstObjectOfElement($oDesktop,"class:=Notepad", $treescope_children)

Removed it and still get the error, as I expected.

[Jibberish]

Using SimpleSpy and some of the code from above I got this working:

#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <constants.au3>
#include <WinAPI.au3>
#include <debug.au3>
#include "CUIAutomation2.au3"
#include "UIAWrappers.au3"

#AutoIt3Wrapper_UseX64=Y  ;Should be used for stuff like tagpoint having right struct etc. when running on a 64 bits os
Local $oDesktop
run("calc.exe")
;~ *** Standard code maintainable ***
AutoItSetOption("MustDeclareVars", 1)

_UIA_setVar("oP1","Title:=Calculator;controltype:=UIA_WindowControlTypeId;class:=ApplicationFrameWindow")   ;Calculator
_UIA_setVar("oP2","Title:=Calculator;controltype:=UIA_WindowControlTypeId;class:=Windows.UI.Core.CoreWindow")   ;Calculator
_UIA_setVar("oP3","Title:=;controltype:=UIA_GroupControlTypeId;class:=LandmarkTarget")  ;

;~ $oUIElement=_UIA_getObjectByFindAll("One.mainwindow", "title:=One;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)
_UIA_setVar("oUIElement","Title:=One;controltype:=UIA_ButtonControlTypeId;class:=Button") ;ControlType:=UIA_ButtonControlTypeId;classname:=Button")

;~ Actions split away from logical/technical definition above can come from configfiles

;~_UIA_Action("oP1","highlight")
_UIA_Action("oP1","setfocus")
;~_UIA_Action("oP2","highlight")
_UIA_Action("oP2","setfocus")
;~_UIA_Action("oP3","highlight")
_UIA_Action("oP3","setfocus")

_UIA_action("oUIElement","highlight")
_UIA_action("oUIElement","click")

Takes a while for the button to be pushed... about 8 seconds. I wonder if there is unnecessary code here. Haven't tried anything else yet. I wanted to post the solution I found so others didn't have to (other solutions are welcome however!)

[Jibberish]

There are two examples from SimpleSpy. The above is from the first example. This is from the 2nd example and starts calc and then apparently clicks the Windows button on my desktop.

 

#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <constants.au3>
#include <WinAPI.au3>
#include <debug.au3>
#include "CUIAutomation2.au3"
#include "UIAWrappers.au3"

#AutoIt3Wrapper_UseX64=Y  ;Should be used for stuff like tagpoint having right struct etc. when running on a 64 bits os
Local $oDesktop
run("calc.exe")
AutoItSetOption("MustDeclareVars", 1)

Local $oP2=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=Calculator;controltype:=UIA_WindowControlTypeId;class:=ApplicationFrameWindow", $treescope_children)
_UIA_Action($oP2,"setfocus")
Local $oP1=_UIA_getObjectByFindAll($oP2, "Title:=Calculator;controltype:=UIA_WindowControlTypeId;class:=Windows.UI.Core.CoreWindow", $treescope_children)
_UIA_Action($oP1,"setfocus")
Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=;controltype:=UIA_GroupControlTypeId;class:=LandmarkTarget", $treescope_children)
;~ First find the object in the parent before you can do something
;~$oUIElement=_UIA_getObjectByFindAll("One.mainwindow", "title:=One;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)
Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=One;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)
;~_UIA_action($oUIElement,"highlight")
_UIA_action($oUIElement,"click")

I had deleted the SimpleSpy output so I ran it again. The entire output is attached for anyone who wants to play with it to see if they get the same results as mine.

SimpleSpyOutputCalc.au3

[Jibberish]

And Finally 1+3=4

expandcollapse popup

#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <constants.au3>
#include <WinAPI.au3>
#include <debug.au3>
#include "CUIAutomation2.au3"
#include "UIAWrappers.au3"

#AutoIt3Wrapper_UseX64=Y  ;Should be used for stuff like tagpoint having right struct etc. when running on a 64 bits os
Local $oDesktop
Local $oUIElement, $oUIElement1, $oUIElement2, $oUIElement3
run("calc.exe")
;~ *** Standard code maintainable ***
AutoItSetOption("MustDeclareVars", 1)

_UIA_setVar("oP1","Title:=Calculator;controltype:=UIA_WindowControlTypeId;class:=ApplicationFrameWindow")   ;Calculator
_UIA_setVar("oP2","Title:=Calculator;controltype:=UIA_WindowControlTypeId;class:=Windows.UI.Core.CoreWindow")   ;Calculator
_UIA_setVar("oP3","Title:=;controltype:=UIA_GroupControlTypeId;class:=LandmarkTarget")  ;

$oUIElement=_UIA_getObjectByFindAll("One.mainwindow", "title:=One;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)
_UIA_setVar("oUIElement","Title:=One;controltype:=UIA_ButtonControlTypeId;class:=Button") ;ControlType:=UIA_ButtonControlTypeId;classname:=Button")

;~ Actions split away from logical/technical definition above can come from configfiles

;~_UIA_Action("oP1","highlight")
_UIA_Action("oP1","setfocus")
;~_UIA_Action("oP2","highlight")
_UIA_Action("oP2","setfocus")
;~_UIA_Action("oP3","highlight")
_UIA_Action("oP3","setfocus")

_UIA_action("oUIElement","highlight")
_UIA_action("oUIElement","click")

$oUIElement1 =_UIA_getObjectByFindAll("One.mainwindow", "title:=Plus;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)
_UIA_setVar("oUIElement1","Title:=Plus;controltype:=UIA_ButtonControlTypeId;class:=Button") ;ControlType:=UIA_ButtonControlTypeId;classname:=Button")

;~_UIA_Action("oP1","highlight")
_UIA_Action("oP1","setfocus")
;~_UIA_Action("oP2","highlight")
_UIA_Action("oP2","setfocus")
;~_UIA_Action("oP3","highlight")
_UIA_Action("oP3","setfocus")

_UIA_action("oUIElement1","highlight")
_UIA_action("oUIElement1","click")

$oUIElement2 =_UIA_getObjectByFindAll("One.mainwindow", "title:=Three;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)
_UIA_setVar("oUIElement2","Title:=Three;controltype:=UIA_ButtonControlTypeId;class:=Button") ;ControlType:=UIA_ButtonControlTypeId;classname:=Button")

;~_UIA_Action("oP1","highlight")
_UIA_Action("oP1","setfocus")
;~_UIA_Action("oP2","highlight")
_UIA_Action("oP2","setfocus")
;~_UIA_Action("oP3","highlight")
_UIA_Action("oP3","setfocus")

_UIA_action("oUIElement2","highlight")
_UIA_action("oUIElement2","click")

$oUIElement3 =_UIA_getObjectByFindAll("One.mainwindow", "title:=Equals;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)
_UIA_setVar("oUIElement3","Title:=Equals;controltype:=UIA_ButtonControlTypeId;class:=Button") ;ControlType:=UIA_ButtonControlTypeId;classname:=Button")

;~_UIA_Action("oP1","highlight")
_UIA_Action("oP1","setfocus")
;~_UIA_Action("oP2","highlight")
_UIA_Action("oP2","setfocus")
;~_UIA_Action("oP3","highlight")
_UIA_Action("oP3","setfocus")

_UIA_action("oUIElement3","highlight")
_UIA_action("oUIElement3","click")

I learn by example; I hope this helps others. However I have a long way to go!

If you have other solutions or ways to shorten this code, please feel free to post examples!

[junkew]

Will put below also as part of education in uia examples thread

1. Understand the concept of the element hierarchy.
   Especially the "technical" elements in the hierarchy confuses people.
   Easiest to learn is the spying tools inspect.exe or uiaverify.exe in windows sdk, see faq 31 for links.
   "Frustrating" is that not all elements in the hierarchy need to be focussed.
   Rule of thumb: Set focus to mainwindow below desktop, Set focus to element to act on
2. Understand the search scope
   Fastest way of finding elements is if you only search childs and not in grandgrandchilds.
   The uia wrapper has logic to fallback to search in itself a different full hierarchy if its not found in the direct childs which is slow by nature but in my solutions highly preffered when running unattended over not finding at all
   This goes fine as long as you understand bullet 1 properly otherwise you will have dozens of elements that take seconds to be found
   So this line can positively influence the speed of searching as it sets the starting point to search from

   _UIA_setVar("RTI.SEARCHCONTEXT", _UIA_action("title:=Calculator","getobject"))

    

3. Understand main window
   In general this is just the direct window under the parent desktop (so calculator window)
4. Understand the concept of identifying elements by 1,n technical properties

   _UIA_Action("title:=Calculator","highlight")

   above will highlight the window that has calculator in the name (case independt partial search in title). 1 property is in this case enough to identify the window and act on it. The disbenefit of above its not language neutral and so a better property to use for identifying is below
    

   _UIA_Action("class:=CalcFrame","highlight")

   Its a matter of style if you put more then 1 property if only 1 is already enough to identify. The more you put in especially class can help in speeding up the search internally when i will improve search algorithm over time. So advice is to keep controltype allways in as thats a base pattern in UIA. Class is also nice as thats the core of the win32 pattern (which is mainly used in AutoIt)

5. Understand the difference between short coding and maintainable coding
   Understand how the unwrapped core uia version would look like

   todo: make later example with core UIA interface

   
   So a basic wrapped short version can be like below

   _UIA_setVar("RTI.SEARCHCONTEXT", _UIA_action("title:=Calculator","getobject"))
   _UIA_Action("title:=Calculator","setfocus")
   _UIA_Action("Title:=One;controltype:=Button","click")
   _UIA_Action("Title:=Plus;controltype:=Button","click")
   _UIA_Action("Title:=Three;controltype:=Button","click")
   _UIA_Action("Title:=Equals;controltype:=Button","click")
   $result=_UIA_Action("Title:=Result;controltype:=Button","gettext")

   This is perfectly fine for a small demoprogram but gets more problematic when you are dealing with multiple screens or want to click 3  times the one button
   You can put the one definition in a variable

   $btnOne ="Title:=One;controltype:=Button"
   _UIA_Action("title:=Calculator","setfocus")
   _UIA_Action($btnOne,"click")
   _UIA_Action("Title:=Plus;controltype:=Button","click")
   _UIA_Action($btnOne,"click")
   _UIA_Action("Title:=Three;controltype:=Button","click")
   _UIA_Action($btnOne,"click")
   _UIA_Action("Title:=Equals;controltype:=Button","click")
   $result=_UIA_Action("Title:=Result;controltype:=Button","gettext")

   Other options are to put it in the array and as I do I feed my definition's from a UID file which is just a name / value properties file like

   calculator.mainwindow = title:=calculator
   calculator.button1    = title:=One
   calculator.button2    = title:=Two

    

6. Understand the slowness sometimes observed of the uiawrappers
   The tough part in any wrapper is how to determine the searchscope and where to start searching and how deep
   a. Starting on mainwindow and searching direct childs will result frequently in not finding the element as its on level 2 or 3
   b. Starting from last element acted on in the wrapper will lead to searching for button 2 below button 1 in hierarchy which is incorrect
   c. Hierarchy of browsers like chrome are sometimes terrible as it starts searching first in the html elements where you are interested in the toolbars.
   So search algorithm in the wrapper is far from perfect. I have some ideas to improve but not implemented them. Ideas to find deep in tree but excluding htmldocument, listitems when search is NOT on childs.

7. Understand the different elements around in the world
   Some elements need an invoke others need a click and some need both and some need first a setfocus before they really do what you logically want (click).
   Microsoft made an excellent UIA framework but unfortunately not all parties adapted it when building windows software.

8. Understand automation frameworks
   This is a nice summary but selenium is definitely no the only reference (selenium has many limitations as it only has focus on browsers/webdriver)
   http://www.softwaretestinghelp.com/test-automation-frameworks-selenium-tutorial-20/So maybe it looks complex compared to directly have coding like below (coded ui testing) the benefit of a framework is abstraction in multiple layers so you can generalize the generic parts like synchronizing, taking snapshots, logging etc after each step executed.

controlclick("class:=calc")

 

Edited December 31, 2017 by junkew

[Jibberish]

Junkew,

Thanks for the help. Yes your suggestions sped the test up quite a bit. I will implement your suggestions in my code.

I said above that our company is converting a Windows app to UWP, and I want to convert my automated tests to work with the new UWP interface.

I'm unable to share our app due to IP, so I built a basic UWP app that simulates part of an app we are converting to UWP. It's just 3 buttons with message boxes responding to clicking any of the main buttons. The package was too big to include, so I included the Visual Studio 2015 source, attached as UITest.7z.

Using your tools I am able to click the any of the 3 main buttons (AutoIt source below only clicks the first button) but am unable to get AutoIt to click the OK or Exit buttons that pop up after clicking one of the main buttons.

My AutoIt source from SimpleSpy is below. I have not implemented any of your recommendations from above yet, so pardon the messy code. 

#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

Local $oP1=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=UITest;controltype:=UIA_WindowControlTypeId;class:=ApplicationFrameWindow", $treescope_children)
_UIA_Action($oP1,"setfocus")
Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=UITest;controltype:=UIA_WindowControlTypeId;class:=Windows.UI.Core.CoreWindow", $treescope_children)
_UIA_Action($oP0,"setfocus")
;~ First find the object in the parent before you can do something
;~$oUIElement=_UIA_getObjectByFindAll("ButtonOne.mainwindow", "title:=Button One;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)
Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=Button One;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)
;~_UIA_action($oUIElement,"highlight")
_UIA_action($oUIElement,"click")

;~~~
; ^ Works
;
; v Doesn't work
;~~~

Local $oP5=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=UITest;controltype:=UIA_WindowControlTypeId;class:=ApplicationFrameWindow", $treescope_children)
_UIA_Action($oP5,"setfocus")
Local $oP4=_UIA_getObjectByFindAll($oP5, "Title:=;controltype:=UIA_WindowControlTypeId;class:=Shell_Dialog", $treescope_children)
_UIA_Action($oP4,"setfocus")
Local $oP3=_UIA_getObjectByFindAll($oP4, "Title:=Dialog window;controltype:=UIA_PaneControlTypeId;class:=FlyoutElement", $treescope_children)
_UIA_Action($oP3,"setfocus")
;~ First find the object in the parent before you can do something
$oUIElement=_UIA_getObjectByFindAll("OK.mainwindow", "title:=OK;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)
Local $oUIElement=_UIA_getObjectByFindAll($oP3, "title:=OK;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)
_UIA_action($oUIElement,"highlight")
_UIA_action($oUIElement,"click")

As I said, the first button gets clicked by the app, opening the dialog box. However I can't get the second part to click the OK button. Apologies if I have missed something simple. I renamed the $op variables in the second part in case duplication was causing problems. I also uncommented two lines in the bottom part just in case they were necessary (no joy there...)

If you see where I'm going wrong trying to get the OK button to press, please let me know! I truly appreciate your assistance!

Jibs

UITest.7z

[junkew]

dont do this if its not a mainwindow

"OK.mainwindow"

the "keyword" mainwindow triggers an implicit RTI.SearchContext to be set to search in this case only below the OK button.

A mainwindow is in 99% of the cases just the window thats a child of the desktop. Some exceptions are MDI windows or different browsertabs. In future version I probably will remove this logic as it confuses people

 

[junkew]

some additional comment,
 

It will not work as you have

a. not given any reference for "ok.mainwindow"

b. as given above do not use .mainwindow it its not a window control

what happens if you just remove this line?

;~ First find the object in the parent before you can do something
;~ $oUIElement=_UIA_getObjectByFindAll("OK.mainwindow", "title:=OK;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)

Add something for RTI.SearchContext like this to specifically tell to search in the the flyoutelement window

;~ First find the object in the parent before you can do something
;~ $oUIElement=_UIA_getObjectByFindAll("OK.mainwindow", "title:=OK;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)
_UIA_setVar("RTI.SEARCHCONTEXT", _UIA_action("Title:=Dialog window;controltype:=UIA_PaneControlTypeId;class:=FlyoutElement","getobject"))
Local $oUIElement=_UIA_getObjectByFindAll($oP3, "title:=OK;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)
_UIA_action($oUIElement,"highlight")
_UIA_action($oUIElement,"click")