UIASpy - UI Automation Spy Tool

[LarsJ]

The GUI looks like shown in the picture:

It contains a main menu, a treeview in left pane that lists UI Automation elements, a listview in right pane that displays detail information for the selected element, and a splitterbar to adjust the width of the two panes.

 

Detect element
The "Detect element" main menu is used to detect elements under the mouse with function keys F1 - F4.
See How to topics number 2.

 

Left pane
The left pane is a treeview that initially lists open programs and windows. Generally, the left pane shows UI Automation elements represented as treeview items.

 

Right pane
The right pane is a listview that displays detail information for the selected treeview element. The information is grouped into different types of information. Only the relevant information for each type is displayed.

 

Invalid elements
When a window is closed the elements in UIASpy becomes invalid. But the elements are not deleted in UIASpy and can still be handled and selected in the treeview. Detail information for an element that was calculated before the window was closed can still be seen in the listview.

In this way it's possible for UIASpy to show information for eg. a context menu that is closed as soon as it loses focus.

 

Listview features
Listview features

 

Help system
With completion of the Help system, a lot of information has been moved from this post and other posts into the Help system.

 

Sample code creation
Sample code creation is implemented through the UI element Detail info listview page and through the Sample code main menu.

Sample code creates code snippets based on data in one or more selected rows in the Detail info listview page or based on the clicked Sample code menu item.

UI Automation code is largely based on COM interface objects and thus on the function ObjCreateInterface(), which is one of the advanced functions in AutoIt.

The main purpose of Sample code functionality is to eliminate the complexity of ObjCreateInterface(). Sample code creation either through the Detail info listview page or through the Sample code main menu can create all the interface objects required in UI Automation code.

Another purpose is of course to make it easier and faster to implement code for simple automation tasks. Through Sample code creation, you can more or less create all code in a simple automation task:

• Create UI Automation initial code
• Create condition and find application window
• Create condition and find control in window
• Get information about windows and controls
• Create pattern objects to perform actions
• Get information related to pattern objects
• Perform actions with pattern object methods
• Add a Sleep() statement if necessary

Note that the UI element used in sample code is named after the selected element in the treeview.

Also note that both the Sample code menu items and the sections in the Detail info listview page are placed in approximately the order they are used in a simple automation task.

Sample code creation through Detail info listview page

Sample code creation through Sample code main menu

 

Supported applications

• Most browsers including
  • Google Chrome
    • To be able to spy on web content in Google Chrome it's necessary to enable accessibility by entering chrome://accessibility/ in the address bar of a new tab item, and then check the five check boxes that are located in a column in upper left corner down along the left edge. Then the accessibility tab can be closed again. It's a global setting that applies to all open and new tabs until Chrome is closed.
      Without accessibility enabled you are only able to investigate the outer elements of Chrome but not web content.
  • Internet Explorer
  • Microsoft Edge
  • Mozilla Firefox
• Most Microsoft applications and applications developed with Microsoft software including
  • Classic Windows applications based on the standard control library
  • Modern Universal Windows Platform apps like the Windows 10 Calculator
  • Programs implemented through .NET Framework eg. Windows Forms applications
• Most applications provided by major development companies

 

Automation issues
Automation issues

 

How to topics
If UI Automation or the UIASpy tool is new to you, then you should read the How to topics.

 

Examples
Examples that demonstrates the features of UIASpy:

• Automating Notepad. Very detailed example.
• Automating Notepad - Windows XP

Examples about Sample code creation:

• Automating Notepad with Sample code - step by step
• Automating Notepad with Sample code - all at once
• Chrome - Clicking an extension. Compact summary example.

Click Save As... issue in Notepad examples:

• Using Expand() instead of Invoke()

 

Updates
Windows 8, Windows 8.1 and Windows 10 updates

 

Threads

• UI Automation UDFs contains all include files.
• In Using UI Automation Code in AutoIt you can find and download examples and read information about using UIA code.
• UI Automation Events is about implementing event handlers and includes GUIs to detect events.
• IUIAutomation MS framework automate chrome, FF, IE, .... created by junkew August 2013 is the first AutoIt thread on UIA code.

 

Zip-file
The zip contains source files for UIASpy GUI.

Note that UI Automation UDFs must be installed in the Includes folder.

You need AutoIt 3.3.12 or later. Tested on Windows XP, Windows 7 and Windows 10.

Comments are welcome. Let me know if there are any issues.

UIASpy.7z

Edited February 9, 2020 by LarsJ
Updates, new zip-file

[mLipok]

as always

Awesome .. tool !
 

[UEZ]

Nothing appears when I start UIASpy.au3 and the script ends -> I've to comment out

If WinExists( "UIASpy - UI Automation Spy Tool" ) Then Return

otherwise it returns always.

 

Nice work. I never worked with UI but maybe somedays...

[Danp2]

2 hours ago, UEZ said:

Nothing appears when I start UIASpy.au3 and the script ends

Same here. Also having issue with the UI highlighting occurring in the wrong location. This appears to be due to the display settings in Windows, which are as follows --

[junkew]

Looks nice, gave no gui.

For whatever reason the - is not fine in winexist (i assume its interpreted as a regex)

changed -minus to a dot for a quick fix

Func UIASpy_Gui()
    ; Only one UIASpy at a time
    consolewrite("hello : " & WinExists( "UIASpy . UI Automation Spy Tool") & @CRLF)
    If WinExists( "UIASpy . UI Automation Spy Tool" ) Then Return
    consolewrite("hello 2")

 

[argumentum]

Was not working for me. I'm like ?!?!?. lol

;~  If WinExists( "UIASpy - UI Automation Spy Tool" ) Then Return
    If WinExists( "[TITLE:UIASpy - UI Automation Spy Tool;CLASS:AutoIt v3 GUI;]" ) Then Return

that did it. Had the browser open in the same title  =)

PS: I just now got to read your posting @UEZ.

PS2: ..since it take a while to load:

;~  If WinExists( "UIASpy - UI Automation Spy Tool" ) Then Return
    If WinExists( "[TITLE:UIASpy - UI Automation Spy Tool;CLASS:AutoIt v3 GUI;]" ) Then Return
    If WinExists( "[TITLE:UIASpy - UI Automation Spy Tool;CLASS:AutoIt v3;]" ) Then Return
    SplashTextOn("UIASpy - UI Automation Spy Tool", "..loading..")
    
    ... bunch of code ...
    
    ; Show GUI
    GUISetState( @SW_SHOW, $hGui )
    SplashOff()

would be cool too 

Edited December 1, 2018 by argumentum

[UEZ]

;~  If WinExists( "UIASpy - UI Automation Spy Tool" ) Then Return
    If Not _Singleton(@ScriptName, 1) Then Return

might help, too.

 

Further, the red rectangle flickers.

Edited December 1, 2018 by UEZ

[LarsJ]

So far I've commented out that run-once check. But please do not start UIASpy more than once. It'll be a mess. New zip in first post.

While I've developed this script I've primarily focused on the UI Automation code. That is, I've focused on detecting all UI elements correctly, inserting the elements into the treeview, and calculating and displaying detail information in the listview. And not at least optimized the code for performance and speed. Any issues with that part of the code?

CUIAutomation2.au3 is copied directly from junkew's UI Automation thread. To be able to detect all elements in Windows 10 Start menu two new elements is added:

Global Const $UIA_SemanticZoomControlTypeId=50039
Global Const $UIA_AppBarControlTypeId=50040

That's the reason for the update of the file.

The red element rectangle and startup splash texts has got less attention. With regard to the red element rectangle, it's incredibly hard to draw without a little flicker. It's certainly not enough just to delete the old rectangle once and to draw the new rectangle once. This may easily end with a half-deleted old rectangle and a half-drawn new rectangle.

If eg. the target window is restored from a minimized state, the Windows OS will do a lot of redrawing of the window. This may easily delete the red rectangle. You have to continue updating the rectangle until the redrawing of the window is finished.

I would prefer not to write multiple thousand line code just to draw the red rectangle. And I think the focus on the UI Automation code is the proper focus for a script like this. It's a tool a developer can use while developing UI Automation code.

So far, I'll only promise that I'll look at the problem that the red rectangle is not properly drawn on a high resolution display.

[LarsJ]

High resolution display
I've multiplied left, top, right and bottom coordinates of the element rectangle with the DPI scale factor in DrawElemRect() near bottom of UIASpy_Detect.au3. Since I don't have a high resolution display I've not been able to test the code. But the rectangle is still drawn correct on my standard screen (DPI scale factor = 1).

Danp2 (or other with a high resolution display) will you test the code and check if the red rectangle is drawn correctly?

Replace the existing version of UIASpy_Detect.au3 with this version:

UIASpy_Detect.au3

[LarsJ]

argumentum, are you able to do a test?

[argumentum]

8 hours ago, LarsJ said:

are you able to do a test?

Yes I did. I don't understand much, but it runs. If there is anything in particular you'd want me to try out, let me know.  

PS: I put one of my monitors at 100% and the other at 150%. Your code behaved well. No complains.

PS2: pressing F4 while in the 150% monitor it gave me this:

..so a picture is like 1000 words. ..if this is what you're looking to test out. It does behave as expected at 100%.

Edited December 4, 2018 by argumentum
clarify

[LarsJ]

argumentum, This is exactly the information I'm looking for, thank you very much.

There was one thing I forgot to tell. Calculation of the DPI scale factor is not performed dynamically. It's only performed once when UIASpy starts up. If the scale setting is changed on the display, UIASpy must be restarted to calculate the new DPI scale factor.

I'll try to review the code once more during the day and maybe return with a new test later this evening.

[Danp2]

@LarsJ Just got back online (3 days without HSI ). Tested your revised code, but still appears to be off on the DPI calcs.

 

[LarsJ]

Screen scaling
On Windows 10 a screen scaling different from 100% will likely mean that the red element rectangle is drawn at a wrong position and with a wrong size. The reason for this seems to be that the rectangle returned by $UIA_BoundingRectanglePropertyId is calculated relatively to the logical screen size that matches the scaling factor, and not to the physical size (in pixels) of the screen.

There may be several reasons for a screen scaling different from 100%. One reason may be a high resolution screen where the scaling as default is set to eg. 150%. Another reason may be a normal screen where the scaling manually is set to eg. 125%.

In the first case, a correct drawing of the rectangle may depend on whether or not the target application is DPI aware. In the last case, the rectangle will probably always be mistakenly drawn.

On Windows 7 the element rectangle is still drawn correct even if the scaling is set to eg. 125%. The physical size of the screen seems to be taken into account when $UIA_BoundingRectanglePropertyId is calculated.

Because there are many different situations it's tricky to handle the screen scaling properly. To get around the problem I've decided to add an Options menu where you have to select the scaling factor if it's not 100%.

If $UIA_BoundingRectanglePropertyId is used to eg. set the mouse position to perform a click on the element, the scaling factor must be included to correctly calculate the mouse position.

If the scaling is different from 100% the rectangle values of $UIA_BoundingRectanglePropertyId are simply multiplied with the scaling factor.

In this situation both the original rectangle as returned by $UIA_BoundingRectanglePropertyId and the scaled rectangle are displayed in the right pane listview.

New zip-file in bottom of first post.

[mLipok]

@LarsJ Your tools just help me to solve one problem.

Now I know what was wrong.

Thanks for this great tool.
 

I check many things with this tool and I have some questions or even feature request,

Here is a screenshot which will help me to describe what I mean.

 

These rose fields are of course cut out for security reasons and GDPR rules.

My problem was that I was checking for WinExist('****','Weryfikacja danych') but I do not know that there are 3 different windows that match this "pattern", as they were obscured by other "INNER WINDOW/PANEL" (btw. how they should be properly called ?) placed inside the same application (inside MAIN WINDOW).

To understand what I mean I change each INNER window size and position to show what I mean in this following screenshot.

 

QUESTION 1:
How I can save entire structure to file ? (when I try ten I get only info about current selected element , but not all sub elements).

QUESTION 2:
Do you think an option to save a selected Treview node, with all sub nodes, to an XML file would be  usefull ?
I can take this subtask as a my part of contribution to your project, if you would accept this offer.

Because I need a tool to compare what change is made in some "UIA"  structures when I change some feautres on CRM WINDOW (which use my Clients), and find out a way to automate some "events" but firstly I must to know which element is changing by user interaction.

QUESTION 3:
Would you move this project to GitHub ?

QUESTION 4:
When I use 

as a effect, right panel change  his own content - like this:

is it intentional ?

QUESTION 5:
Do you think that this project could be moved here ? maybe in a near future.

https://www.autoitscript.com/forum/forum/49-autoit-projects-and-collaboration/

QUESTION 6:

Is there a way to use this Tool to search for a specyfic word in this entire structure ?
"Find" and "Find Next" feature would be great.

Regards,

mLipok

 

EDIT:
QUESTION 7:
I was trying to get UIA structure from some menu....

I found that this is imposible because , when I open menu in my CRM DOCUMENT Editor then I can't use "Update all chlids of elements", as using this options causes the menu to be closed in the CRM Editor.

So I thought about new feature:
Could you add an additional option "Update all chlids of elements after X seconds" where X should be each time indicated by user ?

I was even do some experiment ....

Case $idUpdateAllChildsOfElement ; Update all childs of element
                                Do
                                    $iUpdateWaitBefor = Number(InputBox('Wait time','Provide amount of seconds to wait before update',0))
                                Until IsInt($iUpdateWaitBefor)
                                Sleep($iUpdateWaitBefor*1000)

                                $idTVExpand = 0 ; Disable $idTVExpand
                                _GUICtrlTreeView_BeginUpdate( $hTV )
                                UIASpy_DelChilds( $iIdx )
                                $iCntElems = 0
                                UIASpy_AddChilds( $iIdx, True )
                                DllCall( "user32.dll", "lresult", "SendMessageW", "hwnd", $hTV, "uint", $TVM_ENSUREVISIBLE, "wparam", 0, "handle", $aElems[$iIdx][4] )
                                _GUICtrlTreeView_EndUpdate( $hTV )
                                $idTVExpand = $idTVExpand0
                                ToolTip("")
                                If $iUpdateWaitBefor > 0 Then MsgBox(0, 'Info', 'Structure is updated')

but this is not working as I intent - I can't get SubItem structure for this Menu - the same problem is with getting NOTEPAD "File" menu structure.

 

Edited December 14, 2018 by mLipok

[LarsJ]

Question 1
"Copy structure to clipboard" does not calculate the element subtree. The procedure to copy the entire subtree structure to clipboard is (right click the treeview item):
1. Update all childs of element
2. Copy structure to clipboard

Question 2
I rarely use the XML format. Usually you can do the same with a plain text file, and with much faster code. But if you'll write the code and it's not too slow, I'll add it to the zip-file in first post.

Question 3
GitHub. No.

Question 4
Show structure in listview. This is intentional.
1. It takes some time to calculate the structure of a subtree with 1000 elements or more. The calculation is complete when the structure appears in the listview.
2. Perhaps you only need a single or a few rows in the structure. You can select these rows in the listview and use "Copy all or selected items to clipboard" in the "Right pane" main menu to copy only those few rows.

Question 6
A Search feature is planned for the next version.

Question 7
Use the function keys to handle a menu. Here's an example with Notepad File menu:
Run UIASpy.au3. Open Notepad. Click File menu. Place mouse cursor over first menu item. Press F1. Press F2. Switch to UIASpy. It should look like this:

Left: F1,  Middle: F2,  Right: Treeview (red items because the corresponding UI elements doesn't exist, the menu is closed)

Question 5
Move example to "Projects and collaboration" forum.
I think such an example is too small for "Projects and collaboration". But a more general topic, eg. "UI Automation Updates" might be interesting to start up under "Projects and collaboration". This can include a discussion of a spy tool.

First of all, an update of the code in CUIAutomation2.au3 (constants and definitions) is required. CUIAutomation2.au3 contains Windows 7 code. To support Windows 8 and Windows 10 probably 50 - 100 new interfaces, methods, properties and constants are added which are not included in CUIAutomation2.au3.

An UI Automation thread in "Projects and collaboration" can certainly be interesting.

[mLipok]

@LarsJ Thanks for all your answers.
I know this is not thread for teaching how to use UIA at all.

..but it would be helpfull to show one example how to use UIASpy for create simple notepad automation.

So here is what I propose.

In your previous post you answer to my Q7 and as an example you post this following screenshot

 

Question/Request:

Could you create few next screenshot which will shows how to get structur for "Edit1" , "FileName" and "SaveButton"

Then would you create an AU3 example to show how to use this information to create a simple script to fill up "Edit1" with HelloWorld, click "Save As..." enter filename and finally save it to disc.

It could then be considered a complementary example of how to use UIASpy to create a simple notepad automation.

Regards,

mLipok

 

Edited December 16, 2018 by mLipok

[junkew]

Regarding cuiautomation2.au3 are we missing functionality that is in urgent need. So far i only needed a very small subset but willing to enhance if there is urgent need.

@mLipok in the uia thread notepad and calc examples are available.

Edited December 17, 2018 by junkew

[mLipok]

Are you refering to my Request ?

EDIT... or to one of my last questions  which I ask recently in this topic ?
or just generally ...

Edited December 17, 2018 by mLipok

[mLipok]

25 minutes ago, junkew said:

 

@mLipok in the uia thread notepad and calc examples are available.

As I stated before.

Quote

I know this is not thread for teaching how to use UIA at all.

My concept was to show how to use UIASpy in order to create "working" Notepad automation, to show corelation with using UIASpy tool and final simple code.