UIASpy - UI Automation Spy Tool

[LarsJ]

Sample code creation through Detail info listview page
The Detail info listview page is also the default listview page. This makes sample code creation easy and fast. But not all code can be created this way. In many cases, there is a need to use the Sample code main menu.

 

Sample code is created by selecting one or more rows in a single listview group. Right-click a selected row and click Create sample code. The code is displayed in the listview code page.

Click Create sample code in the picture above to generate this code:

; --- Find window/control ---

ConsoleWrite( "--- Find window/control ---" & @CRLF )

Local $pCondition0
$oUIAutomation.CreatePropertyCondition( $UIA_ClassNamePropertyId, "Notepad", $pCondition0 )
If Not $pCondition0 Then Return ConsoleWrite( "$pCondition0 ERR" & @CRLF )
ConsoleWrite( "$pCondition0 OK" & @CRLF )

Local $pWindow, $oWindow
$oDesktop.FindFirst( $TreeScope_Descendants, $pCondition0, $pWindow )
$oWindow = ObjCreateInterface( $pWindow, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
If Not IsObj( $oWindow ) Then Return ConsoleWrite( "$oWindow ERR" & @CRLF )
ConsoleWrite( "$oWindow OK" & @CRLF )

Click the selected treeview item in the left pane window to return to the detail information and add more sample code.

Or paste the code into the code editor. Create sample code also copies the code to clipboard.

 

Only selected rows that specifically belongs to the listview group in which a row is right-clicked are included in sample code.

The only exception to this are the groups "Element Properties (identification)" and "Element Properties (session unique)". For these two groups, selected rows from both groups are included in sample code. This makes it possible to use session unique properties in window identification:

Click Create sample code in the picture to generate this code:

; --- Find window/control ---

ConsoleWrite( "--- Find window/control ---" & @CRLF )

Local $pCondition0, $pCondition1, $pAndCondition1
$oUIAutomation.CreatePropertyCondition( $UIA_ClassNamePropertyId, "Notepad", $pCondition0 )
$oUIAutomation.CreatePropertyCondition( $UIA_NativeWindowHandlePropertyId, 0x0000000000050314, $pCondition1 )
$oUIAutomation.CreateAndCondition( $pCondition0, $pCondition1, $pAndCondition1 )
If Not $pAndCondition1 Then Return ConsoleWrite( "$pAndCondition1 ERR" & @CRLF )
ConsoleWrite( "$pAndCondition1 OK" & @CRLF )

Local $pWindow, $oWindow
$oDesktop.FindFirst( $TreeScope_Descendants, $pAndCondition1, $pWindow )
$oWindow = ObjCreateInterface( $pWindow, $sIID_IUIAutomationElement, $dtagIUIAutomationElement )
If Not IsObj( $oWindow ) Then Return ConsoleWrite( "$oWindow ERR" & @CRLF )
ConsoleWrite( "$oWindow OK" & @CRLF )

 

See Automating Notepad with sample code - step by step and Automating Notepad with sample code - all at once for examples of creating sample code.

Edited February 23, 2019 by LarsJ
Update

[LarsJ]

Windows XP
You can use UI Automation code on a fully updated Windows XP SP3.

I've tested UIASpy under Windows XP through Windows XP Mode in Windows Virtual PC for Windows 7.

Apart from some relatively small differences in treeview functionality from Windows XP to later Windows versions, UIASpy works flawlessly under Windows XP.

You can find an example of using UIASpy under Windows XP in Automating Notepad - Windows XP.

Edited February 9, 2020 by LarsJ
Text update

[LarsJ]

Automation issues

• The Shell issue in Windows XP and Windows 7 is canceled (2019-03-09).
  The Shell issue is caused by calculating $UIA_LegacyIAccessibleSelectionPropertyId for some elements (Control Host and Namespace Tree Control elements) in the Shell programs. The problem is avoided by omitting the calculation of this property.

  This is the original description of the issue:
  In Windows XP and Windows 7, there is a problem with Windows Shell programs, such as Windows Explorer. When attempting to identify UI elements in these programs with F1-F4, the Shell programs may at worst crash.
  A workaround is to use the UIASpy treeview to inspect UI elements instead of F1-F4.
  This Shell issue appears only in Windows XP and Windows 7. Not in Windows 10.

• On Windows 7 UI elements are not identified in web content in Mozilla Firefox. Only outer elements of Firefox are identified. When attempting to identify UI elements in web content with F1-F4, Firefox may crash.
  On Windows 10 UI elements in web content are identified properly.

Edited March 9, 2019 by LarsJ
The Shell issue is canceled

[LarsJ]

Listview features

New listview features have been added. These are new listview pages and help pages, double click and right click on listview rows and sample code creation. The latter is reviewed in Sample code creation through Detail info listview page.

 

Listview pages
In the previous version there were only two listview pages: The main listview page to display detail information for the UI element corresponding to the selected treeview item. And a listview page to display the treeview structure (UI element tree structure) of child items below the selected item.

In the new version, there are more listview pages that are used to display and create sample code and to display help pages.

Detail information
UI element detail information is divided into more groups than in the previous version.

It's easier to see which Control Pattern Properties belongs to a specific Control Pattern (element action), because the properties are divided into matching groups by alternating background colors as shown in the picture:

Note that some of the magic numbers have been replaced by or have been expanded with UI Automation constants. See eg. $UIA_LegacyIAccessibleRolePropertyId and $UIA_LegacyIAccessibleStatePropertyId lines in the picture.

 

An entire new group has been added: Control Pattern Methods.

Note the SetValue() method of the Value Pattern object (near bottom of picture) that's used to set the text in an Edit control.

Also note that not all methods are included in the list. Only methods that are relevant to actual client automation are included. A large number of methods used in the caching technique are omitted. Advanced and rarely used methods are also omitted.

The colored lines (cyan and gray) indicates that double-click opens Microsoft documentation.

 

Listview Help pages
The Help menu items opens AutoIt web pages with information and examples, opens listview Help pages and opens Microsoft docu pages.

In this version there are four listview Help pages:

• UI Automation Controls (elements) - Lists all registered UI Automation control types
• UI Automation Patterns (actions)  - Lists all registered UI Automation patterns (actions)
• UI Automation object (main obj)   - Lists the most relevant methods for the UI Automation main object
• UI Automation Element object      - Lists the most relevant methods for the UI Automation Element object

The colored lines (cyan and gray) on the listview pages indicates that double-click opens Microsoft documentation.

Note that the Microsoft documentation for the two first listview Help pages ("UI Automation Controls (elements)" and "UI Automation Patterns (actions)") is provider documentation (unlike client documentation). The provider documentation contains some valuable overview pages that are not found in the client documentation. On the other hand, they also contain some information that's not relevant to client automation.

Microsoft provider documentation for "UI Automation Controls (elements)":

• Control Description
• Typical Tree Structure
• Relevant Properties
• Control Patterns

For the two last listview Help pages ("UI Automation object (main obj)" and "UI Automation Element object") only methods that are relevant to actual client automation are included. A large number of methods used in the caching technique are omitted. Advanced and rarely used methods are also omitted.

Click the selected treeview item in the left pane window to return to the detail information listview page.

 

Listview double click
Most listview pages supports both double and right clicks. Double clicks are used to open the matching Microsoft documentation.

All listview rows that can be double-clicked are marked with a cyan, red, gray or yellow background color. However, double-clicks are not implemented for all colored rows. An exception are the gray colored rows used to clarify "Control Pattern Property" groups as shown in the top picture above.

Note that a double-click on the "Treeview Element" row (first row) at the element detail information listview page opens Microsoft provider documentation (see above) for the current element.

 

Listview right click
All listview pages and nearly all listview rows (except completely empty rows) supports right clicks. Right clicks are used to open a context menu. The context menu may vary depending on the listview page and listview row.

Edited February 22, 2019 by LarsJ
Update

[LarsJ]

Sample code creation through Sample code main menu

 

The Sample code main menu is used for Sample code creation. Much more code can be created through the menu than through the Detail info listview page. In addition, the menu can be used to set Sample code options and for Sample code maintenance.

Menu items with a 3-point ending opens an intermediate listview page. These pages works in the same way as the Detail info listview page. Select one or more rows and right-click to create Sample code. Unlike the Detail info page, rows can be selected in several sections at once.

The upper part of the menu through Sleep(1000) creates common Sample code. Code snippets... adds functions and code sections to solve tasks of a more specific nature. If the Check error setting is off, the Check error menu can be used to add error checking as needed. Corrections adds a list of necessary and optional corrections to the code. In the Options menu, settings can be turned on/off. The lower part of the menu is about Sample code maintenance.

The Help system includes a review of the menu items.

Edited February 23, 2019 by LarsJ
Update

[LarsJ]

Updates today are about the Sample code menu, Sample code creation and the Help menu. Sample code creation includes code creation through the Detail info listview page and the Sample code main menu.

All planned functionality in relation to the listview, Sample code creation and the Help system is hereby implemented.

With completion of the Help system, a lot of information has been moved from first post and other posts into the Help system.

There may be another few updates on element detection and treeview features.

And more examples will be added in Using UI Automation Code in AutoIt, especially on advanced topics.

New zip-file in bottom of first post. Also new zip-file in Using UI Automation Code in AutoIt.

You should download the new zip-file. The updated code is much better than the old code.

[LarsJ]

F7 and F8 can be used to navigate up and down along parent elements.

The Shell issue in Windows XP and Windows 7 has been identified and bypassed.

Copy to sample code item added to listview right-click menu.

Yellow listview cells indicates that an example exists.

Topic list at the top of Use of UIASpy help pages.

A few new How to topics are created.

Patterns (actions) demo examples.

New zip-file in bottom of first post.

Edited March 10, 2019 by LarsJ
F7 and F8

[LarsJ]

The most important update today is the introduction of UIA_SafeArray and UIA_Variant UDFs to be able to handle pattern (action) methods using safearrays as either input or output parameters.

The GetCurrentSupportedViews() method of the MultipleView pattern object returns supported views as a safearray of integers.

New pattern (action) examples shows how to create UI Automation element arrays and how to traverse these arrays.

Other updates:

• Corrected a few of the usual minor issues I find when doing examples. But this time also a serious error in connection with the Undo/Redo functionality in the Sample code menu.
• Added cross-references between a few of the Use of UIASpy help pages (the help pages found at the bottom of the main menus and context menus). The cross-references are marked in gray. Double-click to go to the reference documentation.
• Updated listviews with more yellow cells in second column, indicating that there is an example.

New zip-file at bottom of first post.

[LarsJ]

Updates:

• New Sample code menu item to create objects and methods that have not been created otherwise. It's a couple of help pages that have been moved to the Sample code menu.
• Better naming of method parameters. Eg. $oGridPattern.GetItem($iRow,$iColumn,$pUIElement*) instead of $oGridPattern.GetItem(int,int,ptr*). Not finished.
• The usual minor errors.

This completes the implementation of all planned UIASpy features. There may well be updates of existing features and new examples and code snippets. But all main features are implemented.

New zip-file at bottom of first post.

[AtomicSeek]

On 12/9/2018 at 4:44 PM, LarsJ said:

Screen scaling
On Windows 10 a screen scaling different from 100% will likely mean that the red element rectangle is drawn at a wrong position and with a wrong size. The reason for this seems to be that the rectangle returned by $UIA_BoundingRectanglePropertyId is calculated relatively to the logical screen size that matches the scaling factor, and not to the physical size (in pixels) of the screen.

There may be several reasons for a screen scaling different from 100%. One reason may be a high resolution screen where the scaling as default is set to eg. 150%. Another reason may be a normal screen where the scaling manually is set to eg. 125%.

In the first case, a correct drawing of the rectangle may depend on whether or not the target application is DPI aware. In the last case, the rectangle will probably always be mistakenly drawn.

On Windows 7 the element rectangle is still drawn correct even if the scaling is set to eg. 125%. The physical size of the screen seems to be taken into account when $UIA_BoundingRectanglePropertyId is calculated.

Because there are many different situations it's tricky to handle the screen scaling properly. To get around the problem I've decided to add an Options menu where you have to select the scaling factor if it's not 100%.

If $UIA_BoundingRectanglePropertyId is used to eg. set the mouse position to perform a click on the element, the scaling factor must be included to correctly calculate the mouse position.

If the scaling is different from 100% the rectangle values of $UIA_BoundingRectanglePropertyId are simply multiplied with the scaling factor.

In this situation both the original rectangle as returned by $UIA_BoundingRectanglePropertyId and the scaled rectangle are displayed in the right pane listview.

New zip-file in bottom of first post.

Dear LarsJ,

For me there still seems to be a problem related to screen scaling.
I'm using the latest version of UIASpy (from March 23), and my screen scaling is set to 125% (because at 100% everything is too small).

After selecting the same scaling in UIASpy:
1. If I'm pressing F1-F4 to detect the element, then the correct (scaled) red rectangle is shown but only for a fraction of a second, and afterwards the wrong (not scaled) red rectangle is shown and remain visible.
2. If I'm selecting the element in UIASpy by clicking it in the left pane, then only the wrong (not scaled) red rectangle is shown and remain visible.

I tested also with other scalings and I have the same problem.
At 100% the red rectangle is shown correctly.

Does this happens also to anyone else?

Thank you,
AtomicSeek

 

[LarsJ]

AtomicSeek, I'm aware of the problem, thank you. There will be a new update tomorrow where the problem should be corrected. Let me know if it doesn't work. It was a stupid mistake. I think I've calculated the scaling correctly. I just forgot to correct the numbers in most red rectangles.

[LarsJ]

Windows 8, Windows 8.1 and Windows 10 updates

This is an update of the Windows 7 constants and definitions in CUIAutomation2.au3 which includes new constants and definitions added in Windows 8, Windows 8.1 and Windows 10 up to and including version 1809.

All constants and definitions (both old and new) are found in UIA_Constants.au3. There is 50% more data. New CLSID_CUIAutomation8 instead of CLSID_CUIAutomation. Many new versions of the main objects $oUIAutomation and $oUIAutomationElement (abbreviated as $oUIElement in UIASpy).

CUIAutomation2.au3 is still included. It's necessary to execute the existing examples.

There are a few changes in UIA_Constants.au3 compared to CUIAutomation2.au3. Among other things is "dtag" changed to "dtag_".

All properties are calculated in UIASpy_ElemInfo.au3. Not all old properties were calculated in the previous version of the file. In the new version, all both new and old properties are calculated. The file has become quite large.

In the updated UIASpy code in the zip-file at bottom of first post, UIASpy always starts up in Windows 7 mode regardless of the Windows OS version. Even if your Windows OS version is Windows 10, UIASpy starts up in Windows 7 mode. Ie. the code generated as Sample code through the Detail info listview page and the Sample code menu can be executed under Windows XP, Windows Vista, Windows 7 and later versions. However, in Windows 7 mode, updated UIA code in newer Windows versions isn't available.

Options | Windows | Mode menu
If your Windows OS version is Windows 8 or later you can through the Options | Windows | Mode menu choose to generate UIA code that is compatible with a specific Windows OS version and thus access updated UIA code in this Windows version. But the code is only executable on this specific Windows version and later versions.

On Windows 8 there is access to new properties, patterns, objects and methods added in Windows 8 as well as all the previous ones from Windows 7. CLSID_CUIAutomation8 is available from Windows 8. New main objects are $oUIAutomation2 and $oUIElement2.

On Windows 8.1 there is access to new properties, patterns, objects and methods added in Windows 8.1 as well as all the previous ones from Windows 8 and so on. New main objects in Windows 8.1 are $oUIAutomation3 and $oUIElement3.

A single new main object in first version of Windows 10 (1507) is $oUIElement4. In Windows 10 First mode, the generated code can be executed on all Windows 10 versions.

New main objects in last version of Windows 10 (1809) are $oUIAutomation4 (1607), $oUIAutomation5 (1607), $oUIAutomation6 (1809), and $oUIElement5 (1703), $oUIElement6 (1703), $oUIElement7 (1703), $oUIElement8 (1803), $oUIElement9 (1809). In Windows 10 Last mode, the generated code can be executed on Windows 10 version 1809 and later.

Note that $oUIAutomation6 (1809) gives access to all new methods and properties in $oUIAutomation6 and all previous methods and properties in $oUIAutomation5, $oUIAutomation4, $oUIAutomation3, $oUIAutomation2 and $oUIAutomation. See UIA_Constants.au3.

And $oUIElement9 (1809) gives access to all new methods and properties in $oUIElement9 and all previous methods and properties in $oUIElement8 - $oUIElement. See UIA_Constants.au3.

In UIASpy, all objects and methods are accessed through the Sample code | Objects/methods menu. Note that this menu does not differentiate between Windows versions. All objects and methods are always available. Of course, if you try to use a Windows 10 object on Windows 7, creating the object will fail. But not the other way around.

Also note that there are no new methods added to UIASpy for $oUIAutomation3, 4, 5 or 6 objects. These methods are merely event handler methods not used in conventional automation, which is what UIASpy is all about.

When you select Windows mode through the Options | Windows | Mode menu, UIASpy will generate correct Sample code for this Windows version and use the correct CLSID_CUIAutomation version and the correct versions of the main objects. See UIA updates.

UIASpy.ini
In UIASpy.ini, through the WindowsMode key, you can specify the Windows mode you want when UIASpy starts up. Use these values: Windows 7 (default), Windows 8, Windows 8.1, Windows 10 First (1507) and Windows 10 Last (1809).

Edited November 21, 2019 by LarsJ
UIA updates to Windows 10 1809

[Exit]

Thanks for the new version 😊 

Just a small proposal: Do not distribute as "7z" file, bat as a real "zip" file. Would be easier to handle with chrome browser.

Thanks

Exit

[SEKOMD]

1. Clean Windows 10 1607

2. Clean AutoIt

result from run UIASpy.au3 - error

please help me)

 

[LarsJ]

I've assumed that all Windows 10 PCs are at least updated to feature update 1803. So I've assumed that a feature update has been postponed for a maximum of one year. I think it's a reasonable assumption. On your Windows 10 1607, you're getting errors on objects that require a higher windows 10 update.

You can use the version here, which is the latest version before the Windows 8, 8.1 and 10 update:

The zip-file has been moved to bottom of first post.

Edited May 25, 2019 by LarsJ
Zip-file moved to bottom of first post

[SEKOMD]

7 minutes ago, LarsJ said:

I've assumed that all Windows 10 PCs are at least updated to feature update 1803. So I've assumed that a feature update has been postponed for a maximum of one year. I think it's a reasonable assumption. On your Windows 10 1607, you're getting errors on objects that require a higher windows 10 update.

You can use the version here, which is the latest version before the Windows 8, 8.1 and 10 update: UIASpy.7z

I use Windows 10 1607 LTSB (thin, lightweight, stable).

Dear LarsJ, many thanks for the promptness!

[LarsJ]

You're welcome.

[SEKOMD]

Please show an example of extracting text from the current forum response. Please..

I suffer all night long ...((

 

 

[LarsJ]

If UI Automation or the UIASpy tool is new to you, then you should read the How to topics.

[junkew]

Check out examples.zip from uia examples thread as that has examples on how to navigate and read html pages.