UIASpy - UI Automation Spy Tool

[Exit]

At @LarsJ:      You're great! Thank you so much.

Now I have 2 1 questions:

13 hours ago, LarsJ said:

The problem is that there are two visitor number controls that you can see when printing the entire tree structure:

How to print the entire tree structure?

Just found the answer here myself. The search button is your friend. 🙂

13 hours ago, LarsJ said:

And you need the second control to read the text

How could I detect that there are two controls and that I should use the second one without looking at the entire tree structure first? 
How did you find out?

Edited May 14, 2019 by Exit
I answered the first question myself.

[Exit]

Now I've worked a bit in UIASpy and I'm really excited.
@LarsJ made a great tool. 

Now I have a few comments on the handling of the function keys.
The keys should not be handled as hotkeys but as accelerator keys.
At least if you use common keys like F1-F7.
These keys are used in many applications.
For example, the key F1, which is used in SciTe to call syntax help.
In order to use them, you always have to switch off the 'detect element' function in the UIASpy menu.

Now I have found the solution for me:
Just use the function keys in shifted mode.
Simply add a plus sign (+) at the beginning of the file 'UIASpy_Menu.au3' for the hotkey definitions.
So the keys F1-F7 are again available for other applications.

Global $aFuncKeys = [ [ "+{F1}", "UIASpy_F1" ], _ ; F-key, Function
                      [ "+{F2}", "UIASpy_F2" ], _
                      [ "+{F3}", "UIASpy_F3" ], _
                      [ "+{F4}", "UIASpy_F4" ], _
                      [ "+{F7}", "UIASpy_F7" ], _
                      [ "+{F8}", "UIASpy_F8" ] ]

If someone wants to copy it, just should do it. 

Thanks again to @LarsJ 

[philkryder]

got an error when trying to run the UIASpy.au3

I just downloaded it today.

line 3787 looks like this:
Local $aElemIndexes = ($aElems[$iIdx][7])[1], $aIndex, $jCnt = 0
 

here is from the console:

>"C:\Program Files (x86)\AutoIt3\SciTE\..\AutoIt3.exe" "C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.au3" /run /prod /ErrorStdOut /in "C:\#\__AutoIT\!! try IUI automation - may work on controls without names\UIAExamples\UIASpy.au3" /UserParams    
+>14:03:15 Starting AutoIt3Wrapper v.18.708.1148.0 SciTE v.4.1.0.0   Keyboard:00010409  OS:WIN_10/  CPU:X64 OS:X64  Environment(Language:0409)  CodePage:0  utf8.auto.check:4
+>         SciTEDir => C:\Program Files (x86)\AutoIt3\SciTE   UserDir => C:\Users\pkryder\AppData\Local\AutoIt v3\SciTE\AutoIt3Wrapper   SCITE_USERHOME => C:\Users\pkryder\AppData\Local\AutoIt v3\SciTE 
>Running AU3Check (3.3.14.5)  from:C:\Program Files (x86)\AutoIt3  input:C:\#\__AutoIT\!! try IUI automation - may work on controls without names\UIAExamples\UIASpy.au3
+>14:03:16 AU3Check ended.rc:0
>Running:(3.3.14.5):C:\Program Files (x86)\AutoIt3\autoit3.exe "C:\#\__AutoIT\!! try IUI automation - may work on controls without names\UIAExamples\UIASpy.au3"    
--> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop
"C:\#\__AutoIT\!! try IUI automation - may work on controls without names\UIAExamples\Includes\UIASpy_ElemInfo.au3" (3787) : ==> Subscript used on non-accessible variable.:
Local $aElemIndexes = ($aElems[$iIdx][7])[1], $aIndex, $jCnt = 0
Local $aElemIndexes = ($aElems[$iIdx][7])^ ERROR
->14:03:17 AutoIt3.exe ended.rc:1
+>14:03:17 AutoIt3Wrapper Finished.
>Exit code: 1    Time: 2.566
 

[LarsJ]

You can download a version of UIASpy that works on all Windows 10 versions in this post.

[cyxstudio]

i just went through the codes and i realise that you did not use third party control functions like controlgettext(). 

May i know how do you get properties of child control?

 

I tried it and it was able to detect controls that autoit info is not able to such as power builder data window.

[junkew]

See faq31 that leads to iuiautomation thread in examples section many examples are given to retrieve properties

[supersonic]

LarsJ, another great tool of yours!

Here in our company we are using a terminal program called "Mocha TN5250 for Windows 7/8/10" from https://mochasoft.dk/ . Currently I retrieve terminal text using ControlSend() - which works 95%. It is somehow "sensible"; e. g. it won't work if - for whatever reason - window lost focus.

I'm not able grab any information with AutoIt Window Info Tool and even your tool can't see this control/part of the GUI.

Do you have ever heard about "invisible" controls? And do you have a clue how to address such controls?

Edited October 22, 2019 by supersonic

[junkew]

If you are in control of the installation check if the activex control of mocha is there then you could use objcreate function. If not search for ehllapi terminal dll. If you have no development background easiest will be screenscraping with controlsend and winactivate commands. Its hard to help with terminal emulators as there are many around. 

https://www.ibm.com/support/knowledgecenter/en/SSEQ5Y_6.0.0/com.ibm.pcomm.doc/books/html/emulator_programming07.htm

 

[supersonic]

junkew, thank you.

Yes, I am in control of the installation/system but Mocha ActiveX is not (and will not be) available, unfortunately. The ActiveX control is also not able to grab text from running sessions initiated by the standard Mocha client. And as far as I understand EHLLAPI it also isn't able to grab session text but I can build my own terminal application... Well, that's somehow beyond the goal 😊

So, I will stick with Win*() and Control*() functions - that's good enough for now...

But... If anyone else have a clue why there could be a hidden/invisible control and how to grab text from it... I would be very pleased...
 

Edited October 23, 2019 by supersonic

[LarsJ]

I think these terminals/consoles are simply windows and not controls. The text in the windows is written through the window's device context (hdc) with GDI functions like DrawText. That's why they can't be automated. There are a few posts on the same topic here.

[junkew]

Building your own terminal emulator is not that hard just long strings send back and forth to a mainframe with some bytes representing colors or fields so it depends on your goals.

Hooking drawtext is an option but hard to interpret and control.

I have never seen a terminal emulator that has no option to grab to text on clipboard. Whats the macro language of mocha offering?

I remember some open source terminal emulator is around on github but do not remember details.

[junkew]

https://www.codeproject.com/Articles/9615/Using-EHLLAPI-in-C

[supersonic]

@LarsJ - Sorry for bothering you again: Maybe it is a GDI window (I don't know how to evaluate that) but it is funny that rows/cols can be selected by mouse (or all text by Ctrl+A). Is this also typical for GDI windows?

@junkew - Thank you for motivating me to code my own terminal emulation. 😊 Here in our company Mocha terminal _must_ be used and unfortunately there is no way around it. I can live with that. Of course, Mocha can grab text (by keyboard, mouse and menue, macro [= disabled for all users]). My aim was to automate text grabbing (and any further processing) by a litte AutoIt script (that actually fully works for a couple of month now). All I wish is to make text grabbing more reliable.
 

[LarsJ]

You can easily manipulate and edit an image and extract text from an image. Rows and columns in an image are just a certain number of pixels in the horizontal and vertical directions.
 
With classic automation code you can automate this set of controls and with UI automation code you can automate this set of controls. There are no such controls as console or terminal controls.

[supersonic]

@LarsJ Thank you - Good to know. I'll try to keep that in mind...

Edited October 24, 2019 by supersonic

[junkew]

https://www.mochasoft.dk/tn5250menus.htm

Shows you can copy paste as text or csv

1. WinActivate the mocha window main window

2. Click xy counted from xy of the window to be in the terminal area

3. Send home to be at terminal position 0 0

4. Send shift arrow down 24 times and shift arrow right 80 or thru menu send select all or just ctrl a

5. Send copy either thru menu or with shortcut key

6. Clipboard functions AutoIt should be able to get it from the keyboard.

Doing 1 and 2 should be 100% reliable to make sure you are in terminal area.

Just post simplespy output based on that we can help you probably a little further at least there is a mainwindow child of desktop.

[junkew]

Seems to be a full GDI black area with drawtext happening.

Control is not detectable so just calculate position based on position info of main info and show that highlighted area and activate it with a mouseclick

Example (long time not played with an as400 or other IBM mainframe  ;-))

1. Download and install mocha tn5250
2. Connect to www.pub400.com (either with or without userid, logonscreen will come allways)
3. Run below script (after you have done file, edit session, connect,) I did not automate the menus as thats straight forward with alt and letter keys

• Some calculation has to be done based on the font size you have installed
• Mocha is a limited terminal emulator, no way to easy find out the blinking cursor position and no x, y in toolbar

Functions are generic

• setPosTN5250 with x, y will calculate position and with mouseclick cursor will be positioned
• getTN5250Text will just return the text in the black area
• getTN5250hwnd will determine hwnd of first windows10.* with text toolstrip1, .... 
• selectBlockTN5250 will select a rectangle of text (will not copy it thats just another send ctrl c)
• _UIA_DrawRect draws a rectangle 😂

Script

 

expandcollapse popup

#include <WinAPI.au3>
#include <WindowsConstants.au3>
#include <Array.au3>
;~ test with mocha tn5250
;~ www.pub400.com

global $g_baseLeft
global $g_baseRight
global $g_baseTop
global $g_baseBottom
global $g_TN5250Window

Example()

func example()
    $g_TN5250Window=getTN5250hwnd()
    WinActivate($g_TN5250Window,"")
    Local $aPos = WinGetPos($g_TN5250Window)
;~   $aPos[0] = X position
;~   $aPos[1] = Y position
;~   $aPos[2] = Width
;~   $aPos[3] = Height
;~ Left,right,top,bottom
    _UIA_DrawRect($aPos[0],$aPos[0]+$aPos[2],$aPos[1],$aPos[1]+$aPos[3])

;~ So some calculations for the black area
    $g_baseLeft=$aPos[0]+8
    $g_baseRight=$aPos[0]+$aPos[2]-24
    $g_baseTop=$aPos[1]+78
    $g_baseBottom=$aPos[1]+$aPos[3]-40
    _UIA_DrawRect($g_baseLeft,$g_baseRight,$g_baseTop,$g_baseBottom)

;~ Lets positon mouse over the first row
    for $iCol=1 to 80
        setPosTN5250($iCol,0)
    Next

;~ Lets positon mouse over the first coloumn
    for $iRow=1 to 32
        setPosTN5250(0,$iRow)
    Next

    selectBlockTN5250(10,10,15,15)
    sleep(2000)

    $screenAsText=getTN5250Text()
    $screenLines=stringsplit($screenAsText, @CRLF,$STR_ENTIRESPLIT)
    _ArrayDisplay($screenlines)

EndFunc

func setPosTN5250($x,$y)
    mouseclick("left",$g_baseLeft + 4 + ($x*9) ,$g_basetop+8 + ($y*13),1,0)
EndFunc

func selectBlockTN5250($x1,$y1,$x2,$y2)
    mousemove($g_baseLeft + 4 + ($x1*9) ,$g_basetop+8 + ($y1*13),0)
    sleep(250)
    mousedown("left")
    sleep(250)
    mousemove($g_baseLeft + 4 + ($x2*9) ,$g_basetop+8 + ($y2*13),0)
    sleep(250)
    mouseup("left")
EndFunc


func getTN5250Text()
    setPosTN5250(0,0)
    sleep(500)
    send("^a")
    sleep(1000)
    send("^c")
    sleep(500)
    $text=clipget()
    return $text
EndFunc

func getTN5250hwnd()
    Local $aWinList = WinList("[REGEXPCLASS:WindowsForms10.Window.8.*]")
;~     _ArrayDisplay($aWinList)
    ; Loop through the array displaying only visable windows with a title.
    For $i = 1 To $aWinList[0][0]
        If $aWinList[$i][0] <> "" And BitAND(WinGetState($aWinList[$i][1]), 2) Then
            $hwnd=$aWinList[$i][1]
            if (stringinstr(wingettext($hwnd), "toolStrip1")  > 0) Then return $hwnd
;~             MsgBox($MB_SYSTEMMODAL, "", "Title: " & $aWinList[$i][0] & @CRLF & "Handle: " & $aWinList[$i][1])
        EndIf
    Next
;~      _ArrayDisplay($aWinList)

EndFunc

; Draw rectangle on screen.
Func _UIA_DrawRect($tLeft, $tRight, $tTop, $tBottom, $color = 0xFF, $PenWidth = 4)
    Local $hDC, $hPen, $obj_orig, $x1, $x2, $y1, $y2
    $x1 = $tLeft
    $x2 = $tRight
    $y1 = $tTop
    $y2 = $tBottom
    $hDC = _WinAPI_GetWindowDC(0) ; DC of entire screen (desktop)
    $hPen = _WinAPI_CreatePen($PS_SOLID, $PenWidth, $color)
    $obj_orig = _WinAPI_SelectObject($hDC, $hPen)

    _WinAPI_DrawLine($hDC, $x1, $y1, $x2, $y1) ; horizontal to right
    _WinAPI_DrawLine($hDC, $x2, $y1, $x2, $y2) ; vertical down on right
    _WinAPI_DrawLine($hDC, $x2, $y2, $x1, $y2) ; horizontal to left right
    _WinAPI_DrawLine($hDC, $x1, $y2, $x1, $y1) ; vertical up on left

    ; clear resources
    _WinAPI_SelectObject($hDC, $obj_orig)
    _WinAPI_DeleteObject($hPen)
    _WinAPI_ReleaseDC(0, $hDC)
EndFunc   ;==>_UIA_DrawRect

 

[LarsJ]

UIASpy updates to Windows 10 version 1809
Documentation for Windows 10 version 1809 updates can be found in Windows 8, Windows 8.1 and Windows 10 updates. This post is pretty much completely rewritten.

In addition, there are a few minor updates that are documented below.

A small but significant update is that in Sample code to find the application top window, $TreeScope_Descendants is replaced by $TreeScope_Children. The application top window is always a direct child of the Desktop. See the treeview in UIASpy. Therefore, this update can be made. This is a major performance update when the application window is not at the top of the Z-order of windows.

UIA_MouseClick() in UIA_Functions.au3 is extended with right-click functionality. Sample code can be generated through Sample code | Code snippets ...

In UIASpy.ini, through the DisplayScale key, you can specify the display scale you want when UIASpy starts up.

Still not implemented update: Integrating $PropertyConditionFlags_MatchSubstring (Windows 10 1809) into Sample code.

New zip-file at bottom of first post.

[argumentum]

#include <WinAPISysWin.au3>  shows missing for AutoIt v3.3.14.5

Thanks for the update

[LarsJ]

Thank you Mr. argumentum. I'm on AutoIt 3.3.14.2 and forgot to test on 3.3.14.5. Zip-file in both examples is updated.