Automate Qt and QWidgets

[seangriffin]

This post explains how to automate applications developed in Qt (http://qt.nokia.com) using AutoIT. This includes such popular applications as VLC, Google Earth, Skype, Virtual Box and Avidemux.

Qt based applications contain widgets which are not immediately accessible through the Au3Info tool. There are many forum posts on the subject of querying Qt based applications using Au3Info, with the result that it returns only a single QWidget object for the entire application or window.

Qt is a framework which wraps, or masks, the underlying native Win32 controls within it's GUI. Like other similar frameworks and languages (such as Java), Qt includes accessibility features which allow other non-Qt applications to interact with the framework. Tools like AutoIT can use these accessibility features to automate Qt applications.

I have not yet found a Qt accessibility bridge which AutoIT can use (such as in the form of a Windows DLL).

However, I have found a very useful tool called Ranorex Spy (http://www.ranorex.com/gui-testing-guide/spy.html), which uses the accessibility features of Qt to expose information about the QWidgets within an application, in a similar manner to Au3Info. There doesn't appear to be any license associated with Ranorex Spy, so I have attached the tool to this post for the benefit of others.

Use Ranorex Spy in the same manner that you use Au3Info. With the application you want to automate already open, drag the Finder Tool (crosshair) in Ranorex Spy over the application, and you'll notice that it can identify the widgets within the application (in the same manner that Au3Info would).

You can then use this information within the standard AutoIT function calls. Here are some examples.

If the QWidget has a "Caption/text", as identified in Ranorex Spy, then use the following AutoIT call to click on that control:

ControlClick("window title", "", "[TEXT:the Caption/text from Ranorex Spy]")

If the QWidget doesn't have a "Caption/text", then you can use the "Handle" from Ranorex Spy instead. Paste the handle value into the following AutoIT script:

Const $handle_to_find = "paste the handle from Ranorex Spy here"

For $i = 1 to 1000

    $handle = ControlGetHandle("window title", "", "[CLASS:QWidget; INSTANCE:" & $i & "]")

    if @error = 1 Then ExitLoop

    if ($handle = $handle_to_find) Then

        ConsoleWrite("Found QWidget handle " & $handle_to_find & " at instance: " & $i & @CRLF)
        ExitLoop
    EndIf
Next

Run the script and it should give you the QWidget instance number of the control. Use this instance number in the following AutoIT call to click on that control:

ControlClick("window title", "", "[CLASS:QWidget; INSTANCE:insert the instance number here]")

Hope this is useful to all those having trouble automating Qt.

Apologies everyone, but I've had to remove the "RanorexSpy.zip" attachment because I've run out of upload quota for the site. Contact me if you'd like a copy.

Edited February 11, 2012 by seangriffin

[IRON]

You're my personal hero! oO

Have long been searching desperately for a way to control Teamspeak3 ... and now it should finally be possible? Thank you!

[seangriffin]

You're my personal hero! oO

Have long been searching desperately for a way to control Teamspeak3 ... and now it should finally be possible? Thank you!

glad to be of service. I too was having a rough time automating Avidemux, until I stumbled over this solution.

I would have thought many frustrated users would like this, so I tried to inform people in older forum posts about Qt automation issues, though updating old posts seems to be some sort of improper behaviour these days, and my posts have been removed. Never used to be the case. Times have changed.

Well hope people like yourself still manage to find this post.

Edited July 2, 2011 by seangriffin

[Werty]

Dang, couldnt you have posted this a year ago? Could really have used it for my old Google Earth project

Joking ofcourse it's nice to know it's available now

[seangriffin]

Dang, couldnt you have posted this a year ago? Could really have used it for my old Google Earth project

Joking ofcourse it's nice to know it's available now

I hear your pain. I remember also trying to automate Avidemux about a year ago, only to give up in frustration! I'm also kicking myself for not finding this earlier.

To my surprise Qt has been around for a very long time (http://en.wikipedia.org/wiki/Qt_%28framework%29). I've actually never heard of it (and I've had to automate my own fair share of technologies in the past), though it's obviously a prominant development platform when you look at the long list of well known apps developed in it.

I hope I can help a few people with this one. And if Ranorex can produce a Qt inspection tool such as Ranorex Spy, then obviously something could be done specifically for AutoIT (as I had done earlier in my Java UDF to bridge Java accessibility with AutoIT via a DLL).

[yanivps]

First, I am really happy that this Thread helped a lot of people and solved main issues.

Unfortunately, I still have a problem with Internet Explorer_Server class (in Kaspersky Anti-virus 2011) and with QWidget class (in Kaspersky Anti-virus 2012).

Here is Kav 2011 screenshot (Nothing change in RanorexSpy program as i move my mouse in the window)

Here is Kav 2012 screenshot (Nothing change in RanorexSpy program as i move my mouse in the window)

I would really appriciate any help about this issue. it is really important to me to access the inner Controls in these applications.

Thanks

[hirak]

hey i've tried ranorex spy on skype but it is not working.

let's say here i've attached skype screenshot and i want to automate it.

but ranorex spy is not helpful to identify search box that is highlighted.

and it is failed to identify other components also.

[saget]

same problem, hope autoit can support QT framework in future....

[DssTrainer]

Thank you seangriffin for pioneering this process.

I've been working with this example the past few days with utter frustration and now I've figured a bit more out.

I am able to use Ranorex Spy to see the field caption/text in my QT application, but found that calling their controls only worked sometimes.

Then I figured it out....

If I use Ranorex finder tool on the element first, then call

ControlClick("MyAppTitle", "", "[TEXT:comboEnv]")

it works fine

If I close the app and then open it again and try to call ControlClick... it fails.

As soon as I use ranorex to scan the field again, THEN try the ControlClick.. it works!

So its like the fields have to be scanned by Ranorex first to be instantiated as their control names.

Very tricky stuff

I am knowingly necroposting since this is still a valid topic and moot to create a new one. I am also contributing new knowledge that may help others.

Edited August 10, 2012 by DssTrainer

[funkey]

@DssTrainer: Last year I tried Ranorex Spy with another application and it was the same for me to. AutoIt could only access the controls if Ranorex Spy scanned the application first.

[MvGulik]

Yep. That "AutoIt could only access the controls if Ranorex Spy scanned the application first." is a general know behavior.

You might like to take a look at the Interface AutoItObject IUIAutomation topic.

[DssTrainer]

^^

Thanks for the link. To simplify things, I instead made a quick-n-dirty RanorexifyMe($title) function that starts ranorexspy, enables HotTracking, hovers the mouse over my QT Application window, then kills ranorexspy... leaving my QT app properly accessible. Works perfectly.

Edited August 27, 2012 by DssTrainer

[jhall39]

What do you do if the application you are trying to automate has a new handle for each control every time you go into it? The instance numbers stay the same but can shuffle around depending on who logs in. So we can't even count on that, and only 1 control has any caption text to it.

I can't count on things being in the same place either, because the window might not be fullscreen or the screen resolutions might be different depending on who is using it. Is there anything else I can try that will let me always get the same buttons and controls all of the time, other than handles or caption text?

[DonAmit]

^^

Thanks for the link. To simplify things, I instead made a quick-n-dirty RanorexifyMe($title) function that starts ranorexspy, enables HotTracking, hovers the mouse over my QT Application window, then kills ranorexspy... leaving my QT app properly accessible. Works perfectly.

Hi, Can you please show me how to write the "quick-n-dirty" function that you are referring to? I tried the following but did not work

Run("E:\Installers\RanorexSpy.exe")
Sleep(2000)
ControlEnable("RanorexSpy", "", "&Enable")
Run(<myExe>)
Sleep(2000)

ControlFocus(<myTitle>, <myText>, "[CLASS:QWidget; INSTANCE:1]")
#include <GuiListView.au3>
_GUICtrlListView_SetHoverTime(<myTitle>, 2000)

Thanks,

-Amit

[Cupidkidd]

What do you do if the application you are trying to automate has a new handle for each control every time you go into it? The instance numbers stay the same but can shuffle around depending on who logs in. So we can't even count on that, and only 1 control has any caption text to it.

I can't count on things being in the same place either, because the window might not be fullscreen or the screen resolutions might be different depending on who is using it. Is there anything else I can try that will let me always get the same buttons and controls all of the time, other than handles or caption text?

Firstly, every time I open my Qt application, the control has a different handle. Secondly, I put the caption/text into the parameter but no action...

Do you know why?

[kevin3]

How do I click on a particular button in the Qwidget? For example with VLC media player, the play button?

Thanks!

[junkew]

With UI automation out of this thread you should be able to automate VLC
'?do=embed' frameborder='0' data-embedContent>>

[DssTrainer]

Hi, Can you please show me how to write the "quick-n-dirty" function that you are referring to? I tried the following but did not work

 

Run("E:\Installers\RanorexSpy.exe")
Sleep(2000)
ControlEnable("RanorexSpy", "", "&Enable")
Run(<myExe>)
Sleep(2000)

ControlFocus(<myTitle>, <myText>, "[CLASS:QWidget; INSTANCE:1]")
#include <GuiListView.au3>
_GUICtrlListView_SetHoverTime(<myTitle>, 2000)

Thanks,

-Amit

 

 

Here's my code:

Func Ranorexify($title, $text = "")

    Dim $winpos, $winsize

    ; Start RanorexSpy
    Run("C:\RanorexSpy.exe", "C:\", @SW_HIDE)

    ; Enable HotTrackerdev on RanorexSpy
    ControlCommand("RanorexSpy", "", "[TEXT:&Enable]", "Check", "")

    ; Activate Application Window
    if WinActivate($title) Then
        $winpos = WinGetPos($title, $text)
        if (IsArray($winpos)) Then
            MouseMove($winpos[0]+200, $winpos[1]+200, 0)
        Else
            MsgBox(1, "Window Not Found - " & @Scriptname, $title)
        EndIf

        ; Restore focus to Application
        WinActivate($title, $text)
    else
        MsgBox(1, "Window Not Active - " & @Scriptname, $title)
    EndIf

    ; Kill RanorexSpy
    WinKill("RanorexSpy")

EndFunc

Edited December 19, 2013 by DssTrainer

[junkew]

I assume Ranorex spy is doing some kind of handshake to activate within QT the accessibility

maybe stuff like this can work

PostMessage( HWND_BROADCAST, WM_WININICHANGE, 
SPI_SETSCREENREADER, 0 );

SystemParametersInfo( SPI_SETSCREENREADER, TRUE, NULL, SPIF_UPDATEINIFILE | 
SPIF_SENDCHANGE);

http://support.microsoft.com/kb/180958

A system environment variable "SPI_GETSCREENREADER" is referenced by some IAccessible2 Clients

or just do this

Make sure you have a folder named ‘accessible’ in your application’s folder with the executable and the file ‘qtaccessiblewidgets.dll’

Edited December 23, 2013 by junkew

[Proph]

Has anyone had luck using this method with Malwarebytes Version 2?  I can't seem to figure out how to ControlClick on the big green Scan Now button?  If someone could show me how to do that step I'm sure I can figure out the rest.  RanorexSpy doesn't seem to be getting any button info from Malwarebytes.