Clicking button or sending shortcut to (stubbern) Windows - (Moved)

[dejhost]

Hi.

I am trying to automate a software called "LabelImg" (https://pypi.org/project/labelImg/).

My autoit-script is started once I selected a folder with images within LabelImg. Pressing the button "Next Image" or pressing the shortcut "d" (https://github.com/tzutalin/labelImg#Hotkeys) jumps to the next image in the selected folder. This shall happen once per second.

#include <Misc.au3>
#include <MsgBoxConstants.au3>
#include <AutoItConstants.au3>
Opt("WinTitleMatchMode", 1)

Local $hDLL = DllOpen("user32.dll")

While 1

    If _IsPressed("1B", $hDLL) Then
        ExitLoop
    Else
        Local $temp = WinActivate("labelImg")
        ConsoleWrite($temp & @CRLF)
        If WinActivate("labelImg") Then
            ConsoleWrite("All Set!" & @CRLF)
        EndIf

        ;Send("d")
        Local $temp = MouseClick($MOUSE_CLICK_RIGHT, 50, 200)
        If $temp <> 1 Then
            MsgBox(1, "$temp", $temp)
            ExitLoop
        EndIf
        Sleep(1000)
    EndIf
WEnd

DllClose($hDLL)

So the Send ("d")-command and the MouseClick are alternative methods to jump to the next image. Both fail.

Both ConsoleWrite's deliver proper feedback (I continiously get the handle and "All set" ).

Could you tell me what I'm doing wrong? 

Thank you.

Edited October 20, 2021 by dejhost

[Jos]

Moved to the appropriate forum, as the Developer General Discussion forum very clearly states:

Quote

General development and scripting discussions.

---

Do not create AutoIt-related topics here, use the AutoIt General Help and Support or AutoIt Technical Discussion forums.

Moderation Team

[dejhost]

Thanks for moving this. My bad. 🙄

[Luke94]

Do you know the ClassName, Instance and ID of the button you're trying to click? You could try ControlClick?

[Danp2]

You should search the forum for the term "Qt5QWindowIcon" and review the past discussions.

[dejhost]

1 hour ago, Luke94 said:

Do you know the ClassName, Instance and ID of the button you're trying to click? You could try ControlClick?

The image above shows all info I retrieved: no data for ID, classname, instance or so :-(

Will give Qt5QWindowIcon a shot tomorrow and report.

[junkew]

read faq31 and use the different spying tools mentioned in faq. QT widgets really depend on what the developers did for accessibility.

By searching the forum you will find

 

[dejhost]

Thanks a lot for the hints!

Simple UIA Spy tells me the following:

expandcollapse popup

Mouse position is retrieved 54-217
At least we have an element title: [Next Image] class: []

Having the following values for all properties: 
Title is: <Next Image>  Class   := <>   controltype:= <UIA_ButtonControlTypeId> ,<50000>    , (0000C350)    0;180;97;43
*** Parent Information top down ***
1: Title is: <labelImg C:/Users/Hidden/Desktop/1.jpg [1 / 1]>   Class   := <Qt5QWindowIcon> controltype:= <UIA_WindowControlTypeId> ,<50032>    , (0000C370)    -7;-7;2208;1208
"Title:=labelImg C:/Users/Hidden/Desktop/1.jpg [1 / 1];controltype:=UIA_WindowControlTypeId;class:=Qt5QWindowIcon"" 
0: Title is: <Tools>    Class   := <>   controltype:= <UIA_ToolBarControlTypeId>    ,<50021>    , (0000C365)    0;43;97;1131
"Title:=Tools;controltype:=UIA_ToolBarControlTypeId;class:=""   


;~ *** Standard code maintainable ***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

_UIA_setVar("oP1","Title:=labelImg C:/Users/Hidden/Desktop/1.jpg [1 / 1];controltype:=UIA_WindowControlTypeId;class:=Qt5QWindowIcon")   ;labelImg C:/Users/Hidden/Desktop/1.jpg [1 / 1]
_UIA_setVar("oP2","Title:=Tools;controltype:=UIA_ToolBarControlTypeId;class:=") ;Tools

;~ $oUIElement=_UIA_getObjectByFindAll("NextImage.mainwindow", "title:=Next Image;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)
_UIA_setVar("oUIElement","Title:=Next Image;controltype:=UIA_ButtonControlTypeId;class:=") ;ControlType:=UIA_ButtonControlTypeId;classname:=")

;~ Actions split away from logical/technical definition above can come from configfiles 

;~_UIA_Action("oP1","highlight")
_UIA_Action("oP1","setfocus")
;~_UIA_Action("oP2","highlight")
_UIA_Action("oP2","setfocus")

;~_UIA_action($oUIElement","highlight")
;~_UIA_action($oUIElement,"click")
;~_UIA_action("oUIElement","highlight")
;~_UIA_action("oUIElement","click")


;~ *** Standard code Flexible***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

Local $oP1=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=labelImg C:/Users/Hidden/Desktop/1.jpg [1 / 1];controltype:=UIA_WindowControlTypeId;class:=Qt5QWindowIcon", $treescope_children)  
_UIA_Action($oP1,"setfocus")
Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=Tools;controltype:=UIA_ToolBarControlTypeId;class:=", $treescope_children) 
;~ First find the object in the parent before you can do something
;~$oUIElement=_UIA_getObjectByFindAll("NextImage.mainwindow", "title:=Next Image;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)
Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=Next Image;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)
;~_UIA_action($oUIElement,"highlight")
_UIA_action($oUIElement,"click")


*** Detailed properties of the highlighted element ***
UIA_title:= <Next Image>
UIA_text:= <Next Image>
UIA_regexptitle:= <Next Image>
UIA_iaccessiblechildId:= <0>
UIA_handle:= <0>
UIA_BoundingRectangle:= <0;180;97;43>
UIA_ProcessId:= <14960>
UIA_ControlType:= <50000>
UIA_LocalizedControlType:= <button>
UIA_Name:= <Next Image>
UIA_HasKeyboardFocus:= <False>
UIA_IsKeyboardFocusable:= <True>
UIA_IsEnabled:= <True>
UIA_Culture:= <0>
UIA_IsControlElement:= <True>
UIA_IsContentElement:= <True>
UIA_IsPassword:= <False>
UIA_NativeWindowHandle:= <0>
UIA_IsOffscreen:= <False>
UIA_Orientation:= <0>
UIA_IsRequiredForForm:= <False>
UIA_IsDockPatternAvailable:= <False>
UIA_IsExpandCollapsePatternAvailable:= <False>
UIA_IsGridItemPatternAvailable:= <False>
UIA_IsGridPatternAvailable:= <False>
UIA_IsInvokePatternAvailable:= <True>
UIA_IsMultipleViewPatternAvailable:= <False>
UIA_IsRangeValuePatternAvailable:= <False>
UIA_IsScrollPatternAvailable:= <False>
UIA_IsScrollItemPatternAvailable:= <False>
UIA_IsSelectionItemPatternAvailable:= <False>
UIA_IsSelectionPatternAvailable:= <False>
UIA_IsTablePatternAvailable:= <False>
UIA_IsTableItemPatternAvailable:= <False>
UIA_IsTextPatternAvailable:= <False>
UIA_IsTogglePatternAvailable:= <False>
UIA_IsTransformPatternAvailable:= <False>
UIA_IsValuePatternAvailable:= <False>
UIA_IsWindowPatternAvailable:= <False>
UIA_ValueIsReadOnly:= <True>
UIA_RangeValueValue:= <0>
UIA_RangeValueIsReadOnly:= <True>
UIA_RangeValueMinimum:= <0>
UIA_RangeValueMaximum:= <0>
UIA_RangeValueLargeChange:= <0>
UIA_RangeValueSmallChange:= <0>
UIA_ScrollHorizontalScrollPercent:= <0>
UIA_ScrollHorizontalViewSize:= <100>
UIA_ScrollVerticalScrollPercent:= <0>
UIA_ScrollVerticalViewSize:= <100>
UIA_ScrollHorizontallyScrollable:= <False>
UIA_ScrollVerticallyScrollable:= <False>
UIA_SelectionCanSelectMultiple:= <False>
UIA_SelectionIsSelectionRequired:= <False>
UIA_GridRowCount:= <0>
UIA_GridColumnCount:= <0>
UIA_GridItemRow:= <0>
UIA_GridItemColumn:= <0>
UIA_GridItemRowSpan:= <1>
UIA_GridItemColumnSpan:= <1>
UIA_DockDockPosition:= <5>
UIA_ExpandCollapseExpandCollapseState:= <3>
UIA_MultipleViewCurrentView:= <0>
UIA_WindowCanMaximize:= <False>
UIA_WindowCanMinimize:= <False>
UIA_WindowWindowVisualState:= <0>
UIA_WindowWindowInteractionState:= <0>
UIA_WindowIsModal:= <False>
UIA_WindowIsTopmost:= <False>
UIA_SelectionItemIsSelected:= <False>
UIA_TableRowOrColumnMajor:= <2>
UIA_ToggleToggleState:= <2>
UIA_TransformCanMove:= <False>
UIA_TransformCanResize:= <False>
UIA_TransformCanRotate:= <False>
UIA_IsLegacyIAccessiblePatternAvailable:= <True>
UIA_LegacyIAccessibleChildId:= <0>
UIA_LegacyIAccessibleName:= <Next Image>
UIA_LegacyIAccessibleDescription:= <Open the next Image>
UIA_LegacyIAccessibleRole:= <43>
UIA_LegacyIAccessibleState:= <1048704>
UIA_LegacyIAccessibleDefaultAction:= <Press>
UIA_IsDataValidForForm:= <False>
UIA_ProviderDescription:= <[pid:14960,providerId:0x0 Main(parent link):Microsoft: MSAA Proxy (unmanaged:uiautomationcore.dll)]>
UIA_IsItemContainerPatternAvailable:= <False>
UIA_IsVirtualizedItemPatternAvailable:= <False>
UIA_IsSynchronizedInputPatternAvailable:= <False>

Is there a way to adress the button?

I also installed Ranorex Studio (Trial), but this seems overkill for just automating simple mouse-clicks.

Edited October 22, 2021 by dejhost

[junkew]

I am not sure did you see simplespy highlight your area of interest?
if so yes then the code you gave above is a solution to automate your button. As you have a uiabounding rectangle you can move the mouse to that location and send a mouseclick on that location MouseMove,  MouseClick, MouseClickDrag, MouseCoordMode (Option), MouseDown, MouseGetPos, MouseUp, MouseWheel or search for sendinput in forum.

If no then you should look at

• developers making the QT application more accessibile (then they are also more reachable for UIA)
• commercial testautomation solutions (which sometimes hook deeper into your system but probably then need admin access to run)
  • Ranorex
  • Testcomplete
  • ....

 

[AlessandroAvolio]

@dejhost I downloaded an executable prebuild for Windows of labelImg, I have no problems with MouseClick and Send. I made this video. In your opinion what can be the explanation? 

[dejhost]

@junkew: simplespy highligts an area close by, displaying a red rectangle. So it doesn't highlight the button I am hovering over. That means, I need to look into alternative methonds 😞

 

@AlessandroAvolio: That's interesting! Thank you for trying. I built labelImg using anconda. That's the only possible reason I can think of. I now downloaded the prebuild binaries - which refuse to run: I just get a brief terminal window displayed. Did version 1.8.0 run straight out of the box in your case?
 

 

 

[AlessandroAvolio]

Yes it runs without problems. First a brief terminal and then the GUI (Win7 64 bit) https://tzutalin.github.io/labelImg/

Edited October 25, 2021 by AlessandroAvolio

[junkew]

Based on the spy output and your comment there is a red rectangle (assuming its the size of your button) I guess your DPI scaling of the screen is not 100%.

As spy is telling you this you know its recognized by using UIA library

"title:=Next Image;ControlType:=UIA_ButtonControlTypeId"

Then your next problem is how to invoke the button. Some options

1. Use the UIA invoke pattern (thats scaling independent, but not allways available)

2. Use the mousemove/mouseclick to the element boundingrectangle position to click (but then your DPI scaling could mean some addtional handling to do)

[dejhost]

@junkew: it seems indeed to be a scaling problem: when I move mousecursor to a button above or below "Next", and press 'control +w' to acivate simplespy, the red rectangle moves accordingly.

However, I got the precompiled version of LabelImg running by deleting a file in the user folder (https://github.com/tzutalin/labelImg/issues/221#issuecomment-368237423). In this version, the mouseclick works just fine - just as well as sending 'd', which is a builtin hotkey for "Next image" in LabelImg.

 

So with this, my problem is solved. Thank you very much for your efforts! I really appreciate it.