Jump to content

Recommended Posts

Posted (edited)

Hi.

I am trying to automate a software called "LabelImg" (https://pypi.org/project/labelImg/).

My autoit-script is started once I selected a folder with images within LabelImg. Pressing the button "Next Image" or pressing the shortcut "d" (https://github.com/tzutalin/labelImg#Hotkeys) jumps to the next image in the selected folder. This shall happen once per second.

#include <Misc.au3>
#include <MsgBoxConstants.au3>
#include <AutoItConstants.au3>
Opt("WinTitleMatchMode", 1)

Local $hDLL = DllOpen("user32.dll")

While 1

    If _IsPressed("1B", $hDLL) Then
        ExitLoop
    Else
        Local $temp = WinActivate("labelImg")
        ConsoleWrite($temp & @CRLF)
        If WinActivate("labelImg") Then
            ConsoleWrite("All Set!" & @CRLF)
        EndIf

        ;Send("d")
        Local $temp = MouseClick($MOUSE_CLICK_RIGHT, 50, 200)
        If $temp <> 1 Then
            MsgBox(1, "$temp", $temp)
            ExitLoop
        EndIf
        Sleep(1000)
    EndIf
WEnd

DllClose($hDLL)

So the Send ("d")-command and the MouseClick are alternative methods to jump to the next image. Both fail.

Both ConsoleWrite's deliver proper feedback (I continiously get the handle and "All set" ).

image.thumb.png.f586837c4a29c54f24a40bb6b1d11970.png

Could you tell me what I'm doing wrong? 

Thank you.

Edited by dejhost
  • Developers
Posted

Moved to the appropriate forum, as the Developer General Discussion forum very clearly states:

Quote

General development and scripting discussions.


Do not create AutoIt-related topics here, use the AutoIt General Help and Support or AutoIt Technical Discussion forums.

Moderation Team

SciTE4AutoIt3 Full installer Download page   - Beta files       Read before posting     How to post scriptsource   Forum etiquette  Forum Rules 
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Posted
1 hour ago, Luke94 said:

Do you know the ClassName, Instance and ID of the button you're trying to click? You could try ControlClick?

The image above shows all info I retrieved: no data for ID, classname, instance or so :-(

Will give Qt5QWindowIcon a shot tomorrow and report.

Posted (edited)

Thanks a lot for the hints!

Simple UIA Spy tells me the following:

Mouse position is retrieved 54-217
At least we have an element title: [Next Image] class: []

Having the following values for all properties: 
Title is: <Next Image>  Class   := <>   controltype:= <UIA_ButtonControlTypeId> ,<50000>    , (0000C350)    0;180;97;43
*** Parent Information top down ***
1: Title is: <labelImg C:/Users/Hidden/Desktop/1.jpg [1 / 1]>   Class   := <Qt5QWindowIcon> controltype:= <UIA_WindowControlTypeId> ,<50032>    , (0000C370)    -7;-7;2208;1208
"Title:=labelImg C:/Users/Hidden/Desktop/1.jpg [1 / 1];controltype:=UIA_WindowControlTypeId;class:=Qt5QWindowIcon"" 
0: Title is: <Tools>    Class   := <>   controltype:= <UIA_ToolBarControlTypeId>    ,<50021>    , (0000C365)    0;43;97;1131
"Title:=Tools;controltype:=UIA_ToolBarControlTypeId;class:=""   


;~ *** Standard code maintainable ***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

_UIA_setVar("oP1","Title:=labelImg C:/Users/Hidden/Desktop/1.jpg [1 / 1];controltype:=UIA_WindowControlTypeId;class:=Qt5QWindowIcon")   ;labelImg C:/Users/Hidden/Desktop/1.jpg [1 / 1]
_UIA_setVar("oP2","Title:=Tools;controltype:=UIA_ToolBarControlTypeId;class:=") ;Tools

;~ $oUIElement=_UIA_getObjectByFindAll("NextImage.mainwindow", "title:=Next Image;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)
_UIA_setVar("oUIElement","Title:=Next Image;controltype:=UIA_ButtonControlTypeId;class:=") ;ControlType:=UIA_ButtonControlTypeId;classname:=")

;~ Actions split away from logical/technical definition above can come from configfiles 

;~_UIA_Action("oP1","highlight")
_UIA_Action("oP1","setfocus")
;~_UIA_Action("oP2","highlight")
_UIA_Action("oP2","setfocus")

;~_UIA_action($oUIElement","highlight")
;~_UIA_action($oUIElement,"click")
;~_UIA_action("oUIElement","highlight")
;~_UIA_action("oUIElement","click")


;~ *** Standard code Flexible***
#include "UIAWrappers.au3"
AutoItSetOption("MustDeclareVars", 1)

Local $oP1=_UIA_getObjectByFindAll($UIA_oDesktop, "Title:=labelImg C:/Users/Hidden/Desktop/1.jpg [1 / 1];controltype:=UIA_WindowControlTypeId;class:=Qt5QWindowIcon", $treescope_children)  
_UIA_Action($oP1,"setfocus")
Local $oP0=_UIA_getObjectByFindAll($oP1, "Title:=Tools;controltype:=UIA_ToolBarControlTypeId;class:=", $treescope_children) 
;~ First find the object in the parent before you can do something
;~$oUIElement=_UIA_getObjectByFindAll("NextImage.mainwindow", "title:=Next Image;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)
Local $oUIElement=_UIA_getObjectByFindAll($oP0, "title:=Next Image;ControlType:=UIA_ButtonControlTypeId", $treescope_subtree)
;~_UIA_action($oUIElement,"highlight")
_UIA_action($oUIElement,"click")


*** Detailed properties of the highlighted element ***
UIA_title:= <Next Image>
UIA_text:= <Next Image>
UIA_regexptitle:= <Next Image>
UIA_iaccessiblechildId:= <0>
UIA_handle:= <0>
UIA_BoundingRectangle:= <0;180;97;43>
UIA_ProcessId:= <14960>
UIA_ControlType:= <50000>
UIA_LocalizedControlType:= <button>
UIA_Name:= <Next Image>
UIA_HasKeyboardFocus:= <False>
UIA_IsKeyboardFocusable:= <True>
UIA_IsEnabled:= <True>
UIA_Culture:= <0>
UIA_IsControlElement:= <True>
UIA_IsContentElement:= <True>
UIA_IsPassword:= <False>
UIA_NativeWindowHandle:= <0>
UIA_IsOffscreen:= <False>
UIA_Orientation:= <0>
UIA_IsRequiredForForm:= <False>
UIA_IsDockPatternAvailable:= <False>
UIA_IsExpandCollapsePatternAvailable:= <False>
UIA_IsGridItemPatternAvailable:= <False>
UIA_IsGridPatternAvailable:= <False>
UIA_IsInvokePatternAvailable:= <True>
UIA_IsMultipleViewPatternAvailable:= <False>
UIA_IsRangeValuePatternAvailable:= <False>
UIA_IsScrollPatternAvailable:= <False>
UIA_IsScrollItemPatternAvailable:= <False>
UIA_IsSelectionItemPatternAvailable:= <False>
UIA_IsSelectionPatternAvailable:= <False>
UIA_IsTablePatternAvailable:= <False>
UIA_IsTableItemPatternAvailable:= <False>
UIA_IsTextPatternAvailable:= <False>
UIA_IsTogglePatternAvailable:= <False>
UIA_IsTransformPatternAvailable:= <False>
UIA_IsValuePatternAvailable:= <False>
UIA_IsWindowPatternAvailable:= <False>
UIA_ValueIsReadOnly:= <True>
UIA_RangeValueValue:= <0>
UIA_RangeValueIsReadOnly:= <True>
UIA_RangeValueMinimum:= <0>
UIA_RangeValueMaximum:= <0>
UIA_RangeValueLargeChange:= <0>
UIA_RangeValueSmallChange:= <0>
UIA_ScrollHorizontalScrollPercent:= <0>
UIA_ScrollHorizontalViewSize:= <100>
UIA_ScrollVerticalScrollPercent:= <0>
UIA_ScrollVerticalViewSize:= <100>
UIA_ScrollHorizontallyScrollable:= <False>
UIA_ScrollVerticallyScrollable:= <False>
UIA_SelectionCanSelectMultiple:= <False>
UIA_SelectionIsSelectionRequired:= <False>
UIA_GridRowCount:= <0>
UIA_GridColumnCount:= <0>
UIA_GridItemRow:= <0>
UIA_GridItemColumn:= <0>
UIA_GridItemRowSpan:= <1>
UIA_GridItemColumnSpan:= <1>
UIA_DockDockPosition:= <5>
UIA_ExpandCollapseExpandCollapseState:= <3>
UIA_MultipleViewCurrentView:= <0>
UIA_WindowCanMaximize:= <False>
UIA_WindowCanMinimize:= <False>
UIA_WindowWindowVisualState:= <0>
UIA_WindowWindowInteractionState:= <0>
UIA_WindowIsModal:= <False>
UIA_WindowIsTopmost:= <False>
UIA_SelectionItemIsSelected:= <False>
UIA_TableRowOrColumnMajor:= <2>
UIA_ToggleToggleState:= <2>
UIA_TransformCanMove:= <False>
UIA_TransformCanResize:= <False>
UIA_TransformCanRotate:= <False>
UIA_IsLegacyIAccessiblePatternAvailable:= <True>
UIA_LegacyIAccessibleChildId:= <0>
UIA_LegacyIAccessibleName:= <Next Image>
UIA_LegacyIAccessibleDescription:= <Open the next Image>
UIA_LegacyIAccessibleRole:= <43>
UIA_LegacyIAccessibleState:= <1048704>
UIA_LegacyIAccessibleDefaultAction:= <Press>
UIA_IsDataValidForForm:= <False>
UIA_ProviderDescription:= <[pid:14960,providerId:0x0 Main(parent link):Microsoft: MSAA Proxy (unmanaged:uiautomationcore.dll)]>
UIA_IsItemContainerPatternAvailable:= <False>
UIA_IsVirtualizedItemPatternAvailable:= <False>
UIA_IsSynchronizedInputPatternAvailable:= <False>

Is there a way to adress the button?

I also installed Ranorex Studio (Trial), but this seems overkill for just automating simple mouse-clicks.

Edited by dejhost
Posted

I am not sure did you see simplespy highlight your area of interest?
if so yes then the code you gave above is a solution to automate your button. As you have a uiabounding rectangle you can move the mouse to that location and send a mouseclick on that location MouseMove,  MouseClick, MouseClickDrag, MouseCoordMode (Option), MouseDown, MouseGetPos, MouseUp, MouseWheel or search for sendinput in forum.

If no then you should look at

  • developers making the QT application more accessibile (then they are also more reachable for UIA)
  • commercial testautomation solutions (which sometimes hook deeper into your system but probably then need admin access to run)
    • Ranorex
    • Testcomplete
    • ....

 

Posted

@junkew: simplespy highligts an area close by, displaying a red rectangle. So it doesn't highlight the button I am hovering over. That means, I need to look into alternative methonds 😞

 

@AlessandroAvolio: That's interesting! Thank you for trying. I built labelImg using anconda. That's the only possible reason I can think of. I now downloaded the prebuild binaries - which refuse to run: I just get a brief terminal window displayed. Did version 1.8.0 run straight out of the box in your case?
 

 

 

Posted

Based on the spy output and your comment there is a red rectangle (assuming its the size of your button) I guess your DPI scaling of the screen is not 100%.

As spy is telling you this you know its recognized by using UIA library

"title:=Next Image;ControlType:=UIA_ButtonControlTypeId"

Then your next problem is how to invoke the button. Some options

1. Use the UIA invoke pattern (thats scaling independent, but not allways available)

2. Use the mousemove/mouseclick to the element boundingrectangle position to click (but then your DPI scaling could mean some addtional handling to do)

Posted

@junkew: it seems indeed to be a scaling problem: when I move mousecursor to a button above or below "Next", and press 'control +w' to acivate simplespy, the red rectangle moves accordingly.

However, I got the precompiled version of LabelImg running by deleting a file in the user folder (https://github.com/tzutalin/labelImg/issues/221#issuecomment-368237423). In this version, the mouseclick works just fine - just as well as sending 'd', which is a builtin hotkey for "Next image" in LabelImg.

 

So with this, my problem is solved. Thank you very much for your efforts! I really appreciate it.

 

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...