Simple mouse clicking in Java app using info from simplespy?

[mrwizardd]

My situation:

I'm trying to automate ticket management software called HP Service Manager that's made in Java. I'm on Windows 7 64 bit and the application is 32 bit. Here's a screenshot of the Application:

We often have Emails (first, second, and third notifications) that need to be send out to users about their computers coming in, or that we received a ticket in their name and for them to stop by so we can work on it.

I have a little experience with Autoit. Definitely never made anything pretty but I could get some simple keyboard key and mouse clicking down when it was needed. That's essentially what I'm trying to do here, so it doesn't need to be pretty, just functional. 

What I'm having trouble with:

I'm having a bit of trouble getting started with this. I'm using the SimpleSpy.au3 to get info from the button I need to click in HPSM. Here's the info I get:

Mouse position is retrieved 1467-590

At least we have an element [General ][]

Having the following values for all properties: 

Title is: <General > Class   := <> controltype:= <UIA_EditControlTypeId> ,<50004> , (0000C354)

*** Parent Information ***

Title is: <Selected notebook tab - General > Class   := <> controltype:= <UIA_HyperlinkControlTypeId> ,<50005> , (0000C355)

*** Detailed properties of the highlighted element ***

UIA_AcceleratorKeyPropertyId :=

UIA_AccessKeyPropertyId :=

UIA_AriaPropertiesPropertyId :=

UIA_AriaRolePropertyId :=

UIA_AutomationIdPropertyId :=

UIA_BoundingRectanglePropertyId :=1458;579;44;21

UIA_ClassNamePropertyId :=

UIA_ClickablePointPropertyId :=

UIA_ControllerForPropertyId :=

UIA_ControlTypePropertyId :=50004

UIA_CulturePropertyId :=0

UIA_DescribedByPropertyId :=

UIA_DockDockPositionPropertyId :=5

UIA_ExpandCollapseExpandCollapseStatePropertyId :=3

UIA_FlowsToPropertyId :=

UIA_FrameworkIdPropertyId :=

UIA_GridColumnCountPropertyId :=0

UIA_GridItemColumnPropertyId :=0

UIA_GridItemColumnSpanPropertyId :=1

UIA_GridItemContainingGridPropertyId :=

UIA_GridItemRowPropertyId :=0

UIA_GridItemRowSpanPropertyId :=1

UIA_GridRowCountPropertyId :=0

UIA_HasKeyboardFocusPropertyId :=False

UIA_HelpTextPropertyId :=

UIA_IsContentElementPropertyId :=True

UIA_IsControlElementPropertyId :=True

UIA_IsDataValidForFormPropertyId :=False

UIA_IsDockPatternAvailablePropertyId :=False

UIA_IsEnabledPropertyId :=True

UIA_IsExpandCollapsePatternAvailablePropertyId :=False

UIA_IsGridItemPatternAvailablePropertyId :=False

UIA_IsGridPatternAvailablePropertyId :=False

UIA_IsInvokePatternAvailablePropertyId :=True

UIA_IsItemContainerPatternAvailablePropertyId :=False

UIA_IsKeyboardFocusablePropertyId :=False

UIA_IsLegacyIAccessiblePatternAvailablePropertyId :=True

UIA_IsMultipleViewPatternAvailablePropertyId :=False

UIA_IsOffscreenPropertyId :=False

UIA_IsPasswordPropertyId :=False

UIA_IsRangeValuePatternAvailablePropertyId :=False

UIA_IsRequiredForFormPropertyId :=False

UIA_IsScrollItemPatternAvailablePropertyId :=False

UIA_IsScrollPatternAvailablePropertyId :=False

UIA_IsSelectionItemPatternAvailablePropertyId :=False

UIA_IsSelectionPatternAvailablePropertyId :=False

UIA_IsSynchronizedInputPatternAvailablePropertyId :=False

UIA_IsTableItemPatternAvailablePropertyId :=False

UIA_IsTablePatternAvailablePropertyId :=False

UIA_IsTextPatternAvailablePropertyId :=False

UIA_IsTogglePatternAvailablePropertyId :=False

UIA_IsTransformPatternAvailablePropertyId :=False

UIA_IsValuePatternAvailablePropertyId :=True

UIA_IsVirtualizedItemPatternAvailablePropertyId :=False

UIA_IsWindowPatternAvailablePropertyId :=False

UIA_ItemStatusPropertyId :=

UIA_ItemTypePropertyId :=

UIA_LabeledByPropertyId :=

UIA_LegacyIAccessibleChildIdPropertyId :=2

UIA_LegacyIAccessibleDefaultActionPropertyId :=Jump

UIA_LegacyIAccessibleDescriptionPropertyId :=

UIA_LegacyIAccessibleHelpPropertyId :=

UIA_LegacyIAccessibleKeyboardShortcutPropertyId :=

UIA_LegacyIAccessibleNamePropertyId :=General 

UIA_LegacyIAccessibleRolePropertyId :=42

UIA_LegacyIAccessibleSelectionPropertyId :=

UIA_LegacyIAccessibleStatePropertyId :=14680128

UIA_LegacyIAccessibleValuePropertyId :=javascript:setSelected(arr_X13,'X14_p');

UIA_LocalizedControlTypePropertyId :=edit

UIA_MultipleViewCurrentViewPropertyId :=0

UIA_MultipleViewSupportedViewsPropertyId :=

UIA_NamePropertyId :=General 

UIA_NativeWindowHandlePropertyId :=0

UIA_OrientationPropertyId :=0

UIA_ProcessIdPropertyId :=92156

UIA_ProviderDescriptionPropertyId :=[pid:92156,hwnd:0x0 Main(parent link):Microsoft: MSAA Proxy (unmanaged:uiautomationcore.dll)]

UIA_RangeValueIsReadOnlyPropertyId :=True

UIA_RangeValueLargeChangePropertyId :=0

UIA_RangeValueMaximumPropertyId :=0

UIA_RangeValueMinimumPropertyId :=0

UIA_RangeValueSmallChangePropertyId :=0

UIA_RangeValueValuePropertyId :=0

UIA_RuntimeIdPropertyId :=

UIA_ScrollHorizontallyScrollablePropertyId :=False

UIA_ScrollHorizontalScrollPercentPropertyId :=0

UIA_ScrollHorizontalViewSizePropertyId :=100

UIA_ScrollVerticallyScrollablePropertyId :=False

UIA_ScrollVerticalScrollPercentPropertyId :=0

UIA_ScrollVerticalViewSizePropertyId :=100

UIA_SelectionCanSelectMultiplePropertyId :=False

UIA_SelectionIsSelectionRequiredPropertyId :=False

UIA_SelectionselectionPropertyId :=

UIA_SelectionItemIsSelectedPropertyId :=False

UIA_SelectionItemSelectionContainerPropertyId :=

UIA_TableColumnHeadersPropertyId :=

UIA_TableItemColumnHeaderItemsPropertyId :=

UIA_TableRowHeadersPropertyId :=

UIA_TableRowOrColumnMajorPropertyId :=2

UIA_TableItemRowHeaderItemsPropertyId :=

UIA_ToggleToggleStatePropertyId :=2

UIA_TransformCanMovePropertyId :=False

UIA_TransformCanResizePropertyId :=False

UIA_TransformCanRotatePropertyId :=False

UIA_ValueIsReadOnlyPropertyId :=True

UIA_ValueValuePropertyId :=javascript:setSelected(arr_X13,'X14_p');

UIA_WindowCanMaximizePropertyId :=False

UIA_WindowCanMinimizePropertyId :=False

UIA_WindowIsModalPropertyId :=False

UIA_WindowIsTopmostPropertyId :=False

UIA_WindowWindowInteractionStatePropertyId :=0

UIA_WindowWindowVisualStatePropertyId :=0

 

I know I must be not doing something very basic to get this to work. When I run my script below, the window will activate and I won't get any errors from Autoit, but it won't actually click this button.

#include <GuiEdit.au3>
#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <WinAPI.au3>
#include "UIAWrappers.au3"
#include <Misc.au3>
#include <Constants.au3>


#cs ----------------------------------------------------------------------------


 AutoIt Version: 3.3.12.0


#ce ----------------------------------------------------------------------------


#AutoIt3Wrapper_UseX64=Y


WinActivate("HPSM Overlay v2")
ControlClick("General", "", "controltype [UIA_HyperlinkControlTypeId]")

I've tried adding the x. before it as well. With that I get an error that Autoit doesn't recognize "x." What am I doing wrong? What would be the correct way to script it to simply click the button? I think once I get something as simple as this down I can get the rest of what I'm hoping to do. If you might have links to basic examples of automating a java app that would be greatly appreciated too! Thanks folks.

 

Edited July 21, 2014 by mrwizardd

[232showtime]

where did you get simplepsy.au3? can you share it to me?

[careca]

What does "AU3Info" show?

It looks like the Id is 50004

So you might want to try

ControlClick("HPSM Overlay v2", "", "50004")

Syntax is: ControlClick ( "title", "text", controlID [, button = "left" [, clicks = 1 [, x [, y]]]] )

From the controlID onwards is optional.

Of course im not sure the id is that, but won't harm for sure.

EDIT: got the title wrong, if you know it is "General" then put that.

Edited July 21, 2014 by careca

[mrwizardd]

where did you get simplepsy.au3? can you share it to me?

 Having trouble finding that original post where a guy made it and required a .jar file and some .au3's in the "Include" folder. But here is the script:

(EDIT: Found it! '?do=embed' frameborder='0' data-embedContent>> )

expandcollapse popup

#include <GuiEdit.au3>
#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <WinAPI.au3>
#include "UIAWrappers.au3"
#include <Misc.au3>


AutoItSetOption("MustDeclareVars", 1)


#AutoIt3Wrapper_UseX64=Y  ;Should be used for stuff like tagpoint having right struct etc. when running on a 64 bits os


dim $oldUIElement ; To keep track of latest referenced element
dim $Form1, $Edit1, $label1, $label2, $msg, $x, $y, $oUIElement, $oTW, $objParent, $oldElement, $text1, $t


;~ Some references for reading
;~ [url=http://support.microsoft.com/kb/138518/nl]http://support.microsoft.com/kb/138518/nl[/url]  tagpoint structures attention point
;~ [url=http://www.autoitscript.com/forum/topic/128406-interface-autoitobject-iuiautomation/]http://www.autoitscript.com/forum/topic/128406-interface-autoitobject-iuiautomation/[/url]
;~ [url=http://msdn.microsoft.com/en-us/library/windows/desktop/ff625914(v=vs.85).aspx]http://msdn.microsoft.com/en-us/library/windows/desktop/ff625914(v=vs.85).aspx[/url]


HotKeySet("{ESC}", "Close") ; Set ESC as a hotkey to exit the script.
HotKeySet("^w", "GetElementInfo") ; Set Hotkey Ctrl+M to get some basic information in the GUI


#Region ### START Koda GUI section ### Form=
$Form1 = GUICreate("Form1", 1024, 768, 192, 124)
$Edit1 = GUICtrlCreateEdit("", 10, 10, 600, 700)
;~ $Edit1 = GUICtrlCreateEdit("", 10, 10, 10, 10)
$Label1 = GUICtrlCreateLabel("Ctrl+W to capture information", 640,10,600,25)
$Label2 = GUICtrlCreateLabel("Escape to exit", 640,45,600,25)
GUISetState(@SW_SHOW)
#EndRegion ### END Koda GUI section ###


_UIA_Init()


; Run the GUI until the dialog is closed
While true
$msg = GUIGetMsg()
sleep(100)
;~ if _ispressed(01) Then
;~ getelementinfo()
;~ endif
If $msg = $GUI_EVENT_CLOSE Then ExitLoop
WEnd


Func GetElementInfo()
Local $hWnd
Local $tStruct = DllStructCreate($tagPOINT) ; Create a structure that defines the point to be checked.
;~ Local $tStruct = DllStructCreate("INT64,INT64")
;~ Local $tStruct =_AutoItObject_DllStructCreate($tagPoint)


ToolTip("")
Global $UIA_oUIAutomation ;The main library core CUI automation reference
Global $UIA_oDesktop, $UIA_pDesktop  ;Desktop will be frequently the starting point


Global $UIA_oUIElement, $UIA_pUIElement  ;Used frequently to get an element
Global $UIA_oTW, $UIA_pTW  ;Generic treewalker which is allways available


$x=MouseGetPos(0)
$y=MouseGetPos(1)
DllStructSetData($tStruct, "x", $x)
DllStructSetData($tStruct, "y", $y)
consolewrite(DllStructGetData($tStruct,"x") & DllStructGetData($tStruct,"y"))


;~ consolewrite("Mouse position is retrieved " & @crlf)
$UIA_oUIAutomation.ElementFromPoint($tStruct,$UIA_pUIElement )
;~ $objUIAutomation.ElementFromPoint(DllStructGetPtr($tStruct),$pUIElement)
;~ consolewrite("Element from point is passed, trying to convert to object ")
$oUIElement = objcreateinterface($UIA_pUIElement,$sIID_IUIAutomationElement, $dtagIUIAutomationElement)


$UIA_oUIAutomation.RawViewWalker($UIA_pTW)
$oTW=ObjCreateInterface($UIA_pTW, $sIID_IUIAutomationTreeWalker, $dtagIUIAutomationTreeWalker)
    If IsObj($oTW) = 0 Then
        msgbox(1,"UI automation treewalker failed", "UI Automation failed failed",10)
    EndIf


local $parentHandle


$oTW.getparentelement($oUIElement,$parentHandle)
$objParent=objcreateinterface($parentHandle,$sIID_IUIAutomationElement, $dtagIUIAutomationElement)
If IsObj($objParent) = 0 Then
msgbox(1,"No parent", "UI Automation failed failed",10)
EndIf


if isobj($oldUIElement) Then
if $oldUIElement=$oUIElement then
return
endif
endif
_WinAPI_RedrawWindow(_WinAPI_GetDesktopWindow(), 0, 0, $RDW_INVALIDATE + $RDW_ALLCHILDREN) ; Clears Red outline graphics.
GUICtrlSetData($Edit1, "Mouse position is retrieved " & $x & "-" & $y & @CRLF)
$oldElement=$oUIElement


If IsObj($oUIElement) Then
;~  ConsoleWrite("At least we have an element "  & "[" & _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId) & "][" & _UIA_getPropertyValue($oUIElement, $UIA_ClassNamePropertyId) & "]" & @CRLF)
GUICtrlSetData($Edit1, "At least we have an element "  & "[" & _UIA_getPropertyValue($oUIElement, $UIA_NamePropertyId) & "][" & _UIA_getPropertyValue($oUIElement, $UIA_ClassNamePropertyId) & "]" & @CRLF,1)
    $text1="Title is: <" &  _UIA_getPropertyValue($oUIElement,$UIA_NamePropertyId) &  ">" & @TAB _
   & "Class   := <" & _UIA_getPropertyValue($oUIElement,$uia_classnamepropertyid) &  ">" & @TAB _
& "controltype:= " _
& "<" &  _UIA_getControlName(_UIA_getPropertyValue($oUIElement,$UIA_ControlTypePropertyId)) &  ">" & @TAB  _
& ",<" &  _UIA_getPropertyValue($oUIElement,$UIA_ControlTypePropertyId) &  ">" & @TAB  _
& ", (" &  hex(_UIA_getPropertyValue($oUIElement,$UIA_ControlTypePropertyId)) &  ")" & @TAB & @CRLF


    $text1=$text1 & "*** Parent Information ***" & @CRLF


    $text1=$Text1 & "Title is: <" &  _UIA_getPropertyValue($objParent,$UIA_NamePropertyId) &  ">" & @TAB _
   & "Class   := <" & _UIA_getPropertyValue($objParent,$uia_classnamepropertyid) &  ">" & @TAB _
& "controltype:= " _
& "<" &  _UIA_getControlName(_UIA_getPropertyValue($objParent,$UIA_ControlTypePropertyId)) &  ">" & @TAB  _
& ",<" &  _UIA_getPropertyValue($objParent,$UIA_ControlTypePropertyId) &  ">" & @TAB  _
& ", (" &  hex(_UIA_getPropertyValue($objParent,$UIA_ControlTypePropertyId)) &  ")" & @TAB & @CRLF


$text1=$text1 & "*** Detailed properties of the highlighted element ***"


$text1= $text1 & @CRLF & _UIA_getAllPropertyValues($oUIElement)


GUICtrlSetData($Edit1, "Having the following values for all properties: " & @crlf & $text1 & @CRLF, 1)




_GUICtrlEdit_LineScroll($Edit1, 0, 0 - _GUICtrlEdit_GetLineCount($Edit1))


$t=stringsplit(_UIA_getPropertyValue($oUIElement, $UIA_BoundingRectanglePropertyId),";")
_UIA_DrawRect($t[1],$t[3]+$t[1],$t[2],$t[4]+$t[2])
EndIf


EndFunc   ;==>Example


Func Close()
Exit
EndFunc   ;==>Close

Edited July 21, 2014 by mrwizardd

[mrwizardd]

What does "AU3Info" show?

It looks like the Id is 50004

So you might want to try

ControlClick("HPSM Overlay v2", "", "50004")

Syntax is: ControlClick ( "title", "text", controlID [, button = "left" [, clicks = 1 [, x [, y]]]] )

From the controlID onwards is optional.

Of course im not sure the id is that, but won't harm for sure.

 

AU3Info is not recognizing any of the Java buttons, just the window. From what I understand, native Autoit does not support Java buttons and controls. But using that script above, it can identify them. May need some of the files I originally downloaded first. See

Junkew put together something that will allow Autoit to send commands to them inside the Java App. I tried your suggestions, thank you, but no dice

Edited July 21, 2014 by mrwizardd

[careca]

Sadly i don't know of any alternative, only mouseclick.

[mrwizardd]

That would actually work great if I could actually get it to click the button being seen by simplespy. I may have to try and message Junkew about how to use that stuff.

[junkew]

Please read the iuiautomation thread and see what can and cannot work. It heavily depends on how the java application is build. For example lotus notes is written in java and supports full accessibility. If its the same for your application you can best assess by looking with simplespy how easy elements are highligthed. Commands and examples are in the first post attached in zip files.

And you should then not us autoit commands to control elements.

Edited July 21, 2014 by junkew

[careca]

Well, at the top of the post with the spy info you got: "Mouse position is retrieved 1467-590"

Therefore:

MouseClick("left", 1467, 590)

[mrwizardd]

 

Well, at the top of the post with the spy info you got: "Mouse position is retrieved 1467-590"

Therefore:

MouseClick("left", 1467, 590)

 But the window may not always be in the same place. Certain line items may move or the constraints may change

[careca]

Maybe you could do a pixel search, for a specific shade of color specific to a button.