JAVA object automation and simple spy

[rlvitorino]

Last Access Bridge Explorer for download.

Access.zip

[junkew]

not sure why you post that zip here. its just on github https://github.com/google/access-bridge-explorer/releases

 

[rlvitorino]

If you want to get a text by its index ID I would use which command?

[junkew]

updated first post with @fablecao javaui.au3 code and updated jabsimplespy to use that

trouble on x64 to find correct path so hardcoded in javaui.au3

If $sJavaHome = "" Then

      consolewrite("  Unable to find Java Registry using hardcoded value" & @CRLF)
      $sJavaHome = "C:\Program Files\Java\jdk1.8.0_172\jre"

   EndIf

• in jabsimplespy source there is also an example function that iterates all java reconizable objects
• added highlighting when spying on object
  • shows also in more detail how to get information about an identified element

 

This is just an example on how to click on the About button similar as was given by @fablecao

• expandcollapse popup

  #include "JavaUI.au3"
  shellexecute("C:\Program Files\Java\jdk1.8.0_172\jre\bin\javacpl.exe")
  sleep(3000)
  
  global $wintitle = "Java Control Panel"
  
  $winHandle = WinActivate($wintitle)
  
  $result = __isJavaWindow($winHandle)
  
  Global $vmId
  Global $ac
  
  global $sName = "About..."
  global $sRole = "push button"
  
  __getAccessibleContextFromHWND($winHandle, $vmID, $ac)
  
  $re_ac=_JAB_getAccessibleContextByRole($vmId, $ac, $sRole, 1)
  
  $sName = _JAB_getName($vmId, $re_ac)
  
  Local $acInfo = DllStructCreate($tagAccessibleContextInfo)
  __getAccessibleContextInfo($vmId, $re_ac, $acInfo)
  $acX = DllStructGetData($acInfo, "x")
  $acY = DllStructGetData($acInfo, "y")
  $acW = DllStructGetData($acInfo, "width")
  $acH = DllStructGetData($acInfo, "height")
  
  ConsoleWrite("Name:" & $sName & @CRLF)
  ConsoleWrite("rect:" & $acX & $acX+$acW & $acy & $acY+$acH & @CRLF)
  
  
  $re_ac = _JAB_getAccessibleContextByFindAll($vmId, $ac, $sName, $sRole)
  $sName = _JAB_getName($vmId, $re_ac)
  
  Local $acInfo2 = DllStructCreate($tagAccessibleContextInfo)
  __getAccessibleContextInfo($vmId, $re_ac, $acInfo2)
  $acX = DllStructGetData($acInfo2, "x")
  $acY = DllStructGetData($acInfo2, "y")
  $acW = DllStructGetData($acInfo2, "width")
  $acH = DllStructGetData($acInfo2, "height")
  ConsoleWrite("Name:" & $sName & @CRLF)
  ConsoleWrite("rect:" & $acX & $acX+$acW & $acy & $acY+$acH & @CRLF)
  
  _JAB_singleAction($vmId, $re_ac)
  
  
  ;~ Must use the AccessibleContext got from the win handle which include the target element
  Func _JAB_getAccessibleContextByRole($vmId, $ac, $sRole, $sIndex)
     Local $find_ac = 0
     Local $iCount =_JAB_getChildrenCount($vmId, $ac)
     If $iCount = 0 Then
        Return
     EndIf
     For $i = 0 To $iCount - 1
        Local $child_ac = __getAccessibleChildFromContext($vmId, $ac, $i)
        Local $s3 = _JAB_getRole($vmId, $child_ac)
  ;     consolewrite($child_ac & "|" & $s1 & "," & $s3 & @CRLF)
        If $s3 = $sRole Then
           $find_ac = $child_ac
           ExitLoop
        Else
           If $find_ac = 0 Then
              $find_ac = _JAB_getAccessibleContextByRole($vmId, $child_ac, $sRole, $sIndex)
           EndIf
        EndIf
     Next
     Return $find_ac
  EndFunc

   

 

@rlvitorino in above example another recursive function to find by role, by index is not there but I made a start in that function

 

[fablecao]

16 hours ago, rlvitorino said:

If you want to get a text by its index ID I would use which command?

Func _JAB_getName($vmId, $ac)
   Local $acInfo = DllStructCreate($tagAccessibleContextInfo)
   __getAccessibleContextInfo($vmId, $ac, $acInfo)
   Return DllStructGetData($acInfo, "name")
EndFunc

[junkew]

@rlvitorino

You mean logic like this?

ControlSend("Untitled - Notepad", "", "[CLASS:Edit; INSTANCE:1]", "This is some text")

[rlvitorino]

On 8/21/2019 at 1:58 PM, junkew said:

@rlvitorino

You mean logic like this?

ControlSend("Untitled - Notepad", "", "[CLASS:Edit; INSTANCE:1]", "This is some text")

yes, in this logic, for setvalue, getvalue, click´s and get tables by ID or Xpath. it would be possible?

[gobills]

@junkew  please give an  example on how to click on the  button  and setvalue for java applet within ie,thanks.

[fablecao]

setvalue example：

Func __setTextContents($vmId, $ac, $text)
    $result = DllCall($hAccessBridgeDll, "bool:cdecl", "setTextContents", "long", $vmId, $c_JOBJECT64, $ac, "wstr", $text)
    If @error Then Return SetError(1, 0, 0)
    Return $result[0]
EndFunc

Java access bridge native interface function __setTextContents can do this.

Due to some bug of early version of JRE, this native function takes no effect.  So I wrote an alternative function.

Func _JAB_setValue($vmId, $ac, $sValue)
   Local $actions = DllStructCreate($tagAccessibleActions)
   __getAccessibleActions($vmId, $ac, $actions)
   Local $s1 = DllStructGetData($actions, "actionsCount")
   If $s1 = 0 Then
      consolewrite("_JAB_setValue: this element has no action" & @CRLF)
      Return
   EndIf
   Local $re1 = 0
   Local $re2 = 0
   For $i = 2 To $s1 + 1
      Local $s = DllStructGetData($actions, $i)
      If $s = "select-all" Then
         $re1 = 1
         ExitLoop
      EndIf
   Next
   If $re1 = 0 Then
      consolewrite("_JAB_setValue: this element doesn't support select-all action" & @CRLF)
      Return
   EndIf
   For $i = 2 To $s1 + 1
      Local $s = DllStructGetData($actions, $i)
      If $s = "paste-from-clipboard" Then
         $re2 = 1
         ExitLoop
      EndIf
   Next
   If $re2 = 0 Then
      consolewrite("_JAB_setValue: this element doesn't support paste-from-clipboard action" & @CRLF)
      Return
   EndIf
   Local $actionsToDo = DllStructCreate($tagAccessibleActionsToDo)
   Local $failure
   ClipPut($sValue)
   DllStructSetData($actionsToDo, "actionsCount", 2)
   DllStructSetData($actionsToDo, 2, "select-all")
   DllStructSetData($actionsToDo, 3, "paste-from-clipboard")
   __doAccessibleActions($vmId, $ac, $actionsToDo, $failure)
EndFunc

Structure $actionsToDo, first member is actions count "2", second member is first action "select-all", third member is second action "paste-from-clipboard".

[fablecao]

;=====================================AccessibleAction================================
Global Const $MAX_ACTION_INFO = 256
Global Const $MAX_ACTIONS_TO_DO = 32

Global Const $tagAccessibleActionInfo = _
"WCHAR name[256]"

Local $tag_ActionInfo = ""
For $i = 1 To 256
   If $tag_ActionInfo = "" Then
      $tag_ActionInfo = $tagAccessibleActionInfo
   Else
      $tag_ActionInfo = $tag_ActionInfo & ";" & $tagAccessibleActionInfo
   EndIf
Next
Global Const $tagAccessibleActions = _
"INT actionsCount;" & _
$tag_ActionInfo                     ; $tAccessibleActionInfo actionInfo[256]

Local $tag_Actions = ""
For $i = 1 To 32
   If $tag_Actions = "" Then
      $tag_Actions = $tagAccessibleActionInfo
   Else
      $tag_Actions = $tag_Actions & ";" & $tagAccessibleActionInfo
   EndIf
Next
Global Const $tagAccessibleActionsToDo = _
"INT actionsCount;" & _
$tag_Actions                            ; $tAccessibleActions actions[32]

This is the usage of action in Java access bridge.

$tagAccessibleActionInfo is a Java element's action's name, 256 byte max.

$tagAccessibleActions is a structure of Java element's can-do actions, 256 members max, first member is total actions count.

$tagAccessibleActionsToDo is a structure of  Java element's to-do actions, 32 queue actions max, first member is total actionsToDo count.

You find actions from can-do list, and put them into to-do list, and execute native interface function.  $failure is used to put the execution result.

__doAccessibleActions($vmId, $ac, $actionsToDo, $failure)

Edited August 26, 2019 by fablecao
improve

[ChuckV]

Regarding JABSimpleSpy.au3 and gettiing AutoIT to interact with a JAVA application...

*Exactly how* is "JABSWITCH /ENABLE" set?

I am a java neophite and also not a java fan at large...

Thank you,
Chuck

EDIT: Answering my own question... found JABSWITCH.EXE and ran it with /ENABLE ... response was "bridge enabled"

Edited August 27, 2019 by ChuckV
answered my own question... and posted for others benefit

[sn1kzZ]

On 7/4/2019 at 11:19 AM, fablecao said:

global $wintitle = "Java Control Panel"

$winHandle = WinActivate($wintitle)

$result = __isJavaWindow($winHandle)

Global $vmId
Global $ac

global $sName = "Edit Site List..."
global $sRole = "push button"

__getAccessibleContextFromHWND($winHandle, $vmID, $ac)

$re_ac = _JAB_getAccessibleContextByFindAll($vmId, $ac, $sName, $sRole)

_JAB_singleAction($vmId, $re_ac)

Perfect example!
Now if you can demonstrate a more complete example, I'm sure we can all learn from it and @junkew can pin it aswell!

Here's what i think will cover usages:
1. click on "Security" tab

2. Click on "Edit Site List"

3. Click on "Add" button

4. setValue "https://google.com"

5. Click "OK" button to close window

6. Click "OK" button to close main Java window aswell

Edited September 6, 2019 by sn1kzZ

[mLipok]

You can do it directly by using this:

 

[sn1kzZ]

32 minutes ago, mLipok said:

You can do it directly by using this:

 

Are you replying to me? if yes then this isn't what were discussing here, javacpl.exe is just an example, but the intention of this thread is to INTERACT with Java applets.
Not Manage Java settings like your .au3 does.

[mLipok]

I thought you want to do only this specific action.
but in relation to automating JAVA - my experience says me that not all JAVA APP can be automated by UIA in so perfect way as in case standard WIN32 APP.
I was trying to use UIA but in some JAVA solution I need to calculate COORDINATES and use MOUSE CLICK

 

[junkew]

JAB and UIA are different technologies with different scope of object recognition but follow similar principles

• JAB
  Focused on JAVA technology: JAVA SWING, JAVA applets
• UIA
  Focused on Windows technology: Win32, WinForms, .... and (partly) HTML
  UIA can deal with some Java technology if that application has build in IAccessible interface
• AutoIt / Win32 sendmessage under the surface
  • AutoIt uses similar principles as UIA but UIA is more embedded in the Windows system and as such recognizes a little more
• Webdriver for HTML and more inner HTML visibility (if its just black box clicking, editing in browser UIA can deal with most important browser GUI)
• Hooking injecting dll's into application of interest
  • Mostly used in commercial testautomation tools to recognize harder to get controls (like Delphi apps, )
    Interested people could look into Deviare Hooking Engine.
     

The patterns they share

• Objects are put in a tree hierarchy
  • JAB starting from Java Window which is a child from Windows desktop
    • So this means you first need AutoIt Win* function or UIA function to find your Java Window in general then AutoIt function is easiest.
  • UIA starting at windows desktop
• Logic your program shouild follow
  • Find mainwindow
    • Scan childs with their properties whereas objecttype and title are most important but all properties can be of interest for identifying
    • Scan childs of childs (as its a hierarchy)
  • Use a spy tool to get initial insight in hierarchy and type of object
    https://github.com/google/access-bridge-explorer/releases/tag/v1.1.0 is a decent one (JABSimpleSpy is not showing hierarchy of tree)

So you should be applying above logic. Example will follow later 😉

 

 

 

[junkew]

Demo fully working

• some workarounds
  • click is done based on a x,y, h, w calculation
  • settext JAB functions I did not get to work so bypassed that with a normal send  keys function
  • some slowdown is done just with sleep function
  • seems some debugging is needed when we want to use specific JAB functions clicking on a button

 

1.  Issue 1 is tackled clicking on a security tab is done with __requestFocus (in code I still do a mousemove and mouseclick as it seems setfocus is not updating the gui in itself)

   __requestFocus($__g_vmId, $SecurityTab)

    

2. Popupwindow is a new windows window so based on HWND the ac should be retrieved.
   This works not in below code but when commenting out all lines with 'works ok '
    and just check the window and convert to ac it works so somehow clearing of variables should happen or speed should be slower
   No clue if tree can be refreshed (without (re)loading the whole JAB)

3.  

;~ see first post

 

Edited September 10, 2019 by junkew
9-Sep-2019: fixed all steps

[KhalidAnsari]

@junkew

Hi,

Sorry to post here as i was following this post.

 __isJavaWindow($winHandle) is returning 0 (false) for Java Control Panel. Ealier it was giving 1 (true) and I can able to click view button.

I am using javaui.au3 file is there any method change.

Thanks

[junkew]

Nothing changed. You should be seeing it with jabsimplespy.

[KhalidAnsari]

@junkew

I got it working. I just debug it. Checked my dll reference. 

Now i m tried on my jar file to click on button. I am getting following response in console.

shellexecute("D:\khalid\d\javaapplication2\chatserver\chatserver\MyClient.jar")
sleep(3000)
global $wintitle = "Login for Chat"
$winHandle = WinActivate($wintitle)
$result = __isJavaWindow($winHandle)
Global $vmId
Global $ac
global $sName = "Send"
global $sRole = "push button"
__getAccessibleContextFromHWND($winHandle, $vmID, $ac)
$re_ac=_JAB_getAccessibleContextByRole($vmId, $ac, $sRole, 1)
_JAB_singleAction($vmId, $re_ac)

Java Home: C:\Program Files\Java\jre1.8.0_221
  We are using OS X64 at cpu X64; Autoit 64 bit version @AutoItX64=0
  PASS: Windows accessbridge WindowsAccessBridge-32.dll opened returns: 1
0 Windows_run returns: 
Name:
rect:0000
Name:
rect:0000
_JAB_singleAction: Only applicable for single action element like button or check box, menu, menu item