JAVA object automation and simple spy

[junkew]

I have no time to help much at the moment. You are having 0 for the rect. So not finding right ac. You find by role and not by title and role. Be aware this is a work in progress and many issues can arise on testing this stuff.

[junkew]

https://github.com/openjdk/jdk/tree/master/src/jdk.accessibility/windows/native/jaccessinspector gives an indication that at least __releaseJavaObject should be called to release memory allocated.

More CPP source on how JAB is made https://github.com/openjdk/jdk/blob/master/src/jdk.accessibility/windows/native/libwindowsaccessbridge/WinAccessBridge.cpp

and apparently I tried also in 2013 some stuff where most important part can be that certain messages have to be handled when sent from JAB to AutoIt client.

• jaccesswalker from jdk10 and jdk11 seems to be most stable tool to spy not crashing
• whereas jaccessinspector just quits when i try to spy on jdk8 javacpl.exe.
• accessibility explorer is also crashing as soon as I spy on the modal dialog.

As jaccesswalker is working fine I only can say this can work but maybe issues due to

* Logic coding error

* separate thread needed

* ....

 

Edited September 9, 2019 by junkew

[junkew]

Demo script updated and fully working (see post 1)

• _JAB_SingleAction seems to hang on sitelistbutton, bypassed by calculating mouse location based on acinfo x,y,h,w

;~  TODO: _JAB_singleAction($__g_vmId, $SiteListButton)
    clickac($__g_vmId, $SiteListButton)

• button add in popup works
• Done: Edit part

Edited September 10, 2019 by junkew

[fablecao]

16 hours ago, KhalidAnsari said:

@junkew

I got it working. I just debug it. Checked my dll reference. 

Now i m tried on my jar file to click on button. I am getting following response in console.

shellexecute("D:\khalid\d\javaapplication2\chatserver\chatserver\MyClient.jar")
sleep(3000)
global $wintitle = "Login for Chat"
$winHandle = WinActivate($wintitle)
$result = __isJavaWindow($winHandle)
Global $vmId
Global $ac
global $sName = "Send"
global $sRole = "push button"
__getAccessibleContextFromHWND($winHandle, $vmID, $ac)
$re_ac=_JAB_getAccessibleContextByRole($vmId, $ac, $sRole, 1)
_JAB_singleAction($vmId, $re_ac)

Java Home: C:\Program Files\Java\jre1.8.0_221
  We are using OS X64 at cpu X64; Autoit 64 bit version @AutoItX64=0
  PASS: Windows accessbridge WindowsAccessBridge-32.dll opened returns: 1
0 Windows_run returns: 
Name:
rect:0000
Name:
rect:0000
_JAB_singleAction: Only applicable for single action element like button or check box, menu, menu item

 

_JAB_getAccessibleContextByRole($vmId, $ac, $sRole, 1)

This function is not my original, you find wrong $re_ac which cause _JAB_singleAction error.

You can use  _JAB_getAccessibleContextByFindAll($vmId, $ac, $sName, $sRole)

[junkew]

Ok will check with the original version.

added function _JAB_getAccessibleChildNByRole

usage: get an element by its index and class/role 

;~  local $re_ac=_JAB_getAccessibleContextByRole($__g_vmId, $__g_acJavaMainWindow, "push button", 1)
    local $re_ac=_JAB_getAccessibleChildNByRole($__g_vmId, $__g_acJavaMainWindow, "push button",1)
    local $re_ac=_JAB_getAccessibleChildNByRole($__g_vmId, $__g_acJavaMainWindow, "push button",2)
    local $re_ac=_JAB_getAccessibleChildNByRole($__g_vmId, $__g_acJavaMainWindow, "push button",3)

function

expandcollapse popup

Func _JAB_getAccessibleChildNByRole($vmId, $ac, $sRole,$index=1, $depth=0)
   Static $matchCount=0

   if ($depth=0) then
       $matchCount=0
   EndIf

   Local $iCount = _JAB_getChildrenCount($vmId, $ac)
;~    consolewrite("Investigating" & $iCount & ";" & $index & ";" & $matchCount & ";" & _JAB_getRole($vmId, $ac) & ";")

    If $iCount = 0 Then
        Return -1
    EndIf

    local $retVal=-1

    For $i = 0 To $iCount - 1
        Local $child_ac = __getAccessibleChildFromContext($vmId, $ac, $i)
        Local $child_ac_s3 = _JAB_getRole($vmId, $child_ac)

;~      consolewrite($child_ac_s3 & @CRLF)

        If $child_ac_s3 = $sRole Then
            $matchCount+=1
;~          consolewrite("Partial match" & $matchCount & @CRLF)
            if $index=$matchCount Then
;~              consolewrite("Full match" & $matchCount & @CRLF)
                Return $child_ac
            endif
        EndIf

        $depth+=1
        local $retVal=_JAB_getAccessibleChildNByRole($vmId, $child_ac, $sRole, $index, $depth)

        if ($retVal <>-1) Then
            exitloop
        EndIf
   Next

   Return $retVal
EndFunc

 

Edited September 10, 2019 by junkew
function changed/added

[junkew]

@fablecao

I am 99.99% sure I have the right AC but as soon as I use _JAB_SingleAction on the sitelist button it behaves weird thats why I made clickAC function

See demo example (that fully works and shows the JAB capabilities, fast and really nice to operate on Java applications) in first post this function

func clickSiteListButton()
    $SiteListButton= _JAB_getAccessibleContextByFindAll($__g_vmId, $__g_acJavaMainWindow, "Edit Site List...", "push button")
    highLightAC($__g_vmId, $SiteListButton)
    printdescription($SiteListButton)
    sleep(2500)

;~ TODO: Somewhere here it hangs
;~ consolewrite("Just before it causes a blocking ..." & @CRLF)
;~ _JAB_singleAction($__g_vmId, $SiteListButton)
;~ consolewrite("Just after it causes a blocking ..." & @CRLF)

    clickac($__g_vmId, $SiteListButton)
    sleep(1500)

    __releaseJavaObject($__g_vmId, $SiteListButton)
;~  consolewrite("Just after it causes a blocking ... 2" & @CRLF)
EndFunc

Output from printDescription($siteListButton)

Name:Edit Site List...
rect:10681209562585
  name: <Edit Site List...>
  description: <Add, Remove, or Edit entries on the Exception Site List>
  role: <push button>
  role_en_US: <push button>
  states: <enabled,focusable,visible,showing,opaque>
  states_en_US: <enabled,focusable,visible,showing,opaque>
  indexInParent: <0>
  childrenCount: <0>
  x: <1068>
  y: <562>
  width: <141>
  height: <23>
  accessibleComponent: <1>
  accessibleAction: <1>
  name: <0>
  description: <0>
  role: <0>
  role_en_US: <0>
  states: <0>
  states_en_US: <0>
  indexInParent: <0>
  childrenCount: <0>
  x: <0>
  y: <0>
  width: <0>
  height: <0>
  accessibleComponent: <0>
  accessibleAction: <0>

 

[fablecao]

3 hours ago, junkew said:

@fablecao

I am 99.99% sure I have the right AC but as soon as I use _JAB_SingleAction on the sitelist button it behaves weird thats why I made clickAC function

See demo example (that fully works and shows the JAB capabilities, fast and really nice to operate on Java applications) in first post this function

func clickSiteListButton()
    $SiteListButton= _JAB_getAccessibleContextByFindAll($__g_vmId, $__g_acJavaMainWindow, "Edit Site List...", "push button")
    highLightAC($__g_vmId, $SiteListButton)
    printdescription($SiteListButton)
    sleep(2500)

;~ TODO: Somewhere here it hangs
;~ consolewrite("Just before it causes a blocking ..." & @CRLF)
;~ _JAB_singleAction($__g_vmId, $SiteListButton)
;~ consolewrite("Just after it causes a blocking ..." & @CRLF)

    clickac($__g_vmId, $SiteListButton)
    sleep(1500)

    __releaseJavaObject($__g_vmId, $SiteListButton)
;~  consolewrite("Just after it causes a blocking ... 2" & @CRLF)
EndFunc

Output from printDescription($siteListButton)

Name:Edit Site List...
rect:10681209562585
  name: <Edit Site List...>
  description: <Add, Remove, or Edit entries on the Exception Site List>
  role: <push button>
  role_en_US: <push button>
  states: <enabled,focusable,visible,showing,opaque>
  states_en_US: <enabled,focusable,visible,showing,opaque>
  indexInParent: <0>
  childrenCount: <0>
  x: <1068>
  y: <562>
  width: <141>
  height: <23>
  accessibleComponent: <1>
  accessibleAction: <1>
  name: <0>
  description: <0>
  role: <0>
  role_en_US: <0>
  states: <0>
  states_en_US: <0>
  indexInParent: <0>
  childrenCount: <0>
  x: <0>
  y: <0>
  width: <0>
  height: <0>
  accessibleComponent: <0>
  accessibleAction: <0>

 

I don't know what behaves weird when you use _JAB_SingleAction on the sitelist button.  I test it successfully in my computer.

And from your output of printDescription($siteListButton), I find that the output of printdescription($siteListButton) inside highlightAC is different from the next output of printdescription($siteListButton).

Does the func clickSiteButton function normal?

Does the func __releaseJavaObject inside highlightAC release the AC?

[fablecao]

By the way, I prefer to use the combination of JAB native function to write UDF.

The JAB native function __requestFocus may implement your function highlightAC.

;~ BOOL requestFocus(const long vmID, const AccessibleContext accessibleContext)
; Request focus for a component. Returns whether successful.
Func __requestFocus($vmId, $ac)
    $result = DllCall($hAccessBridgeDll, "bool:cdecl", "requestFocus", "long", $vmId, $c_JOBJECT64, $ac)
    If @error Then Return SetError(1, 0, 0)
    Return $result[0]
EndFunc

[junkew]

I agree on preferring native functions over alternatives like mousemove and click .

the problem I observe

Function clicks on the button

Popup dialogwindow appears

Script hangs

When I use this below function so action is executed but then script is hanging somewhere 

func clickSiteListButton()
    $SiteListButton= _JAB_getAccessibleContextByFindAll($__g_vmId, $__g_acJavaMainWindow, "Edit Site List...", "push button")
    highLightAC($__g_vmId, $SiteListButton)
    printdescription($SiteListButton)
    sleep(2500)

;~ TODO: Somewhere here it hangs
 consolewrite("Just before it causes a blocking ..." & @CRLF)
_JAB_singleAction($__g_vmId, $SiteListButton)
consolewrite("Just after it causes a blocking ..." & @CRLF)

;~    clickac($__g_vmId, $SiteListButton)
    sleep(1500)

    __releaseJavaObject($__g_vmId, $SiteListButton)
;~  consolewrite("Just after it causes a blocking ... 2" & @CRLF)
EndFunc

Edited September 11, 2019 by junkew

[mLipok]

53 minutes ago, junkew said:

When I use this below function so action is executed but then script is hanging somewhere 

Maybe the same problem like in IE.au3 UDF , I mean when you click in "OpenFile Button".
IE object is waiting for end of selecting..... and for this reason, AutoIt is waiting for IE......

 

[KhalidAnsari]

@junkew

Thanks for reply. 

Following code is closing my Java screen unexpectedly. I am trying to click on "Send button" but $re_ac returning -1 from method _JAB_getAccessibleChildNByRole

#include "Javaui.au3"
shellexecute("D:\khalid\d\javaapplication2\chatserver\chatserver\MyClient.jar")
sleep(5000)
global $wintitle = "Login for Chat"
$winHandle = WinActivate($wintitle)
$result = __isJavaWindow($winHandle)
Global $vmId
Global $ac
__getAccessibleContextFromHWND($winHandle, $vmID, $ac)
ConsoleWrite("$vmID : " & $vmID & @CRLF)
ConsoleWrite("$ac : " & $ac & @CRLF)
Sleep(3000)
local $re_ac =_JAB_getAccessibleChildNByRole($vmId, $ac, "push button",1)
ConsoleWrite("$re_ac : " & $re_ac &  @CRLF)

OutPut

Java Home: C:\Program Files\Java\jre1.8.0_221
 We are using OS X64 at cpu X64; Autoit 64 bit version @AutoItX64=0
C:\Program Files\Java\jre1.8.0_221\bin\WindowsAccessBridge-32.dll  PASS: Windows accessbridge WindowsAccessBridge-32.dll opened returns: 1
0 Windows_run returns: 
$vmID : 1247780
$ac : 1627349361864
Investigating1;1;0;frame; -- Not getting out from _JAB_getAccessibleChildNByRole
Investigating0;1;0;; -- Not getting out from _JAB_getAccessibleChildNByRole
$re_ac : -1

 

[fablecao]

5 hours ago, junkew said:

I agree on preferring native functions over alternatives like mousemove and click .

the problem I observe

Function clicks on the button

Popup dialogwindow appears

Script hangs

When I use this below function so action is executed but then script is hanging somewhere 

func clickSiteListButton()
    $SiteListButton= _JAB_getAccessibleContextByFindAll($__g_vmId, $__g_acJavaMainWindow, "Edit Site List...", "push button")
    highLightAC($__g_vmId, $SiteListButton)
    printdescription($SiteListButton)
    sleep(2500)

;~ TODO: Somewhere here it hangs
 consolewrite("Just before it causes a blocking ..." & @CRLF)
_JAB_singleAction($__g_vmId, $SiteListButton)
consolewrite("Just after it causes a blocking ..." & @CRLF)

;~    clickac($__g_vmId, $SiteListButton)
    sleep(1500)

    __releaseJavaObject($__g_vmId, $SiteListButton)
;~  consolewrite("Just after it causes a blocking ... 2" & @CRLF)
EndFunc

Oh, maybe there is something wrong in my JAB native function.  "NONE" should be changed to "NONE:cdecl".  All of which should be corrected.

Func __releaseJavaObject($vmId, $object)
    $result = DllCall($hAccessBridgeDll, "NONE", "releaseJavaObject", "long", $vmId, $c_JOBJECT64, $object)
    If @error Then Return SetError(1, 0, 0)
    Return $result[0]
EndFunc

[fablecao]

On 11/24/2018 at 3:08 AM, junkew said:

Just checked and still works nicely in 64 bits windows. 

>"C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.exe" /run /prod /ErrorStdOut /in "C:\Users\x\Documents\UIA\JABSimpleSpy.au3" /UserParams    
+>19:58:07 Starting AutoIt3Wrapper v.14.801.2025.0 SciTE v.3.4.4.0   Keyboard:00020409  OS:WIN_81/  CPU:X64 OS:X64    Environment(Language:0413)
+>         SciTEDir => C:\Program Files (x86)\AutoIt3\SciTE   UserDir => C:\Users\x\AppData\Local\AutoIt v3\SciTE\AutoIt3Wrapper   SCITE_USERHOME => C:\Users\x\AppData\Local\AutoIt v3\SciTE 
>Running AU3Check (3.3.14.5)  from:C:\Program Files (x86)\AutoIt3  input:C:\Users\x\Documents\UIA\JABSimpleSpy.au3
+>19:58:08 AU3Check ended.rc:0
>Running:(3.3.14.5):C:\Program Files (x86)\AutoIt3\autoit3_x64.exe "C:\Users\x\Documents\UIA\JABSimpleSpy.au3"    
--> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop
C:\Program Files\Java\jre1.8.0_181
We are using X64 at cpu X64 Autoit 64 bit version @AutoItX64=1
Opening C:\Program Files\Java\jre1.8.0_181\bin\WindowsAccessBridge-64.dll  Windows accessbridge 32 opened 1
 0 initializeAccessBridge is finishedWindows_run passed :
0hello  name: <Settings...>
  description: <<html>Modify settings for temporary files</html>>
  role: <push button>
  role_en_US: <push button>
  states: <enabled,focusable,visible,showing,opaque>
  states_en_US: <enabled,focusable,visible,showing,opaque>
  indexInParent: <0>
  childrenCount: <0>
  x: <1028>
  y: <447>
  width: <83>
  height: <0>
+>19:58:30 AutoIt3.exe ended.rc:0
+>19:58:30 AutoIt3Wrapper Finished.

Initially I learned a lot from NVDA to use this stuff and found another nice access-bridge-explorer that can be used for porting things from JAB into AU3

• https://github.com/nvaccess/nvda/blob/master/source/JABHandler.py
• https://github.com/google/access-bridge-explorer

 

How to use autoit3_x64.exe in SciTE?

[fablecao]

#AutoIt3Wrapper_UseX64 = y

[fablecao]

10 hours ago, fablecao said:

#AutoIt3Wrapper_UseX64 = y

Oh, my God.  It's the bug of SciTE-Lite, this setting takes no effect in SciTE-Lite.

[fablecao]

11 hours ago, fablecao said:

#AutoIt3Wrapper_UseX64 = y

It's a pity that SciTE-Lite doesn't support autoit3wrapper.keywords.properties and other properties.

[Mich90]

I finally got a simple Java automatisation working 🙂

Inow have got the problem that I need to select the first row of a table. Is this possible?
Thanks for any hint!

[junkew]

It depends. Just chech what different tools highlight for you when you spy.  It basically starts with below function.

getAccessibleTableInfo

[duzers]

Hello,

It is possible to click Page tab e.g. "Java" or radio button in Security tab in Java Control Panel.

[junkew]

Yes it is you can see jabsimplespy highlights it. You have to study java accessibility code examples to understand the concept of what and how you can retrieve the accessible objects.