JAVA accessibility bridge - respond to upfront unknown message

[junkew]

Hi,

In this thread

'?do=embed' frameborder='0' data-embedContent>>

we made a sidestep to accessibility for Java objects with UI Automation this is not possible but with the Java Accessibility bridge this should be possible
 

I have all examples working from JavaFerret, JavaMonkey on my system so Java Accessibility bridge is working but I am unable to get it working from AutoIT.

It seems I have to respond to a message that is in JAB registered with RegisterWindowMessagA after loading the dll or after making the call to function windows_run within the dll (assumption application is doing this with postmessageA)

Some kind of handshake with strings like

'AccessBridge-FromWindows-Hello' and 'AccessBridge-FromJava-Hello'

 

 

• But as I don't have a GUI (and at the moment I don't want that either) I have no clue how to see if a message is send that I should act upon and how to see if its arriving. The JAB creates a window with createdialog ands is doing a postmessaga

 

Below requires to have a JAVA application running, i used JEdit to simplify

expandcollapse popup

#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <constants.au3>
#include <WinAPI.au3>
#include <debug.au3>

local $hwnd
local $i
local $result
local $vmID
local $ac

local $bridgeDLL=DLLOPEN("WindowsAccessBridge-32.dll")
;~ sleep(500)
If $bridgeDLL=true Then
    consolewrite($bridgeDLL & @CRLF)
Else
    consolewrite("DLL not found try to change to WindowsAccessBridge-64.dll if you use 64 bits autoit and windows")
endIf

;~ TODO: Handle messages received from initialize

$result =dllcall($bridgeDLL,"int:cdecl", "Windows_run")
sleep(500)
consolewrite("Windows_run passed :" & $result & @CRLF)
sleep(500)

;~ TODO: Handle messages received from initialize

;~ Get a Java window like jEdit http://www.jedit.org/index.php?page=download
local $handle=wingethandle("jEdit")
$result =dllcall($bridgeDLL,"BOOL:cdecl", "isJavaWindow", "hwnd", $handle)
if @error <> 0 Then
    consolewrite("There is an Dll error: " & @error & @CRLF)
Else
    consolewrite("Handle for Java Window=" & $handle & @TAB & " res: " &  $result & @CRLF)
EndIf

local $result =dllcall($bridgeDLL,"BOOL", "shutdownAccessBridge")
Exit

Background

This thread mainly tells what has to happen

http://stackoverflow.com/questions/1161142/not-receiving-callbacks-from-the-java-access-bridge

Quoted from that thread

• The call to 'initializeAccessBridge' REQUIRES you to have an active windows message pump.

overtime there seems to have com in addition something like windows_run function

Inside 'initializeAccessBridge', it (eventually) creates a hidden dialog window (using CreateDialog). Once the dialog is created, it performs a PostMessage with a registered message. The JavaVM side of the access bridge responds to this message, and posts back another message to the dialog that was created (it appears to be a 'hello' type handshake between your app and the java VM). As such, if your application doesn't have an active message pump, the return message from the JavaVM never gets received by your app.

This is important as until this message is received, the bridge is never properly initialized and as such all calls to 'IsJavaWindow' fail (internally, the bridge initializes an internal structure once the message is received -- as such, no active message pump, no initializing). I'm guessing that this is why you never receive callback messages as well.

Not only that, but you must call initializeAccessBridge at a point where the message pump can process messages before you can call IsJavaWindow.

This is why JavaFerret and JavaMonkey work -- they initialize at startup, and then enumerate on response to a menu message, well after the bridge has received the initialization message via the message pump.

The way I was able to solve this in my MFC dialog app (and our MFC-based application), is to make sure that you call 'initializeAccessBridge' at a point such that the built-in MFC message pump can push the 'hello' message back to this hidden dialog BEFORE you use it. In the simple MFC dialog case, it meant calling initializeAccessBridge in OnInitDialog, and calling the enum procedure in response to a button call (for example). If you want the enum to occur as soon as the dialog appears, you could use a timer to fire (eg. 10ms) after the OnInitDialog completes to allow processing of the initialization message.

If you are planning to use this in a console app, you will need to write your own custom message pump to handle the initialization message.

Anyway, I hope this is clear enough! Whilst there is no way to know whether this is the 'correct' way (other than to pay for a Sun engineer to tell us), it definitely solved my problem.

[junkew]

Weird enough now this seems to be working without any further change

 

Only thing I only have run once on my system ChangeWindowMessageFilter (with value 1)

and then tried to break it again with value 2 

 

1. start a javaprogram for example jEdit

2. run the script and it will show you the java windows that are visible and open to JAB

3. TODO: Dump the Userinterface tree of the java widgets

;~ Make messages elevated
for $i=$WM_USER to $WM_USER+65536
_ChangeWindowMessageFilter($i, 1)
Next


Func _ChangeWindowMessageFilter($iMsg, $iAction)
    Local $aCall = DllCall("user32.dll", "bool", "ChangeWindowMessageFilter", "dword", $iMsg, "dword", $iAction)
    If @error Or Not $aCall[0] Then Return SetError(1, 0, 0)
    Return 1
EndFunc

expandcollapse popup

;~ By inspecting the WindowsAccessBridge-32.dll it reveals some information about the hidden dialogs
;~   So it seems the hidden dialog is shown after you call windows_run() no clue if interaction is needed
;~
;~ Somehow it sends a message unclear if this is to the JVM to respond to
;~   push   SSZ6E73E320_AccessBridge_FromJava_Hello
;~   push   SSZ6E73E300_AccessBridge_FromWindows_Hello
;~   db 'AccessBridge-FromWindows-Hello',0
;~   db 'AccessBridge-FromJava-Hello',0

;~ Copied from this NVDA reference and translated to AutoIT
;~ http://www.webbie.org.uk/nvda/api/JABHandler-pysrc.html
;~
;~ def initialize():
;~         global isRunning
;~         if not bridgeDll:
;~                 raise NotImplementedError("dll not available")
;~         bridgeDll.Windows_run()
;~         #Accept wm_copydata and any wm_user messages from other processes even if running with higher privilages
;~***         ChangeWindowMessageFilter=getattr(windll.user32,'ChangeWindowMessageFilter',None)
;~***         if ChangeWindowMessageFilter:
;~***                 if not ChangeWindowMessageFilter(winUser.WM_COPYDATA,1):
;~***                         raise WinError()
;~***                 for msg in xrange(winUser.WM_USER+1,65535):
;~***                         if not ChangeWindowMessageFilter(msg,1):
;~***                                 raise WinError()
;~         #Register java events
;~         bridgeDll.setFocusGainedFP(internal_event_focusGained)
;~         bridgeDll.setPropertyActiveDescendentChangeFP(internal_event_activeDescendantChange)
;~         bridgeDll.setPropertyNameChangeFP(event_nameChange)
;~         bridgeDll.setPropertyDescriptionChangeFP(event_descriptionChange)
;~         bridgeDll.setPropertyValueChangeFP(event_valueChange)
;~         bridgeDll.setPropertyStateChangeFP(internal_event_stateChange)
;~         bridgeDll.setPropertyCaretChangeFP(internal_event_caretChange)
;~         isRunning=True

;~ #AutoIt3Wrapper_UseX64=Y

#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <constants.au3>
#include <WinAPI.au3>
#include <debug.au3>

local $hwnd
local $i
local $result
local $vmID
local $ac

;~ Make messages elevated
_ChangeWindowMessageFilter($WM_COPYDATA,1)
for $i=$WM_USER to $WM_USER+65536
    _ChangeWindowMessageFilter($i, 1)
Next

Func _ChangeWindowMessageFilter($iMsg, $iAction)
    Local $aCall = DllCall("user32.dll", "bool", "ChangeWindowMessageFilter", "dword", $iMsg, "dword", $iAction)
    If @error Or Not $aCall[0] Then Return SetError(1, 0, 0)
    Return 1
EndFunc

local $bridgeDLL=DLLOPEN("WindowsAccessBridge-32.dll")
;~ sleep(500)
If $bridgeDLL=true Then
    consolewrite($bridgeDLL & @CRLF)
Else
    consolewrite("DLL not found try to change to WindowsAccessBridge-64.dll if you use 64 bits autoit and windows")
endIf

;~ TODO: Handle messages received from initialize

$result =dllcall($bridgeDLL,"int:cdecl", "Windows_run")
consolewrite($result & " " & @error & " initializeAccessBridge is finished")

sleep(250)

consolewrite("Windows_run passed :" & $result & @CRLF)

Local $var = WinList()
consolewrite("Before loading all Windows:" & $var[0][0] & @CRLF)

For $i = 1 To $var[0][0]
; Only display visble windows that have a title
    If IsVisible($var[$i][1]) Then
        local $handle=wingethandle($var[$i][0])
        $result =dllcall($bridgeDLL,"BOOL:cdecl", "isJavaWindow", "hwnd", $handle)
        if @error=0 Then
            if $result[0]=1 Then
                consolewrite( $i & " Java Window Title=" & $var[$i][0] &  " Handle=" & $var[$i][1] & @TAB & " res: " &  $result[0] & @CRLF)

            local $ac=0
            local $vmID=0

            $result =dllcall($bridgeDLL,"BOOL:cdecl", "getAccessibleContextFromHWND", "hwnd", $handle, "long*", $vmID, "ptr*", $ac)

                if @error=0 Then
                    consolewrite("We have a context " & @CRLF)
                    $vmID=$result[2]
                    $ac=$result[3]
                    ;create the struct
;~                  http://docs.oracle.com/javase/accessbridge/2.0.2/api.htm
;~ SHORT_STRING_SIZE    256
;~                  struct AccessBridgeVersionInfo {
;~  wchar_t VMversion[SHORT_STRING_SIZE];              // version of the Java VM
;~  wchar_t bridgeJavaClassVersion[SHORT_STRING_SIZE]; // version of the AccessBridge.class
;~  wchar_t bridgeJavaDLLVersion[SHORT_STRING_SIZE];   // version of JavaAccessBridge.dll
;~  wchar_t bridgeWinDLLVersion[SHORT_STRING_SIZE];    // version of WindowsAccessBridge.dll
;~ };
                    Local $AccessBridgeVersionInfo=DllStructCreate("WCHAR VMversion[256];WCHAR bridgeJavaClassVersion[256];WCHAR bridgeJavaDLLVersion[256];WCHAR bridgeWinDLLVersion[256]")
;~                  Local $AccessBridgeVersionInfo=DllStructCreate("wchar[1024]")
                    if @error > 0 then consolewrite("Struct error")

;~                  $result =dllcall($bridgeDLL, "BOOL:cdecl", "getVersionInfo", "long", $vmId, "struct", $AccessBridgeVersionInfo)
                    $result =dllcall($bridgeDLL, "BOOL:cdecl", "getVersionInfo", "long", $vmId, "ptr", DllStructGetPtr($AccessBridgeVersionInfo))
                    consolewrite( @error & " context found of " & $vmID & @CRLF)
                    if @error=0 Then
;~                      for $i=1 to 2048
;~                          consolewrite(hex(dllstructgetdata($AccessBridgeVersionInfo,$i,1)))
;~                      next
                        $s1=dllstructgetdata($AccessBridgeVersionInfo, "VMVersion")
                        $s2=dllstructgetdata($AccessBridgeVersionInfo, "bridgeJavaClassVersion")
                        $s3=dllstructgetdata($AccessBridgeVersionInfo, "bridgeJavaDLLVersion")
                        $s4=dllstructgetdata($AccessBridgeVersionInfo, "bridgeWinDLLVersion")

                        consolewrite("VMVersion: <" & $s1 & ">" & @CRLF)
                        consolewrite("bridgeJavaClassVersion: <" & $s2 & ">" & @CRLF)
                        consolewrite("bridgeJavaDLLVersion: <" & $s3 & ">" & @CRLF)
                        consolewrite("bridgeWinDLLVersion: <" & $s4 & ">" & @CRLF)

                    EndIf
                Else
                    consolewrite( @error & " No context found" & @CRLF)
                endif
            EndIf
        Else
        EndIf
    endif
Next
;~ http://www.autohotkey.com/board/topic/95343-how-to-send-to-unseen-controls-in-a-java-app/
local $result =dllcall($bridgeDLL,"BOOL", "shutdownAccessBridge")

Func IsVisible($handle)
    If BitAND(WinGetState($handle), 2) Then
        Return 1
    Else
        Return 0
    EndIf

EndFunc   ;==>IsVisible

Edited September 3, 2013 by junkew