How to automate Java windows/controls?

[Jon]

So I had a look at a couple of Java programs. Notably the Java Control Panel that most people will have. And it's like a total black box. Using Window Spy++ I could see the main window but even though the window was full of buttons and checkboxes and stuff - there were no child windows!

I thought automating Java controls would be a case of finding out which messages it accepts to do what - but if it doesn't even present a child window to send to I'm a bit stumped. Spy++ couldn't even see any unknown/registered messages when you click on things.

Is there a Java SDK type thing for programming these sorts of apps - that might shown how they are control internally and help out. I mean, there must be an internal command for selecting a checkbox.

Ideas welcome.

[jvanegmond]

Here's what the professionals say about Java Applets.

AutoMate is able to find the Java Applet window but recognizes nothing inside it. The Java controls are basically "invisible" to AutoMate. From Windows' point of view the Java Applet looks like a window containing one large bitmap. Because of this, most of AutoMate's interactivity actions (i.e. 'Move Mouse to Object,' 'Press,' or 'Set Text' actions) will not work within a Java Applet window.

Solutions

The only possibility for AutoMate to interact with controls within a Java Applet Window is to set the focus to the Java Window and use the 'Send Keystrokes' action to tab through the fields to set and get data or press buttons.

http://www.networkautomation.com/automate/...23252F9A5553FE0

Edit: What Larry said completely contradicts this information.

Edited March 16, 2007 by Manadar

[Jon]

Not quite. Junit looks to be a general java testing thing. But you need to use some addins to test gui stuff through a awk.robot java class. This class can mess around with a few buttons but appears to have no c++/external interface - so it's of no use

[jvanegmond]

I've seen a few programs that decompile the .class files, that could at least give you information about the Java applet. This probably should not be considered an option...

My guess is that if you should succeed, Jon, you would be the first ever.

[big_daddy]

By the sounds of it this may be worth taking a look at.

IBM Rational Functional Tester

Edited March 16, 2007 by big_daddy

[TimothyGirard]

What if we were able to capture user defined images and then OCR any text. Most text within these UI's (Flash and Java) is usually in standard fonts.

this would be a very powerfull ability for AutoIt. The ability to define a "virtual" object that could be manipulated via standard functions would no doubt push a large group of app automators (Like me) to AutoIt a lot faster. Until I can do that easily in AutoIT, I'm stuck in QuickTest Pro.

[DaleHohm]

I have always been on the lookout for such tools and methods, but had never found anything other than trying to exploit methods that might be exposed by the applet to Javascript.

I did recently find a couple of resources dedicated to discussions of this but I have not had a chance to dig into them to see if there is anything promising.

This Wiki has a bunch of tool pointers: TestingGUIApplications and this Yahoo! group: java-gui-testing · Java GUI Testing

Dale

Edit: also seeing the source may help you... Google the DJ Java Decompiler

Edited May 4, 2007 by DaleHohm

[b990595]

I would be nice to read java windows, maybe this can help ...

Sun have developed a Java Access Bridge, which contains DLLs to read Java windows ..

http://java.sun.com/products/accessbridge/

API documentation is here ..

http://java.sun.com/products/accessbridge/docs/api.html

JavaFerret is a simple app, that reads java screens ...

http://java.sun.com/products/accessbridge/...javaferret.html

-Rasmus, Denmark

[Xenobiologist]

Hi,

any progress on this?

We use this : IBM Rational Functional Tester

But it is written in Java.

So long,

Mega

[MadMike]

Sorry I'm not an experienced developer just yet but I'm very interested if anyone has made any progress on this as well.

I've lookep up a load of how to automate Java windows/controls and this Java Access Bridge seems the best way.

I've been trying to use the Java Access Bridge and the Java Ferret and I'm not having much luck.

Just have to use DllCall to the WindowsAccessBridge.dll with the Java Access Bridge then I can start making calls to the classes within the program.

[ValeryVal]

I added ALisp

here

Apart from other things you can create by newLisp's tools java-based GUI.

For doing it newLisp has special guiserver.jar (GUI server) and guiserver.lsp.

Creation of java-based gui is the following process:

1. Start GuiServer demon

java.exe -jar "c:/Program Files/newlisp/guiserver.jar" 47011

You will see the demon's window worked (black window)

2. Start ALisp.au3

3. Evaluate by ALisp the expression:

(load "c:/Program Files/newlisp/guiserver.lsp")

4. Now you can evaluate lines (step by step or all as one) of some example from newlisp/guiserver folder.

ie button-demo.lsp

(gs:init)

(gs:set-trace true)

(gs:frame 'ButtonDemo 100 100 400 300 "Click on button or color panel")

(gs:set-resizable 'ButtonDemo nil)

(gs:panel 'ColorPanel 360 200)

(gs:set-background 'ColorPanel '(0 1 0) 0.2)

(gs:button 'aButton 'button-action "color")

(gs:set-flow-layout 'ButtonDemo "center" 2 15)

(gs:add-to 'ButtonDemo 'ColorPanel 'aButton)

(gs:set-visible 'ButtonDemo true)

(gs:mouse-event 'ColorPanel 'mouse-action)

;;;; define actions

(define (button-action id)

(gs:set-color 'ColorPanel (random) (random) (random)))

(define (mouse-action id type x y button cnt mods)

(gs:set-text 'ButtonDemo (format "%8s %3d:%3d %d %d %2d" type x y button cnt mods))

)

;(gs:listen)

Thus, you will create really java-based GUI having two handlers - button-action and mouse-action.

If you uncomment last line then GUI will independent window which handles button and mouse events.

This is a two-side body - ALisp & java window.

Maybe it is simplest platform for experiment with interaction between AutoIt and java, I think.

Remark. You should to install newlisp from this site, I think

BTW. ALisp (with help newlisp) shows how you can build simplest java-based bridge from Windows into Linux/Mac OS X/Solaris/BSD's platform

[ValeryVal]

Feed back from java-based Gui can be implemented by AutoItX.

See, example of AutoItX using (not tested)

here

However there are other more fast method, I think

;-)

[Antiec]

Hi!

I had to automate a Java program at my work. I found two kinda nice java-based automation programs (marathon & abbot), but marathon wouldn't work with our huge program (it did work with some smaller applications) and I couldn't really do what I wanted with abbot. So I found the already mentioned Java Access Bridge. I didn't/don't know how to use it with autoit, but I managed to automate our program with JAB & C++, mostly by customizing Java Ferret though. I know a bit about how JAB works and I might be able to answer some questions if you have any. And maybe I'll try to use it with autoit too (I'd really love it).

Though... Marathon seems to be pretty nice program for the job. And I couldn't find a way to use button IDs (like name etc) straight, but I had to run the parent/child tree's (there's java monkey for finding the right routes). And somehow my program stalls when there's a new dialog window, but that might not happen with a new code.

Yours,

Lassi

Edit: And actually I didn't use the DLLs, but the source code. Anyways, atleast I think I have somekind of image how the calls should work... gotta test it.

Edited September 14, 2007 by Antiec

[Rocc]

Perhaps this web site might help.

http://jacareto.sourceforge.net/

[junkew]

And maybe when somebody comes around to read

http://java.sun.com/docs/books/jni/

it can be programmed in C (and with samples it looks like it could be done in AutoIT) but definitely is some effort.

[cbadal]

For automating Java you would really need something that understands Java.

Java swing controls are all drawn in Java drawing functions (these functions eventually call native platform dependent drawing functions).

This makes it nice for Java because they only need to write the drawing code once (and only update the low level platform dependent drawing functions in the actual JVM code).

This also means that their controls don't adhere to the Windows API (except for perhaps a top level Window, or some container).

The very old Java AWT toolkit uses native controls and if you came across a Java GUI that was written in AWT, you may be able to automate it with AutoIt.

[daluu]

What about:

The free FEST framework/tool for Java? Though it's more developer & unit test centric. Works for Java applets too. Not sure how one can use it with AutoIt though.

http://code.google.com/p/fest/

http://fest.easytesting.org/swing/wiki/pmw...ESTAppletViewer

[Xenobiologist]

Hi,

another tool which works quiet well is Loadrunner from HP.

Mega

[bo8ster]

Thanks guys for all the links, really helpful!

[vixLT6B]

I am not sure, but look at this: [link removed]

Edited March 19, 2009 by SmOke_N