Java UDF

[seangriffin]

This UDF provides automation support for Java applications.

This UDF currently automates standalone applications and browser-based applets, developed in either Swing or the Abstract Window Toolkit (AWT).

REQUIREMENTS:

• AutoIt3 3.2 or higher
• Java Runtime Environment (JRE) 1.4 or above
• Java Access Bridge for the Microsoft Windows Operating System
• Microsoft Visual C++ 2008 Redistributable
• AutoIT for Java Access dynamically linked library (DLL)
• Tesseract UDF 0.6
• Manual check and update of the Java Access Bridge

INSTALLATION:

To install the latest Java Runtime Environment (JRE):

• Open the web page http://java.com/en/download/manual.jsp.
• Click Windows XP/Vista/2000/2003 Offline.
• Save the file to your computer and run it (ie. jre-6u11-windows-i586-p-s.exe).
• Follow the installation instructions.
• Restart your computer when prompted.

To install the Java Access Bridge for the Microsoft Windows Operating System:

• Open the web page Java Access Bridge.
• Click Download Java Access Bridge 2.0.1.
• Click I agree to the Software License Agreement.
• Click Continue.
• Click accessbridge-2_0_1.exe.
• Save the file to a location on your computer.
• Run the file accessbridge-2_0_1.exe from your computer.
• Follow the installation instructions.
• Restart your computer when prompted.

To install the Microsoft Visual C++ 2008 Redistributable:

• Open the web page http://www.microsoft.com/downloads/details.aspx?FamilyID=9b2da534-3e03-4391-8a4d-074b9f2bc1bf&displaylang=en.
• Click Download.
• Save the file to a location on your computer.
• Run the file vcredist_x86.exe from your computer.
• Follow the installation instructions.

To install the AutoIT for Java Access dynamically linked library (DLL):

• Right-click on the link AutoITJavaAccess.dll below.
• Save the file to your Windows System32 folder (ie. C:\WINDOWS\system32).

To install the Tesseract UDF 0.6:

• Open the link http://www.autoitscript.com/forum/index.php?showtopic=89542.
• Save the file to your AutoIT include folder (ie. C:\Program Files\AutoIT\Include).

To manually check and update the Java Access Bridge:

Note:- This step should be performed in addition to the install of the Java Access Bridge above.

• Using Windows Explorer, go to your Java folder C:\Program Files\Java.
• Open the first subfolder (ie. j2re1.4.2_19).
• Go to the subfolder lib\ext.
• If the files access-bridge.jar and jaccess.jar already exist then go to step 9 below.
• Copy the file access-bridge.jar to this subfolder (ie. C:\Program Files\Java\jre1.4.2_19\lib\ext).
• If the subfolder in step 2 above starts with j2re1.2 or jre1.2, then copy the file jaccess-1_2.jar to this subfolder (ie. C:\Program Files\Java\jre1.4.2_19\lib\ext).
• If the subfolder in step 2 above starts with j2re1.3 or jre1.3, then copy the file jaccess-1_3.jar to this subfolder (ie. C:\Program Files\Java\jre1.4.2_19\lib\ext).
• If the subfolder in step 2 above starts with j2re1.4 or jre1.4, then copy the file jaccess-1_4.jar to this subfolder (ie. C:\Program Files\Java\jre1.4.2_19\lib\ext).
• Repeat steps 3 to 8 for every subfolder within the Java folder.

To confirm that the installation process was successful:

• Using Windows Explorer, go to C:\Program Files\Java Access Bridge and double-click JavaFerret.exe.
• In the Java Ferret window, select the menu item JavaEvents -> Track Mouse Events.
• Drag your mouse over any Java application, and the Java Ferret window should dynamically update with information about the Java objects as you drag your mouse over them.

LIST OF FUNCTIONS:

_JavaAttachAndWait($win_title)

_JavaObjValueGet($autoit_cntl_id, $javaferret_obj_name = "", $javaferret_obj_role = "", $instance_num = 1)

_JavaObjIndexGet($autoit_cntl_id, $javaferret_obj_name = "", $javaferret_obj_role = "", $instance_num = 1)

_JavaObjValueSet($autoit_cntl_id, $javaferret_obj_name, $javaferret_obj_role, $object_value, $instance_num = 1, $partial = 1, $get_last_capture = 0, $delimiter = "", $expand = 1, $scrolling = 1, $cleanup = 1, $max_scroll_times = 5, $scale = 2, $left_indent = 0, $top_indent = 0, $right_indent = 0, $bottom_indent = 0, $show_capture = 0)

_JavaObjSelect($autoit_cntl_id, $javaferret_obj_name = "", $javaferret_obj_role = "", $instance_num = 1)

_JavaObjValuesGet($autoit_cntl_id, $javaferret_obj_name = "", $javaferret_obj_role = "", $instance_num = 1, $get_last_capture = 0, $delimiter = "", $expand = 1, $scrolling = 1, $cleanup = 1, $max_scroll_times = 5, $scale = 2, $left_indent = 0, $top_indent = 0, $right_indent = 0, $bottom_indent = 0, $show_capture = 0)

_JavaObjDeselect($autoit_cntl_id, $javaferret_obj_name = "", $javaferret_obj_role = "", $instance_num = 1)

_JavaObjPropertyGet($autoit_cntl_id, $javaferret_obj_name = "", $javaferret_obj_role = "", $instance_num = 1, $property_name = "")

_JavaTableRowSelect($autoit_cntl_id, $javaferret_obj_name = "", $instance_num = 1, $javaferret_panel_name = "", $panel_instance_num = 1, $col_names = "", $col_values = "")

_JavaTableCellSelect($autoit_cntl_id, $javaferret_obj_name = "", $instance_num = 1, $javaferret_panel_name = "", $panel_instance_num = 1, $col_names = "", $col_values = "", $col_name_select = "")

_JavaTableCellValueGet($autoit_cntl_id, $javaferret_obj_name = "", $instance_num = 1, $javaferret_panel_name = "", $panel_instance_num = 1, $col_names = "", $col_values = "", $col_name_select = "")

_JavaTableCellValueSet($autoit_cntl_id, $javaferret_obj_name = "", $instance_num = 1, $javaferret_panel_name = "", $panel_instance_num = 1, $col_names = "", $col_values = "", $col_name_select = "", $object_value = "")

DEMONSTRATION:

<Under Construction>

EXAMPLES:

_JavaObjValueGet.au3

_JavaObjValueSet.au3_JavaObjValueSelect.au3

_JavaObjValuesGet.au3_JavaObjPropertyGet.au3

DOWNLOAD:

Latest Version - v0.7 (17/03/09)

AutoITJavaAccess.dllJava.au3

Edited October 5, 2009 by seangriffin

[yehia]

Edit: Wow looks cool yet

and thanks for posting

Edited January 19, 2009 by yehia

[Xenobiologist]

Great project!

I'll give it a try.

Mega

[Valuater]

Repeat steps 3 to 8 for every subfolder within the Java folder.

Maybe write a quick little installer...???

8)

[seangriffin]

Maybe write a quick little installer...???

8)

Thanks everyone for the positive feedback already (after only posting this last night!)

Good call Valuater. I'm going to try and do this using NSIS. I was hoping this step wouldn't be necessary, but I've found (on my computer anyway) that after installing the Java Access Bridge it hadn't automatically placed the Bridge into all the Java Runtime Environments (JREs) on my computer (like it said it would). Maybe it's a bug in V2.0.1 of the Bridge. At least my own installer should get the job done.

[seangriffin]

Can someone do me a favour and download AutoITJavaAccess.dll from my post above, and save it to their C:\windows\system32 folder, and run the following command from an AutoIT script:

ConsoleWrite(dllopen("AutoITJavaAccess.dll"))

The command outputs a "1" in the console on my home computer, which is great. On the computer I use at work, it outputs a "-1". I hope no-one else gets a "-1".

Edited January 20, 2009 by seangriffin

[ReFran]

Can someone do me a favour and download AutoITJavaAccess.dll from my post above, and save it to their C:\windows\system32 folder, and run the following command from an AutoIT script:

ConsoleWrite(dllopen("AutoITJavaAccess.dll"))

The command outputs a "1" in the console on my home computer, which is great. On the computer I use at work, it outputs a "-1". I hope no-one else gets a "-1".

quick answer:

-1+>13:41:54 AutoIT3.exe ended.rc:0

Computer on work, with firewall, ...., but as Admin. (because its a notebook).

Perhaps it has to be registered first (regedit ...)

best regards, Reinhard

[seangriffin]

quick answer:

-1+>13:41:54 AutoIT3.exe ended.rc:0

Computer on work, with firewall, ...., but as Admin. (because its a notebook).

Perhaps it has to be registered first (regedit ...)

best regards, Reinhard

Thanks so much Reinhard. Glad to see the issue is consistent across all computers except the computer I'm developing on. I ran "regsvr32 AutoITJavaAccess.dll" on another computer and got an error message that seems to indicate that the file is not correctly installed.

"AutoITJavaAccess.dll" is my first attempt at creating a DLL using MS Visual C++. I can build it and use it fine on my own computer, but after distributing it to other computers it doesn't work. It seems I have to build a (re)distributable package or installer of sorts, in order to distribute this DLL to other computers.

I'm going through a learning curve with Visual C++. You'll all have to bare with me until I sort this out. I think I'm nearly there though. Maybe in the next few days. I'm also having Windows Message Queue issues in my code, but again it's just because I'm new to C++ programming in Windows.

Edited January 21, 2009 by seangriffin

[ptrex]

@seangriffin

Maybe this can help, not sure :

Microsoft Visual C++ 2008 Redistributable Package

regards,

ptrex

[IndyUK]

Hi seangriffin...

This is great. Will this work with Java applications that use Swing? I've recently developed a script that drives a Java application which uses alot of mousemove and click command. Of course the problem there is the screen must remain the same geometery as it was designed on. However, if this allows me to reference the control directly...well that's just brill.

Do you have any examples or documentation that I could use?

Thanks for the great UDF.

Well done.

[seangriffin]

@seangriffin

Maybe this can help, not sure :

Microsoft Visual C++ 2008 Redistributable Package

regards,

ptrex

Thanks ptrex. After doing some Googling I also discovered that this was probably going to be the easiest and quickest way for me to get my AutoITJavaAccess.dll installed on everyone's computer. I tried it and it worked. The DLL is now recognised as a DLL in AutoIT (phew). I've now included the "Microsoft Visual C++ 2008 Redistributable Package" in the REQUIREMENTS section above.

Edited January 23, 2009 by seangriffin

[seangriffin]

Hi seangriffin...

This is great. Will this work with Java applications that use Swing? I've recently developed a script that drives a Java application which uses alot of mousemove and click command. Of course the problem there is the screen must remain the same geometery as it was designed on. However, if this allows me to reference the control directly...well that's just brill.

Do you have any examples or documentation that I could use?

Thanks for the great UDF.

Well done.

I'm getting there IndyUK. Yes, so long as the JavaFerret.exe program (included in the Java Access Bridge) can recognise Java objects in your swing application, then this UDF should work for you. A couple of years ago I wrote my own Java application using Swing, and the Java Ferret works with it.

This post is under construction. I need to add much much more information about using this UDF. It's a little bit tricky, due to the nature of the Java Access Bridge. I have high hopes for this UDF. We'll see if I have what it takes to pull it off. This is the first time I've seen real Java applications automated, and I'm pretty excited about the concept. I think I may have overcome most of the issues I've been having in developing "AutoITJavaAccess.dll".

I'll remove the "Under Construction" disclaimer very soon. I think I may have something almost working.

Edited January 23, 2009 by seangriffin

[ivan]

Sounds great, must try it. Thx 4 posting

Regards,

IVAN

[seangriffin]

Just a quick note to let everyone know that this UDF is now officially at version 0.1 (and no longer "Under Contruction")

Please give it a try and provide lots of feedback. I'm really impressed with what it can do so far, but I think there is some way to go.

This version includes the ability to get values from Java objects, set the value of Java objects, and select Java objects.

I know the installation is not that simple at the moment. Hopefully I'll be able to automate the process somewhat in the near future. Once you've completed the installation, then I recommend you run the example _JavaObjValueGet.au3. If this script is able to retrieve values from Java applications, then the entire UDF should work for you.

Edited January 26, 2009 by seangriffin

[dynty]

it is great thanks a lot i have been looking for it for some time too..

i have small problem,a lot of application does not have object in them properly "named"

if i noticed correctly,the only difference is that "Index in parent" is different for them

Isnt it possible to read the object based on this value instead of "Name" ?

Thanks a lot

Edited January 27, 2009 by dynty

[seangriffin]

it is great thanks a lot i have been looking for it for some time too..

i have small problem,a lot of application does not have object in them properly "named"

if i noticed correctly,the only difference is that "Index in parent" is different for them

Isnt it possible to read the object based on this value instead of "Name" ?

Thanks a lot

Yes that's right. I have also noticed this. So I placed an "$instance_num" parameter at the end of my functions. You should be able to see this parameter at the end of all my functions so far (except for _JavaAttachAndWait of course).

I haven't described this parameter very well in Java.au3 yet. You can use this parameter to specify which instance of a java object to use in the function. It's designed to work with the second parameter "$javaferret_obj_role". Populate "$javaferret_obj_role" with the role of the object you want to work with (by getting the role from JavaFerret.exe), and specify the instance of that role to work with (a number from 1 onwards).

If you set "$instance_num", then you need to make sure the first and second parameters are not set ($autoit_cntl_id and $javaferret_obj_name).

There's examples of using instance_num in "_JavaObjValueGet.au3" and "_JavaObjValueSet.au3". For example:

$java_obj3 = _JavaObjValueGet("", "", "check box", 2)

$java_obj1 = _JavaObjValueSet("", "", "text", "5", 3)

Edited January 28, 2009 by seangriffin

[seangriffin]

V0.2 has been released.

The downloads on page 1 been updated to V0.2.

New functions include:

• Java Combo box support
• Java List box support
• Deselecting Java Objects
• Getting all values from multi-value Java Objects (ie. Java Combo boxes and Java List boxes)

I'd love to hear anyone's feedback on the UDF so far.

I'm having issues getting values from Java combo boxes and list boxes. The values returned appear to be encoded/encrypted. I get a feeling that Java has them encoded in some way, and I'm going to have to decode/decrypt the values. If anyone has any information on getting the proper text from Java combo boxes and list boxes, that will be a big help. The issue is evident when running the examples. jEdit does not have the problem, but piface and web applets do. Seems to be related to the fact that the AutoIT Info Window can recognise controls in piface and web applets, but not jEdit. Therefore the way jEdit exposes it's Java lists must be different to the other two apps.

The way I retrieve Java list box values from apps like piface and web applets is using the AutoIT "_GUICtrlListBox_GetText" function. This function makes a call to "_SendMessageA" which in turn calls the function "SendMessageA" in the DLL "user32.dll". The problem must therefore be in the data that this "SendMessageA" function is returning. SendMessageA is a forced ANSI call. Perhaps I should try calling "_SendMessage" instead. Maybe it's this forced ANSI conversion that's screwing up the text coming back from the listbox. I'll give this a try.

No luck. I'm convinced AutoIT and Java Access Bridge cannot read Java listboxes and combo boxes in some apps. I found other people with the same frustration on other forums. I'm going to move to screen scraping for these apps. I'm developing a Textract UDF which seems to have good results with recognising text inside Java list boxes. Unhappy that it's not freeware, and that there is no good free OCR packages out there. It's all I've got for now though. Will post an update soon on this.

Update 16/02/09:

I've developed a Tesseract UDF which performs nicely, and is free. Very happy. I now have a solution for reading listboxes and combo boxes for those troublesome Java apps I've described above.

Cheers,

Sean.

Edited March 17, 2009 by seangriffin

[seangriffin]

I've released V0.6 of the UDF today.

The downloads on page 1 been updated.

New features include:

• More properties available to "_JavaObjPropertyGet"
• Improvements to "_JavaAttachAndWait", "_JavaObjSelect" and "_JavaObjValueSet"
• New function for getting the selected index in combo boxes and list boxes ("_JavaObjIndexGet")
• Some basic table functions ("_JavaTableRowSelect", "_JavaTableCellSelect", "_JavaTableCellValueGet", "_JavaTableCellValueSet")

A big thanks goes to tbui who helped enormously in the improvements to "_JavaAttachAndWait", "_JavaObjSelect" and "_JavaObjValueSet". The result was that he was able to automate the web-based Oracle Application Server Forms Services application. This is a big step for the UDF to be able to automate a commercial grade application.

The new table functions will need more work, so I'd love anyone's feedback who has tried using them.

Cheers,

Sean.

Edited March 15, 2009 by seangriffin

[seangriffin]

Released V0.7.

Very small update, mainly to add support for Tesseract UDF 0.6. Also made a small fix to "_JavaAttachAndWait".

[ProgAndy]

As far as i can see, the JComboBox has a panel as child-item. This panel includes a label for each entry Possibly this helps you to add native support without OCR.