Any framework for the automation testing

[netnet227]

autoit is very powerful. but seems i can't find any framework(or example scripts) for the automation testing.

For framework, I mean supporting test suite, test case, "data-driven" file loading, logging and report features.

anybody can give me some hint?

[zfisherdrums]

For many of these, you will need to roll your own.

At my company, I've implemented data-driven and keyword driven frameworks using AutoIt but always needed to extend its base functionality to meet our needs. This included a unified logging library, a GUI Mapping library, App-Action library, etc.

Fortunately, many of the elements you will need are already available via UDFs and other posted items. I would offer my own, but they are never finished and therefore useless to you/anyone else right now.

I would like to hear more about the project you are working on, though. This would help contextualize your need, and help us help you better.

Zach...

[netnet227]

For many of these, you will need to roll your own.

At my company, I've implemented data-driven and keyword driven frameworks using AutoIt but always needed to extend its base functionality to meet our needs. This included a unified logging library, a GUI Mapping library, App-Action library, etc.

Fortunately, many of the elements you will need are already available via UDFs and other posted items. I would offer my own, but they are never finished and therefore useless to you/anyone else right now.

I would like to hear more about the project you are working on, though. This would help contextualize your need, and help us help you better.

Zach...

My project is about the GUI automation. i need to choose one software to do it. one is autoit, another is google's robotframework.

I found autoit is powerful but lack of some basic framework for testing purpose. and robotframework have all of those features but seems it's not so powerful for the GUI automation.

I am not sure whether i am not quite good at the search on this site or not. I tried to search "logging library", "GUI Mapping library", "App-Action library". but i didn't find what i need. would you share with your scripts even they are not finished? I just want to have a idea of how to do it and the possibility of doing it.

[DjDeep00]

@netnet227...Give an example of a testcase you would like to automate. What is expected and what you want to verify?

[zfisherdrums]

GUI Automation as a task was a given.

Additional questions:

What kind of application are you attempting to automate? Is it a client-side only? Web-app? Both?

What language was the app written in?

Do you have access to the developers?

Does the app use any third party controls in the UI?

What kind of tests are you being charged with running? System-level functionality? End-to-end sanity checks?

How frequently will these tests be run?

Your answers will greatly aid in determining the applicability of AutoIt ( vs. something else ) in your environment.

<two_cents>I continue to come back to AutoIt due to its interpreted nature and its focus on the windows domain</two_cents>

Edited December 31, 2008 by zfisherdrums

[netnet227]

What kind of application are you attempting to automate? Is it a client-side only? Web-app? Both?

A: client side. c/s application

What language was the app written in?

A: borland c++

Do you have access to the developers?

A: yes

Does the app use any third party controls in the UI?

A: No. but the GUI is complicated. i mean the logic of the UI. (for example check different options will pop up different inputboxes).

What kind of tests are you being charged with running? System-level functionality? End-to-end sanity checks?

A: The app is just for the order entry. we would like do the system-level testing. but since it involve other applications(more than 4) which is on unix, maybe it's not so easy to achieve this. At least, we can do some end-to-end sanity checks at this point and will include others later. the reason why we need an automation testing tool is this app has too many bugs and it's a legacy system which is hard to refactor it. You know, nobody want to change it a little bit coz once you do this, there will always something weird happens. but lots of app depend on it, so it's impossible to get rid of it.

How frequent will these tests be run?

A: It's hard to say. It depends on how often we change the software. at least once per week.

the robotframework use the phthon script which i believe more powerful than VB script. the problem is this tool is quite new , so has few documentation on this. As you said, VB script is quite good on the windows domain, maybe it's more fit to do the GUI automation testing on windows platform.

[zfisherdrums]

Cool. Now we're getting somewhere! And questions beget questions, but I'll spare you for now. Please understand all items below are strictly my own 2 cents. They are not gospel, and your mileage will definitely vary.

AutoIt (natively) doesn't have the logging/reporting capability that we observe in the Robot Framework. And I agree with your assessment of Robot's apparent lack of Win32 automation ability OTB.

However, it does appear that you would be able to call AutoIt Scripts from a thin Python/JPython layer inside of Robot. Or, utilize the Win32 automation libraries written for Python. A great deal depends on how the communication happens between those layers ( i.e., how does one obtain Pass/Fail codes, debug statements for inclusion a log file ). Just a thought.

Another thought: Write AutoIt Scripts that run your UI tests and report results in an XML format that can be consumed by an xUnit derivative. Then, use that xUnit tool to define your test suites and consume/report the results. Run the tests from the command-line and voila! Reports abound. If your devs are using any kind of CI environment, it should be easy to merge these into their existing workflow; to be triggered by changes in CVS or with the creation of a new distribution file. The scripts would need only concern themselves with running the UI tests correctly. The Setup and Teardown activities would be handled in the xUnit tool itself, as would the reporting. Overall, it may be less grunt work as the scripts are generally your only point of maintenance.

A final thought/option with respect to UI Automation toolsets, you may want to consider ThoughtWorks White project utilizes the .NET UIAutomation namespace( and supports Java/Swing as well). This namespace would be able to interface with native Win32 controls, many .NET controls, and utilizes the MSAA-bridge for additional control support. If your devs are working in .NET for any other apps, this may be a good move. Even if they're not, it may be a good move inasmuch as the UIAutomation namespaces are baked into .NET 3.+, and they aren't going anywhere anytime soon. You could still use the xUnit wrapper pattern, only the UI automation tool would be different.

Zach...

Edited December 31, 2008 by zfisherdrums

[zfisherdrums]

One caveat to the xUnit approach is the need to have a wrapper DLL that both references the xUnit namespace AND wraps around the scripts/scriptrunner. I could see this being a headache for anyone else trying to grok what's going on. Borders on Rube Goldberg, in retrospect.

The most pragmatic thing to do may simply be this: write all your test output to a file with the script's name and place it in a date-time stamped folder that is accessible to everyone. Cheap. Easy. If the system is so brittle, you will probably be running all tests with even the slightest change to the AUT. Consuming this into one nice report would be an exercise left to a post-run process, but at least all individual script results are there and archived.

Edited December 31, 2008 by zfisherdrums

[netnet227]

Thank you for you patience, zfisherdrums. I considered that the idea of putting the RobotFramework and autoit together. but need some time to look into their capability in advance. It's maybe easy, maybe not. I will try to feed back later. the thing is RobotFramework has too few documents and examples.

We didn't use the .net. but we do use Java. so maybe junit is one way to wrap the autoit. but i am not quite understand the DLL problem you mentioned. could you please explain?

[zfisherdrums]

Thank you for you patience, zfisherdrums. I considered that the idea of putting the RobotFramework and autoit together. but need some time to look into their capability in advance. It's maybe easy, maybe not. I will try to feed back later. the thing is RobotFramework has too few documents and examples.

We didn't use the .net. but we do use Java. so maybe junit is one way to wrap the autoit. but i am not quite understand the DLL problem you mentioned. could you please explain?

No, thank you for YOUR patience. I'm good at making a lot of alternate suggestions. My problem sometimes is focusing on the real problem.

First, you can scratch my "DLL" problem. On further review, it is unnecessary. Let me pull together some of those library files for you and post them later.

[netnet227]

[quote Let me pull together some of those library files for you and post them later.

[Judit]

Hi everybody,

Can any one let me know can we generate Reports of the test run in Autoit and also if not is there any other ways to generate the reports.

[netnet227]

Hi everybody,

Can any one let me know can we generate Reports of the test run in Autoit and also if not is there any other ways to generate the reports.

seems there's not way to generate the Reports from the autoit. there's other way, but still you need write you own code to feed other report generation tools.

I also want to know which way is the best. or the simplest way is to write some log file and you parse it by yourself

[zfisherdrums]

seems there's not way to generate the Reports from the autoit. there's other way, but still you need write you own code to feed other report generation tools.

I also want to know which way is the best. or the simplest way is to write some log file and you parse it by yourself

Sorry so slow on the update. But what items are you needing to report? There are many things that "could" be captured, but your context will define what "should" be.

Zach...

[Judit]

Sorry so slow on the update. But what items are you needing to report? There are many things that "could" be captured, but your context will define what "should" be.

Zach...

Hi friends,

Thanks for the reply. As you have asked the items i need as a report are

1. no of assertions verified , no of assertions passed and failed

2. total no of test cases , total passed and failed.

These details i want as a report.

[CMTaylor]

I've built a Robot Framework extension library that integrates AutoIt capabilities into Robot Framework via Python and AutoIt's COM interface. I've discussed with my manager the possibility of making this available as Open Source. He is agreeable and is seeking corporate approval through the appropriate channels. No promises, but I hope to be able to make this RF AutoIt library available soon on the Google code site.

[zfisherdrums]

I've built a Robot Framework extension library that integrates AutoIt capabilities into Robot Framework via Python and AutoIt's COM interface. I've discussed with my manager the possibility of making this available as Open Source. He is agreeable and is seeking corporate approval through the appropriate channels. No promises, but I hope to be able to make this RF AutoIt library available soon on the Google code site.

Sounds awesome!

[daluu]

For many of these, you will need to roll your own.

At my company, I've implemented data-driven and keyword driven frameworks using AutoIt but always needed to extend its base functionality to meet our needs. This included a unified logging library, a GUI Mapping library, App-Action library, etc.

...

Zach...

This is a good thread. One thing I'm curious to know is how effective is it to roll your own GUI mapping library to "detect" GUI components for test automation (i.e. really more of the test "validation" portion than automation/driving the app), especially for complex GUIs and ones that use 3rd party non-standard GUI controls.

I mean how does rolling your own AutoIt customizations compare to commercial GUI test tools like QTP and SilkTest, etc. Is AutoIt just as effective? Or does AutoIt fail to detect certain types of GUI components, even when you try to roll your own customizations, compared to the commercial tools?

Unfortunately this specific topic I'm discussion seems to be seldom elaborated online. All the "automation/test" examples are about Windows calculator or notepad, and I hear/see no complaints about the limitations of AutoIt in GUI test automation other than maybe this article:

http://plosquare.blogspot.com/2010/04/windows-gui-automation-with-autoit-and.html

I do know the effectiveness of AutoIt for test automation also depends on the app being tested. But still, why is there not much info on people's experience w/ AutoIt as a professional GUI "test" tool? Not many people use AutoIt for test automation? Avoidance of revealing intellectual property and trade secrets? Few actual success stories? Few actual failure stories? Presenting one's experience in this area will help newbies to GUI test automation with AutoIt better evaluate the usefulness of AutoIt for the job before one even starts work with it.

In my experience so far AutoIt works great as an automation tool to drive an application to do stuff but is limited to validating application state via GUI controls as it's GUI detection functions are limited. You can drive an app via multiple ways - keyboard, mouse, GUI control manipulation, but you can only validate (from GUI perspective) via the GUI controls. Use of keyboard typing or mouse clicks won't validate anything in terms of application state. This is the area I'd like to hear other's experience on.

[netnet227]

Agree with daluu about the "validation" part. Autoit group should focus on this part. Compare to the fancy "GUI builder" lib, which is not a must for the test, "validation" lib is more important.

Maybe since it's not a commercial product, I feel somehow autoit was lost its way. There are bunch of UTF created by freelancer but still lack of some core component itself. Just personal opinion.