Introduction

EMOS FRM is a test framework that allows you to quickly develop a complex test automation environment. We have named this product "framework" as it consists of two main parts: the methodology and the tools that support the methodology.

Our practice has repeatedly shown that a successful application of GUI-based test tools requires a systematic resolution of certain classes of problems. These problems seem to be independent of the application which is to be tested and of the project structure.

In other words, no matter what system you test or what project you are in, if you are going to test it, you will be faced to similar problems in all of them. We distinguish two types of problems: technical and organisational (structural).

Technically there are huge differences between the systems being developed. From the testing point of view the problem is always similar: 1) make sure you know what you want to test and you know how you want to do it, 2) perform the necessary actions, 3) check the necessary parts along the way and 4) act upon unexpected differences. However, the technical differences between the tests of a graphical application, a text processor, a database or a business application full of simple forms will be in some (if not most) parts radically different. Not even mentioning the real-time systems, embedded systems and all sorts of other nice things out there.

Don't worry! We are not going to try to convince you that we have found the holy grail. We are just trying to explain that if somebody is trying to find some "generic" solutions to "generic" problems, then she or he meets our focus. Our focus is:

functional testing via the application's user interface.

Why are we "digging" so much on our scope? It is because people apparently tend to forget important things about concepts over time with bizarre effects on the effectiveness of these very same concepts. We are exploiting the capabilities of the commercial of-the-shelf tools for performing functional testing. Nothing more, nothing less.

Nothing spectacular, yet still we believe that with the simple means, cleverly organised, one can reach beyond many of the existing approaches.

The problems

So what are those technical and organisational problems with automated testing? Let us scratch the surface.

Limitations of test tools

One of the technical challenges is the support for "custom" graphical objects. These are the GUI elements deployed by the application which do not belong to the family of so called "standard" objects. All vendors of GUI test tools provide a built-in support for certain classes of GUI objects which are then called standard objects. Typically, these are the elements of the most popular GUI libraries such as MFC, JFC, etc. If the particular tool does not support GUI objects of your application, you are left with several options:

The literature is full of advises to carefully verify the test tools before deciding to use (i.e. pay for) them. No doubt these are good advises. The problem with it is that only a few do it properly. Assume for the moment you had done it well. The tool(s) you evaluated is so expensive that you will want to use them for many years and/or for different applications. There is always something you will overlook simply because you did not know at the time how you are really going to use the tool.

Our advise is, of course, to check first. However, do not get upset when you experience problems. Get creative! Remember, you are not testing because you have the tool - you are using the tool because you are testing. Look for the workarounds. There are so many ways you can go with testing - use them. We will show you some.

Complexity

When you automate your test you will soon realize how quickly testware grows. Testware seems to be the established term for everything that you produce in order to run your automated tests - test scripts, test data, stubs, whatever. Managing the chaos of numerous test scipts acting on complex test data which is all being applicable only to a certan version(s) of tested software at some point in time is a nightmare. You should not get scared. But you should not ignore it either.

The best way to cope with the complexity is to anticipate it and actively look for the solutions to reduce it. Some design techniques are superior to others in respect to complexity. Do not underestimate the benifits of such techniques. It is them that will make yout test automation a long lasting activity (from the technical point of view).

One of the main design premisses for EMOS FRM is the reduction of complexity.

Testware is software

Many people seem to be easily trapped with nice words of skilled sales personell promissing that anybody can use their product. If not imediately, then with some training - and off they fly.

Forget it!

There is not a single GUI test tool in the market that is capable of testing complex GUIs and, at the same time, being easy to use. They are all complex. They may provide some "simple" means for capturing tests (busszord C/R - capture & replay). What they produce in the background is unfortunately everything else but simple. If you never had to look at the code that was produced, this would never be a problem.

A robust test can practically never be developed by just capturing some actions. You have to extend the code, write routines to handle different situations and do all sorts of other things to make good test scripts. Even if you could develop the robust test just by clicking around, it is intrinsic to testing to "live with" bugs. Within the tests itself and, of course, within the software that is being tested. Imagine your perfect test script one day suddenly fails just because the developers decided to remove or rename some graphical elements your script was using. No test scripts that we know of are able to "fix" themselves without requiring somebody to look into the code.

It is not the coincidence why we address this issue after dicussing the complexity problem. Leave you test scripts in hands of non-programmers and your test automation will drown in complexity. Leave them to non-skilled programmers, the same will happen.

Testautomation is primarily a programming task. If you do it cleverly, the test you produce can be understood, run or even created by non-programmers. But this is only because you had some programmers that made it possible.

EMOS FRM gives you a great platform for separating the expertise. If you develop test scripts the way we do, then a single test programmer can provide material for many non-technical people. If they work closely together, you will be amased how much can be automated in a short time.

Moving targets

If the application you are trying to test does not show any stability in areas you are trying to automate, you will loose the game. No test tool in the world and no test programming guru can compensate for bad development processes.

Do not waste money, time and people's nerves by asking or letting them write test scripts that cannot be run twice without having to be modified.

Wrong expectations

Test automation can do miracles but if you expect them, you will be disapointed.