Automated tests: completion criteria

During development of automated tests as well as any other program components we have to make our code satisfy the following characteristics:

• Readability
• Extensibility
• Transportability
• Optimality
• Correspondence to requirements
• Maintenability
• Others ( some project-specific features )

Perfection has no boundaries. That's why priorities and necessities of features listed above are usually defined in correspondence with current project needs. In order to prevent endless improvement process for each compoment we have to define some criteria indicating that current component is good enough to apply no more improvements. If we look closer to automated tests as particular kind of program components we'll see that autotests have their own specifics requesting some specific requirements.

First of all, once written each test is used many times and it's usual thing that application under test is being changed from time to time. Also there's high possibility that test should be modified and adapted to the most recent product changes. As the result we should pay attention to such points as readability and maintenability. The single coding style helps a lot for this purpose even if there's only one person involved in test automation development. There are some common naming, formatting conventions to keep on. As the result, common reading rules appear for the code written. That's why we need to develop some coding standards . They are either developped by some team to fit specific project requirements or some existing standards are taken as basis. For example, widely used programming languages already have common standards.

Secondly, due to automated tests are intended to be executed by machine then we have to keep in mind that one day these tests are going to be delegated to automatic execution completely. It means that there will be no human to control verifications correctness. At least it happens sooner or later. That's why we have to be sure that test performs all things it is supposed to do.

Thirdly, in most cases automated tests are developed on one workstation but executed on another workstation. So, we should minimize or parametrize any workstation-specific features ( e.g. absolute paths ). As ideal result, all tests should work in the same way in all conditions the application under test is supposed to work. This is quite necessary especially for configuration testing.

Fourthly, one of subjective test automation advantages is that machine performs testing faster than human. At least it must be so. But if it is not so then there is a point to decide whether test automation is really needed or there are some other serious reasons for test automation. It means that all automation tasks should be implemented in the most optimal way.

There can be a lot of requirements. I've enumerated just a few major ones. From the requirements listed above we may define some criteria indicating that automated test is developed and ready to use:

1. Hard-coding is minimized. There should be no "magic words" and "magic numbers" because their meaning may be forgotten in the nearest future. It means that explicitly defined values should be used in very rare cases like:
   
   
   
   • data input/output format definition
   • external system resource reference ( referenced resource is part of the system )
   • values delimiters in case of explicit format
2. Components scope is limited by the usage area - it means that variable, function,class,object declarations should be accessible only from modules they are actually used and nothing more. Also glocal variables/objects usage should be minimized.
3. Components naming and formatting is done in accordance with coding standards and other conventions
   
4. All actions and verifications provided by test must be implemented - in any way, there are some instructions being the base for automated test. It can be different specifications or test scenarios. In case of testing delegation to machine we have to guarantee that required tasks are performed exactly the same way as they supposed to be. Thus, all actions and verifications have to be either implemented or there should be some remark describing the reason of some parts missing.
5. Automated tests should have infomative output - well-written test after execution should exactly inform if there were errors and in case of error there should be a message which clearly defines problem and the place where problem occurs.
6. Test implements actions in the most optimal way - many algorythms perform some tasks in different ways. Some of them are better and some of them are less effective. We have to provide the most effective task implementation with minimal sensitivity to system corrections/modifications.
7. Test should have minimal dependence on specific workstation settings - binding to system should be minimized. As a rule, it is preference to use relative paths instead of absolute ones. Some key values should be moved to some configuration file or any other external resource.

These are only several criteria. But they can form a basis for other rules. And these rules may allow us define whether test is ready for automatic execution.