Synchronization in autotests (part 1)

One of the most serious problem of automated tests ( especially GUI level tests ) is synchronization between test execution and application under test work. In other words, all actions being performed by the automated test have to be executed exactly at the moment when application under test is on the correct state allowing to perform required actions. Otherwise we are at risk to come across the situation when test tries to click on some object, enter some text while the form to perform these operations on is simply not available. As the result, our test goes by its fixed steps but the results and system behavior is unpredictable. And if at the same time there are no recovery scenarios the whole test set execution may be failed at once.

So, we have to provide some autotest execution speed in such a way that it should correspond to application under test execution speed. How can we do this?

The simpliest solution is to set delays in the test code. Almost any automated functional testing tool has instructions making pause for some period of time. Most tools of this type have functions like: sleep( nSec ) or wait( nSec ). TestComplete does it by means BuiltIn.Delay() instruction call, Java allows making call like Process.sleep( ). All these solutions have similar structure with single argument specifying pause timeout.

Advantages:

1. Simplicity. Built-in functionality usage is quite simple and understandable
2. Sometimes such pauses prevent execution "jamming" of automation tool in case two resource-intensive operations are performed one after another ( e.g. one second pause in SilkTest sometimes prevents Agent from hanging ).
3. It is effective enough solution for operations with fixed delay time. For instance, message box being invoked by client script on web page require just one or two second to be sure that this window is really available.

Nevertheless, such solution has disadvantages:

1. Irrational execution time waste. Some operations especially client-server request processing may take different time even for the same operation. As the result, there is a reason to make pause to cover maximal acceptable period of time. So, if action was performed even before specified timeout the execution is suspended to the fixed time period. And this makes us wait up to several minutes for single delay. In case of a number of such delays the total time loss makes up to several hours.
2. Application under test may work with different speed on different environments. As the result, all occurences of delay instructions should be corrected to fit appropriate waiting time.
3. Such pauses do not reflect the reasons of delays. This makes code less readable. Also test code may contain too many delay instructions making this code rapidly grow in size without any visible reasons.
4. These pauses do not take any guaranties that application under test achieved required state

As the alternative there is solution providing possibility to slow down automated test execution speed. One of the automated test execution advantages is operations execution speed. But sometimes it becomes a major disadvantage because instructions are performed so fast that some application under test event handlers do not manage to handle all operations in time. This results as a number of errors which can't be reproduced manually. Most of automated functional testing tools have settings providing ability to slow down keyboard entry and mouse operations speed. Some solutions have specific instructions for this purpose. For example, Watir library has such field as Watir:IE:speed , accepting either :slow or :fast valuesdepending on required execution speed. Selenium RC has the analog presented by setSpeed method of main client class. Such approach has the following advantages:

1. Simplicity. It is enough to make required delay settings in a single place.
2. Possibility to adjust tests to real conditions execution speed to make
3. Optimized test code volume. There is no necessity to specify pauses everywhere. Tests implementation contains actions to perform only

Several disadvantages of explicit pause settings is omitted using this approach. But some disadvantages are still available:

1. Irrational execution time waste. Most of delays may be unnecessary. As the result, we are at risk to lose such automated testing advantage as high execution speed. For example, we have to fill in some fields in the form and press the button. It's more convenient to fill in data fast rather than use delays for some seconds because it makes time loss
2. There is no stabilization in case of complex queries which take too much time. Actually, such approach is not applicable for long time range synchronization.
3. Still there is no guarantee that application under test achieved required state

So, such approach is more or less applicable for local applications with stable navigation.

Both approaches listed above do not solve similar problem like application under test state control though it's very important point. And there's a reason why. If application under test hasn't achived required state further test execution is useless and test execution has to be either interrupted or restore required state to continue test run. But we have to verify that application behaves unexpectly before decision making about further execution. In order to solve problems like this there are different methods providing ability to examine application under test. For instance, all automated GUI functional testing tools have functions/methods like Exists ( in SilkTest, RFT, Waitr these are methods, in TestComplete it is implemented by properties ), win_exists, obj_exists (WinRunner) or something like isElementPresent as it is implemented in Selenium RC. Such solutions verify specified window object availability. So, in this case synchronization may be implemented by the next algorythm:

bObjFound = false;

while( !timeOutExeeded ) {

        if( elementPresent( element ) ) {

              bObjFound = true;

              break;

        }

        verifyTimeOut();      

}

if( !bObjFound ) {

        Error( "No object available" );

}

The pseude-code listed above works by means test execution circularity until either required element appears or waiting time is out. Maximal waiting time is some value being set up according to application under test performance abilities. In most cases this is average execution time for typical request under stable environment. The template from above is initially built into web-page load completion verification. For example, Watir has the mothod like Watir::IE.wait . Selenium RC has methods waitForFrameToLoad,
waitForPageToLoad, waitForPopup etc.

As long as such solutions allow some window object availability verification they provide application state changes control because in most cases application state can be defined by either availability or absence of some window or page element. But there is more generalized situation when application state change is reflected by means some window object state changes. it means that we have to wait until some window object property has some specified value. TestComplete has WaitProperty method describing interface for such solution in the most descriptive way:



 obj.WaitProperty( propertyName , propertyValue , timeOut )

This method returns True in case property specified by propertyName has the value specified by propertyValue parameter before timeout specified by timeOut parameter runs out. Otherwise False value is returned. Similar solutions can be implemented in any other similar automated testing tools ( by the way, it's too convenient to have such analogs ). In this case synchronization is actually combined with application state change verification ( to be more exact,  window object state modification verification ). Of course, the code volume increases but at the same moment we gain the following advantages: 

1. Automated test execution requires no more time than it is really needed ( as soon as required state occurs the test execution goes on )
2. Application state changes are well-fixed
3. We may report informative error message in case required application state isn't reached because we make explicit verification point
4. Readability is made better because expected application state may be brought to conformity with automated test implementation code by means such verification points

These are the most typical solutions for synchronization problem. The most effective solutions usage provides simplification of code writing and maintainance. It's true especially taking into account that about 60% of test execution errors being found during development state are synchronization problems. So, correct synchronization problems solution approach applied at the early test automation stages may highly decrease maintainance costs.