Automated Testing Principles

The biggest challenge of automated testing is results certainty. The failed test does not give us assurance that we found a defect. This happens because there are a lot of possible reasons for failures. For instance, issues with environment, issues with tools, eligible changes due to application development or even purely designed tests. Due to a survey, which was conducted in the end of 2020, roughly 50% of 264 recipients had more than 5% flaky tests. Test failing requires human attention, which makes feedback time bigger. The worse thing is that automated tests cease to be automatic. I think there are 4 principles, which could make automated tests more reliable.

The first principle is isolation. The test should be isolated as much as possible from the outer environment and other tests. I often see that one test indirectly or even directly affects another or when some global environment config affects a group of tests. Obviously, isolation capabilities depend on an application. One approach is to use unique data for every test. For example, use different users, products, etc. The good idea is to analyze where and how different tests can interact with each other and reduce these interactions. The ultimate solution is to use different environments.

In this example the tests are isolated by testing data. The TEST 1 could be affected by TEST 2 without isolation.

The next principle is managing application state. An engineer should be able to manage application state through API or directly with database interaction. All outer services should be mocked. This gives us predictable application behavior. An engineer is sure that the application has the correct state when the test starts.

Also, the principal helps to create small, atomic tests, which are more readable and faster.

In this example the special product is created through API and we are sure that product is available. The product is removed after TEST 3 execution because we need to keep the environment clean.

The next principle is API over UI. The principle is heavily based on the previous one. The aim is to use the user interface just on a page where we want to check something. Proxy is a tool, which helps to discover what happens under the hood and then simulate it with API and Database interactions.

In this example the required authentication token is obtained through API. The product is added through API as well and we open just one page, where we do some checks.

The last principle is functionality driven tests coverage. UI is extremely changeable on the development stage. I think this problem could be avoided by covering just core functionality. This is less possible that the buy button will disappear on a page, which aim is to sell something.

Following these principles requires special application design, which allows isolation and state managing. This requires additional efforts. However, I think this would make work easier for all team members, from QA to developers and what is more important is increasing the quality of the product.