Load tests analyze the behavior of the application under load. Performance tests check the speed of a system, and capacity tests examine how many users or requests the system is able to handle. All of these tests evaluate the efficiency of the application. For this purpose, they use similar tools that measure response times and generate load. Such tests can also monitor the use of resources or check whether errors occur under a certain load. Tests that investigate whether a system is able to cope with high load over an extended period of time are called endurance tests.
The distribution of tests is illustrated by the test pyramid (Figure 10.1): The broad base of the pyramid represents the large number of unit tests. They can be rapidly performed, and most errors can be detected at this level. There are fewer integration tests since they are more difficult to create and take longer to run. There are also fewer potential problems related to the integration of the different parts of the system. The logic itself is also safeguarded by the unit tests. UI tests only have to verify the correctness of the graphical user interface. They are even more difficult to create since automating UI is complicated, and a complete environment is necessary. Manual tests are only required now and then.
Test-driven development usually results in a test pyramid: For each requirement there is an integration test written, and for each change to a class a unit test is written. This leads to many integration tests and even more unit tests being created as part of the process.
The test pyramid achieves high quality with low expenditure. The tests are automated as much as possible. Each risk is addressed with a test that is as simple as
Figure 10.1 Test Pyramid: The Ideal
possible: is tested by simple and rapid unit tests. More difficult tests are restricted to areas that cannot be tested with less effort.
Many projects are far from the ideal of the test pyramid. Unfortunately, in reality tests are often better represented by the ice-cream cone shown in Figure 10.2. This leads to the following challenges:
- • There are comprehensive manual tests since such tests are very easy to implement, and many testers do not have sufficient experience with test automation. If the testers are not able to write maintainable test code, it is very difficult to automate tests.
- • Tests via the user interface are the easiest type of automation because they are very similar to the manual tests. When there are automated tests, it is normally largely UI tests. Unfortunately, automated UI tests are fragile: Changes to the graphical user interface often lead to problems. Since the tests are testing the entire system, they are slow. If the tests are parallelized, there are often failures resulting from excessive load on the system rather than actual failures of the test.
- • There are few integration tests. Such tests require a comprehensive knowledge about the system and about automation techniques, which testers often lack.
- • There can actually be more unit tests than presented in the diagram. However, their quality is often poor since developers often lack experience in writing unit tests.
Figure 10.2 Test Ice-Cream Cone: Far Too Common
Other common problems include unnecessarily complex tests that are often used for certain error sources and UI tests or manual tests being used to test logic. For this purpose, however, unit tests would normally be sufficient and much faster. When testing, developers should try to avoid these problems and the ice-cream cone. Instead the goal should be to implement a test pyramid.
The test approach should be adjusted according to the risks of the respective software and should provide tests for the right properties. For example, a project where performance is key should have automated load or capacity tests. Functional tests might not be so important in this scenario.
Try and Experiment
- • In which places does the approach in your current project correspond to the ice-cream cone rather than the test pyramid?
- • Where are manual tests used? Are the most important tests automated?
- • What is the relationship of UI to integration and unit tests?
- • How is the quality of the different tests?
- • Is test-driven development used? For individual classes or also for requirements?
Continuous Delivery Pipeline
The continuous delivery pipeline (Figure 4.2, section 4.1) illustrates the different test phases. The unit tests from the test pyramid are executed in the commit phase. UI tests can be part of the acceptance tests or could also be run in the commit phase. The capacity tests use the complete system and can therefore be regarded as integration tests from the test pyramid. The explorative tests are the manual tests from the test pyramid.
Automating tests is even more important for microservices than in other software architectures. The main objective of microservice-based architectures is independent and frequent software Deployment. This can only be achieved when the quality of microservices is safeguarded by tests. Without these, Deployment into production will be too risky.