To achieve these goals it is important to write good tests. Because just as good tests improve quality and increase the speed of development, bad tests can have the opposite effect. But what makes tests good or bad?

Testing is a big field and it’s worth questioning some assumptions and actively think about which testing strategy fits best for your system. In this blog post, I’d like to present a slightly different form of the test pyramid, let’s call it the ’test diamond'.

As a rough guideline, the test pyramid has evolved over the last 20 years. This pyramid describes the structure of the test suite divided into Unit Tests, Service Tests and UI Tests. The exact definitions of these terms can be argued, but the basic idea is that you have a lot of fine-granular tests and only a few coarse-granular tests.

The test pyramid is often justified by the fact that Unit Tests help you to localize the defect quickly. This is reasonable, because if a SortingAlgorithmTest fails, it is clear that there is a bug in the class SortingAlgorithm. However, nowadays, if you use version control and you make many small commits instead of a few big ones (yes, you should!), you can quickly locate the defect anyway. If the tests were just green, the defect must have obviously been introduced by the changes you just made.

Another argument for a broad base of Unit Tests is a fast feedback cycle. Fast feedback is important to really develop in a test-driven fashion. So you write a test, make a small change to your code, run the test, add the next change etc. without being slowed down by the execution time of the tests.

However, Unit Tests have a serious disadvantage: they require a lot of work during refactorings. Most SW engineers are familiar with this situation: The changes to the code could be done in 1-2 hours, but changing the test suite might take 1-2 days.

Service Tests, on the other hand, are considered slow and time-consuming. But does this really need to be like that? If it is possible to execute service tests quickly, then service tests are strictly superior to unit tests:

  • Unit Tests cover less code per test. That means with the same number of Service Tests I can cover more lines of code and have a higher chance to find defects and avoid future regressions.

  • Service Tests better describe the use case, so they also serve of documentation.

  • Because Service Tests only describe the external behavior, they do not have to be adapted during refactorings. On the contrary: they accelerate refactorings refactorings immensely as they serve as a guardrail and ensure that the code does the same after refactoring as it did before.

Therefore, my ideal test pyramid is actually a diamond. At the top we have a few UI tests for critical areas. Then we have a thick belly of Service Tests (also called scenario-based tests) that test the interaction of our application with the outside world. Then at the bottom we have a few Unit Tests that helper methods and similar functionality that cannot be assigned to a specific scenario.

With this test diamond, you get a test suite that helps the team to quickly develop new features while still allowing for safe refactorings without reducing the development speed.

What do you think? We are looking forward to hear about your experiences and opinions.

Webinar: Scaling software development as a SaaS company without becoming consumed in the growth process.

The webinar addressed exactly the pain points that are occurring as we grow. With an ever-increasing number of employees on the Dev & Product team, we've had to find out for ourselves how difficult it is to set up a sensible structure that brings out the maximum efficiency and performance in everyone, while also leading to a healthy working atmosphere.
Pascal von Briel, Co-Founder & CPO @ exporto GmbH