Write better specifications with testing

logical test case and physical test case


A specification written by the customer – often ends as incomplete, fragmented and with parts that contradict each other – makes it difficult to build for the engineers.

A specification written by the engineer – often ends with a focus on details than on needs, and to difficult for the business to read/understand.

So how to do it? By using Given, When, Then!


Lego and IKEA have some of the best manuals/documentation, because they write them as test cases using “Given, When, Then”, mentioned in Write better documentation with testing

Given is don't start without it. When is what to do. Then is what to expect.

Documentation is about:

  • What givens do I need to have
  • and what when’s do I need to do
  • to achieve my then’s (goals)”

A specification is almost the same, but with a different focus:

  1. How do I build the given’s,
  2. so the users can do the when’s 
  3. to achieve their then’s (goals)

We think of specification and documentation as two different groups of documents, but they can technically be the same documents – if they are written as test cases.

Focus on needs than details

It is easy to forget the needs when the attention is on the details (read my 10 facts about details). In testing, there is something called a logical and physical test case, to split the design from the implementation.

Physical test case (operational steps)

A physical test case describes all the operational steps (given, when, then) to complete the test:

Given I open https://...com/webshop
And accept the "cookies information"
When I click the tab "tickets"
And I click button "Buy tickets"
And select dropdown "customer type" option "adult"
And click the "Continue" button
And move the slider to 7 zones
And click the "Continue" button
And rejects the special offer window
Then I should see the price of 18 €

The test case drowns in the operational details, and it becomes difficult to read, what the test case is actually testing.

Logical test case (business process)

A logical test case describes the business behavior (given, when, then) to only focus on what the test is about, and not on the implementation itself:

Given I am buying a ticket on the web shop
When buying a <child / adult / student / retired> ticket
And to travel <2 - 8> zones
Then the price should be <price> €

The logical test case describes the business rules (behavior), is easy to understand, and can be reused whenever a new version of the product needs to be implemented.

Logical test cases make it easier for the customer and supplier to understand each other.


We all know the situation when a developer and a customer agree verbally on something, and it gets implemented. Nobody wrote it down, so it will never end up in the specification and never in the documentation.

In Behavior Driven Development (BDD), the logical test case is written before the implementation. It might seem counter-intuitive, for how can we test something that doesn’t exist?

Logical test cases are our assumption of what the product should do, and can be used to measure the status of the implementation, and later be used as documentation.

Without contradictions

Sometimes two business processes can contradict each other and nobody will notice.

  1. When the first process is implemented, then its test case will pass.
  2. When the second process is implemented, then its test case will pass, but also make the first test case fail.
  3. Fixing the first process, so its test case doesn’t fail anymore, will make the second test case fail.
  4. Fixing the second process, so its test case doesn’t fail anymore, will make the first test case fail.
  5. etc..

A specification written with test cases is executable and can catch contradictions in business processes.


The customer can put the logical test cases in their end-to-end business flows, to see if every critical behavior is mapped. Should only be applied for critical behavior and not every behavior).

Missed or misunderstood business behaviors can always be added or updated on a later stage as a change request.

The specification will not be complete to begin with, but will become more and more complete over time.

It can later be used as documentation and reused as a specification when a new version of the product needs to be implemented.

Layers of Complexity and when to apply testing

Product requires test automation. Test automation requires its own test automation.


Testing helps with complexity, but need to be applied at the right situation, to deal with complexity, without adding increasing it unnecessary.

Complexity level 0 – clarity

The code is simple and easy to read.

How to deal with it:

No need to test it and no documentation needed.

Avoid comments – because you will need to maintain both the code and the comments. Outdated comments confuse more than help.

A single comment can sometimes be acceptable, when a single line of code is unintuitive, because of poorly implemented method in a framework or legacy code.

Clean code, SOLID principles, linting, etc. can help extend this level.

Level 1 – documentation

Multiple if/switch statements, loops, etc. makes a class or method grow in complexity.

The complexity even grows further, when multiple classes begin to work together.

How to deal with it:

Documentation with diagrams can give a great overview of how a complex class or component (multiple classes) work.

The documentation needs to be accurate, understandable, and up to date.

UML diagrams, flow charts, etc. can extend this level.

Level 2 – live documentation (manual testing)

It becomes challenging to keep documentation up-to-date, when it grows in size. Size can be lowered but will often cost either understand-ability or accuracy.

How to deal with it:

Documentation can be replaced by “live documentation”, which is a set of tests. These tests can be perceived as:

  • Specification (how a method, component, or system is expected to work).
  • Documentation (how a method, component, or system is expected to be used). Any other usage is not supported (but can become).

A change request can apply changes to the live documentation. This change will make the test cases fail, which can be fixed by the developers by updating the product.

The live documentation is always up to date, as long all the test cases pass. Live documentation still needs to be accurate and understandable.

Level 3 – live documentation (automated testing)

When the size of the live documentation grows (the number of tests increases), then the manual testing effort will also increase.

Multiple manual testers can reduce the execution time, but more testers require more planning and better logistics.

Humans also become blind when repeating the same test case over and over again. Complex test cases with many steps can be complicated for humans to perform without missing a steps.

Exploratory testing is excellent but needs to be written down; otherwise, some of the test cases will be forgotten and never used as regression testing.

How to deal with it:

Test automation is the solution.

Unit-tests and unit-integration-tests should be fully automated.

System-tests, system-integration-tests, and end-to-end-tests can be fully automated, but it is better to automate 40 % of the test cases instead of 100 %. The 40 % are cheap low hanging fruits, that will reduce the manual testing effort significantly. The last 20 % will cost many times more than the 80 % combined.

Acceptance testing should only be a few test cases and remain manual. In principle, the acceptance criteria for acceptance testing should only be that all the required tests on lower levels (unit, system, system-integration etc.) have passed.

Level 4 – Optimized automated testing

“Linear scripting” within test automation is a concept that each test case is coded individually without reusing any code. It is easy to write and read, but the maintenance is difficult because a single change in the product might require 300 test cases to be updated manually.

The number of automated test cases will grow over time, which will impact the maintainability of the test cases and slow down the time-to-market of the product.

How to deal with it:

“Structured scripting” is about reusing test automation code, by putting it into libraries, e.g., the page object model.

“Data-driven” is about reusing the same code, by only changing the input parameters and expected output. A valuable low hanging fruit, but not always available.

“Keyword-driven” and “process-driven” tries to make “structured scripting” and “data-driven” more readable for non-technicians such as product owners. An example could be Cucumber with “Given”, “When”, “Then”.

“Model-Based-Testing” is even better, since it can create its own test cases, from a specification. For example, to test a range between a and b, then the model-based-testing will automatically create 4 test cases:

  1. a-1 (below minimum) – this should fail.
  2. a (minimum) – this should succeed.
  3. b (maximum) – this should succeed.
  4. b+1 (above maximum) – this should fail.

This example can be excellent for unit-testing within some programming languages and can merge four unit tests into one.

Level 5 – A test automation product

“Structured scripting”, “Keyword-driven”, “process-driven”, and “Model-Based-Testing” is smart, but also it grows complexity. Changing a single line of code in a library can, for example, influence 300 test cases, so how can we know, that these 300 test cases still work as intended?

We can run the whole test suite to see if any test cases failed in order to fix them. A slow process, because a large test suite can take many hours to run. Imagine:

  1. fixing a few lines of code.
  2. Run the complete test suite for 2 hours.
  3. Fix found errors.
  4. Rerun the test suite for 2 hours.
  5. Fix more errors. Rerun the test suite for another 2 hours.
  6. Etc.
  7. and the maintenance becomes slow as a nightmare.

But this is not even the biggest problem, since we might have broken some test cases so they always pass. These test cases will never test what they were intended to test and the only way to catch those is to go through all the test cases to verify them manually – which is not an option.

A test suite that cannot be trusted is valueless.

How to deal with it:

A test automation setup is a product itself, and just like any other product, it moves through the layers of complexity. When it is simple, it doesn’t need any documentation. When it becomes complex, it needs its own set of unit-tests, system-tests, end-to-end-tests, etc.

Product requires test automation. Test automation requires its own test automation.

We can end up in a situation, where the product requires test automation, and the test automation requires its own test automation. Each step should reduce the complexity.

Read more at: https://bartek.dk/wordpress/avoiding-complexity-and-chaos-2/