Types of Behavior Driven Development (BDD)

Process-driven automation

Introduction

Many test-automation-frameworks support Behavior Driven Development (BDD), but they differ in how they apply BDD.

There are at least three layers of BDD:

  • Keyword Driven BDD
  • Process Driven BDD
  • Model-Based Testing

Both Keyword and Process-driven will be explored in this article, while Model-Based Testing will come out next week.

Before BDD

Linear scripting is about hard coding each operational test step in a test case. Tools for capture and replay of user actions is just to visually hard code the test cases.

Structured scripting is about reusing the most used operational test steps by putting them in support libraries. 300 test will be updated with a single update in a support library, instead of reapplying the same change 300 times manually.

Data-driven automation runs the same test case multiple times, but with different parameters. It is very cheap to maintain.

Keyword-driven BDD (first layer)

The lowest version of BDD.

Keyword-driven automation is almost the same as structured scripting since it reuses the operational steps for multiple test cases. The difference is that the language is more readable in keyword-driven.

Structured scripting

A test case with structured scripting can look like this:

Browser.navigate("https://...com/webshop")
PageGeneral.cookiesButton.click()
pageTopMenu.tickets.click()
pageTickets.buyButton(click()
pageTickets.dropdown.select("adult")
pageTickets.continueButton.click()
pageZones.zoneSlider.slideTo("7")
pageZones.continueButton.click()
pageSpecialOffer.rejectButton.click()
assert( pageTickets.price.getText(), "18 €", "ERROR: price is incorrect")

All the operational steps can be reused in many test cases, and are categorized in this example with the Page Object Model.

Keyword-driven

A Keyword-driven will be much easier to read because it uses action keywords:

Navigate to "https://...com/webshop"
Click the tab "tickets"
Click button "Buy tickets"
Select dropdown "customer type" option "adult"
Click the "Continue" button
Move the slider to 7 zones
Click the "Continue" button
Click "Reject" button on the special offer window
Verify the price of 18 €

It can also use Gherkin keywords to split it into Preconditions (Given), Actions (when), and Assertions (Then):

Given the browser starts on https://...com/webshop
And accept the "cookies information"

When clicking the tab "tickets"
And clicking button "Buy tickets"
And selecting dropdown "customer type" option "adult"
And clicking the "Continue" button
And moving the slider to 7 zones
And clicking the "Continue" button
And rejecting the special offer window

Then the price must be 18 €

Process-driven (second layer)

Keyword-driven automation is describing the operational steps in a test case, while process-driven automation describes the business steps in a test case.

A test case written with process-driven automation has more focus on the needs than on the details. Therefore much easier to read for the business and see what the test case is actually testing

Given I am buying a ticket on the web shop
When buying a <child / adult / student / retired> ticket
And to travel <2 - 8> zones
Then the price should be <price>

The operational steps are hidden within the business steps:

Given I am buying a ticket on the web shop
Navigate to "https://...com/webshop"
When buying a <child / adult / student / retired> ticket
Click the tab "tickets"
Click button "Buy tickets"
Select dropdown "customer type" option "adult"
Click the "Continue" button
And to travel <2 - 8> zones
Move the slider to 7 zones
Click the "Continue" button
Click "Reject" button on the special offer window
Then the price should be <price> €
Verify the price of 18 €

Model-Based Testing (third layer)

Are you ready for the next level?

Coming soon.

Automation frameworks

The following automation frameworks support process-driven automation:

Please add any other frameworks in the comments

Write better specifications with testing

logical test case and physical test case

Introduction

A specification written by the customer – often ends as incomplete, fragmented and with parts that contradict each other – makes it difficult to build for the engineers.

A specification written by the engineer – often ends with a focus on details than on needs, and to difficult for the business to read/understand.

So how to do it? By using Given, When, Then!

Understandable

Lego and IKEA have some of the best manuals/documentation, because they write them as test cases using “Given, When, Then”, mentioned in Write better documentation with testing

Given is don't start without it. When is what to do. Then is what to expect.

Documentation is about:

  • What givens do I need to have
  • and what when’s do I need to do
  • to achieve my then’s (goals)”

A specification is almost the same, but with a different focus:

  1. How do I build the given’s,
  2. so the users can do the when’s 
  3. to achieve their then’s (goals)

We think of specification and documentation as two different groups of documents, but they can technically be the same documents – if they are written as test cases.

Focus on needs than details

It is easy to forget the needs when the attention is on the details (read my 10 facts about details). In testing, there is something called a logical and physical test case, to split the design from the implementation.

Physical test case (operational steps)

A physical test case describes all the operational steps (given, when, then) to complete the test:

Given I open https://...com/webshop
And accept the "cookies information"
When I click the tab "tickets"
And I click button "Buy tickets"
And select dropdown "customer type" option "adult"
And click the "Continue" button
And move the slider to 7 zones
And click the "Continue" button
And rejects the special offer window
Then I should see the price of 18 €

The test case drowns in the operational details, and it becomes difficult to read, what the test case is actually testing.

Logical test case (business process)

A logical test case describes the business behavior (given, when, then) to only focus on what the test is about, and not on the implementation itself:

Given I am buying a ticket on the web shop
When buying a <child / adult / student / retired> ticket
And to travel <2 - 8> zones
Then the price should be <price> €

The logical test case describes the business rules (behavior), is easy to understand, and can be reused whenever a new version of the product needs to be implemented.

Logical test cases make it easier for the customer and supplier to understand each other.

Defragmented

We all know the situation when a developer and a customer agree verbally on something, and it gets implemented. Nobody wrote it down, so it will never end up in the specification and never in the documentation.

In Behavior Driven Development (BDD), the logical test case is written before the implementation. It might seem counter-intuitive, for how can we test something that doesn’t exist?

Logical test cases are our assumption of what the product should do, and can be used to measure the status of the implementation, and later be used as documentation.

Without contradictions

Sometimes two business processes can contradict each other and nobody will notice.

  1. When the first process is implemented, then its test case will pass.
  2. When the second process is implemented, then its test case will pass, but also make the first test case fail.
  3. Fixing the first process, so its test case doesn’t fail anymore, will make the second test case fail.
  4. Fixing the second process, so its test case doesn’t fail anymore, will make the first test case fail.
  5. etc..

A specification written with test cases is executable and can catch contradictions in business processes.

Completeness

The customer can put the logical test cases in their end-to-end business flows, to see if every critical behavior is mapped. Should only be applied for critical behavior and not every behavior).

Missed or misunderstood business behaviors can always be added or updated on a later stage as a change request.

The specification will not be complete to begin with, but will become more and more complete over time.

It can later be used as documentation and reused as a specification when a new version of the product needs to be implemented.

Write better documentation with testing

Given is don't start without it. When is what to do. Then is what to expect.

Introduction

Documentation is “How a product is expected to be used” and needs to be accurateunderstandable, and up-to-date.

Documentation that is:

  • Accurate can become long and cryptic (bad for understanding and updating)
  • Often updated, can become fluffy and fragmented (bad for accuracy and understanding)
  • Understandable can become long and shallow (bad updating and or accuracy)

So how to do it? By using Given, When, Then!

Make it Understandable

Lego and IKEA have some of the best manuals / documentation, because they write them as test cases using “Given, When, Then”:

Given is don't start without it.
When is what to do.
Then is what to expect.

Given – don’t start building without these elements.
When – what actions you are required to do (and in what order)
Then – what you should expect of the end result.

Documentation written this way is easy to understand.

Make it Accurate

“Given, When, Then” is a test case to describe product behavior (read: Definition of a test case)

A good test case describes only a single product behavior, which makes it accurate. A bad test case will describe multiple behaviors at once and be confusing.

A set of good test cases is called a test suite, and will describe a set of behaviors called a feature.

Documentation written this way is accurate.

Make it Always up-to-date

Documentation written as test cases, makes the documentation testable.

Given the product has changed. When the test cases and pass, then the documentation is up-to-date:

  • given the product has changed
  • when some test cases (documentation) fail,
  • then the test cases (documentation) needs to be updated.

Documentation written this way
is called “Living Documentation”,
because it is always up-to-date.

Putting back the “engineer” in “software engineer”

Bad craftsmen don't test. God craftsmen test their product Engineers test also the foundation of the product

Foreword

What is the difference between a “software developer” and a “software engineer”?

I understand that titles such as “software developer” and “software engineer” have been overused so much that they can mean almost anything today.

Knowing the difference is essential, to know what value an engineer brings.

The craftsman

A craftsman can build a solution.

A good craftsman will also test if the solution works and try to make it work.

Software developers are craftsmen.

The engineer

An engineer will also test how well the solution works.

It requires the engineer to know: “what the solution is supposed to do” and not only: “what the solution is doing”.

Engineers also test the tools, methods, materials, etc., like bricks in construction, in order to use the optimal one: Absorption test, Crushing strength test, Hardness test, etc.

Bad craftsmen don't test. God craftsmen test their product Engineers test also the foundation of the product

Therefore, a software engineer needs skills in both “software development” and “testing”, which enables proper modelling and prototyping.

The software engineer also needs to know how to test the tools, frameworks, algorithms, coding language, etc. to make a baseline of them.

Also read:

Layers of Complexity and when to apply testing

Product requires test automation. Test automation requires its own test automation.

Introduction

Testing helps with complexity, but need to be applied at the right situation, to deal with complexity, without adding increasing it unnecessary.


Complexity level 0 – clarity

The code is simple and easy to read.

How to deal with it:

No need to test it and no documentation needed.

Avoid comments – because you will need to maintain both the code and the comments. Outdated comments confuse more than help.

A single comment can sometimes be acceptable, when a single line of code is unintuitive, because of poorly implemented method in a framework or legacy code.

Clean code, SOLID principles, linting, etc. can help extend this level.


Level 1 – documentation

Multiple if/switch statements, loops, etc. makes a class or method grow in complexity.

The complexity even grows further, when multiple classes begin to work together.

How to deal with it:

Documentation with diagrams can give a great overview of how a complex class or component (multiple classes) work.

The documentation needs to be accurate, understandable, and up to date.

UML diagrams, flow charts, etc. can extend this level.


Level 2 – live documentation (manual testing)

It becomes challenging to keep documentation up-to-date, when it grows in size. Size can be lowered but will often cost either understand-ability or accuracy.

How to deal with it:

Documentation can be replaced by “live documentation”, which is a set of tests. These tests can be perceived as:

  • Specification (how a method, component, or system is expected to work).
  • Documentation (how a method, component, or system is expected to be used). Any other usage is not supported (but can become).

A change request can apply changes to the live documentation. This change will make the test cases fail, which can be fixed by the developers by updating the product.

The live documentation is always up to date, as long all the test cases pass. Live documentation still needs to be accurate and understandable.


Level 3 – live documentation (automated testing)

When the size of the live documentation grows (the number of tests increases), then the manual testing effort will also increase.

Multiple manual testers can reduce the execution time, but more testers require more planning and better logistics.

Humans also become blind when repeating the same test case over and over again. Complex test cases with many steps can be complicated for humans to perform without missing a steps.

Exploratory testing is excellent but needs to be written down; otherwise, some of the test cases will be forgotten and never used as regression testing.

How to deal with it:

Test automation is the solution.

Unit-tests and unit-integration-tests should be fully automated.

System-tests, system-integration-tests, and end-to-end-tests can be fully automated, but it is better to automate 40 % of the test cases instead of 100 %. The 40 % are cheap low hanging fruits, that will reduce the manual testing effort significantly. The last 20 % will cost many times more than the 80 % combined.

Acceptance testing should only be a few test cases and remain manual. In principle, the acceptance criteria for acceptance testing should only be that all the required tests on lower levels (unit, system, system-integration etc.) have passed.


Level 4 – Optimized automated testing

“Linear scripting” within test automation is a concept that each test case is coded individually without reusing any code. It is easy to write and read, but the maintenance is difficult because a single change in the product might require 300 test cases to be updated manually.

The number of automated test cases will grow over time, which will impact the maintainability of the test cases and slow down the time-to-market of the product.

How to deal with it:

“Structured scripting” is about reusing test automation code, by putting it into libraries, e.g., the page object model.

“Data-driven” is about reusing the same code, by only changing the input parameters and expected output. A valuable low hanging fruit, but not always available.

“Keyword-driven” and “process-driven” tries to make “structured scripting” and “data-driven” more readable for non-technicians such as product owners. An example could be Cucumber with “Given”, “When”, “Then”.

“Model-Based-Testing” is even better, since it can create its own test cases, from a specification. For example, to test a range between a and b, then the model-based-testing will automatically create 4 test cases:

  1. a-1 (below minimum) – this should fail.
  2. a (minimum) – this should succeed.
  3. b (maximum) – this should succeed.
  4. b+1 (above maximum) – this should fail.

This example can be excellent for unit-testing within some programming languages and can merge four unit tests into one.


Level 5 – A test automation product

“Structured scripting”, “Keyword-driven”, “process-driven”, and “Model-Based-Testing” is smart, but also it grows complexity. Changing a single line of code in a library can, for example, influence 300 test cases, so how can we know, that these 300 test cases still work as intended?

We can run the whole test suite to see if any test cases failed in order to fix them. A slow process, because a large test suite can take many hours to run. Imagine:

  1. fixing a few lines of code.
  2. Run the complete test suite for 2 hours.
  3. Fix found errors.
  4. Rerun the test suite for 2 hours.
  5. Fix more errors. Rerun the test suite for another 2 hours.
  6. Etc.
  7. and the maintenance becomes slow as a nightmare.

But this is not even the biggest problem, since we might have broken some test cases so they always pass. These test cases will never test what they were intended to test and the only way to catch those is to go through all the test cases to verify them manually – which is not an option.

A test suite that cannot be trusted is valueless.

How to deal with it:

A test automation setup is a product itself, and just like any other product, it moves through the layers of complexity. When it is simple, it doesn’t need any documentation. When it becomes complex, it needs its own set of unit-tests, system-tests, end-to-end-tests, etc.

Product requires test automation. Test automation requires its own test automation.

We can end up in a situation, where the product requires test automation, and the test automation requires its own test automation. Each step should reduce the complexity.

Read more at: https://bartek.dk/wordpress/avoiding-complexity-and-chaos-2/