Layers of Complexity and when to apply testing
Introduction
Testing helps with complexity, but need to be applied at the right situation, to deal with complexity, without adding increasing it unnecessary.
Complexity level 0 – clarity
The code is simple and easy to read.
How to deal with it:
No need to test it and no documentation needed.
Avoid comments – because you will need to maintain both the code and the comments. Outdated comments confuse more than help.
A single comment can sometimes be acceptable, when a single line of code is unintuitive, because of poorly implemented method in a framework or legacy code.
Clean code, SOLID principles, linting, etc. can help extend this level.
Level 1 – documentation
Multiple if/switch statements, loops, etc. makes a class or method grow in complexity.
The complexity even grows further, when multiple classes begin to work together.
How to deal with it:
Documentation with diagrams can give a great overview of how a complex class or component (multiple classes) work.
The documentation needs to be accurate, understandable, and up to date.
UML diagrams, flow charts, etc. can extend this level.
Level 2 – live documentation (manual testing)
It becomes challenging to keep documentation up-to-date, when it grows in size. Size can be lowered but will often cost either understand-ability or accuracy.
How to deal with it:
Documentation can be replaced by “live documentation”, which is a set of tests. These tests can be perceived as:
- Specification (how a method, component, or system is expected to work).
- Documentation (how a method, component, or system is expected to be used). Any other usage is not supported (but can become).
A change request can apply changes to the live documentation. This change will make the test cases fail, which can be fixed by the developers by updating the product.
The live documentation is always up to date, as long all the test cases pass. Live documentation still needs to be accurate and understandable.
Level 3 – live documentation (automated testing)
When the size of the live documentation grows (the number of tests increases), then the manual testing effort will also increase.
Multiple manual testers can reduce the execution time, but more testers require more planning and better logistics.
Humans also become blind when repeating the same test case over and over again. Complex test cases with many steps can be complicated for humans to perform without missing a steps.
Exploratory testing is excellent but needs to be written down; otherwise, some of the test cases will be forgotten and never used as regression testing.
How to deal with it:
Test automation is the solution.
Unit-tests and unit-integration-tests should be fully automated.
System-tests, system-integration-tests, and end-to-end-tests can be fully automated, but it is better to automate 40 % of the test cases instead of 100 %. The 40 % are cheap low hanging fruits, that will reduce the manual testing effort significantly. The last 20 % will cost many times more than the 80 % combined.
Acceptance testing should only be a few test cases and remain manual. In principle, the acceptance criteria for acceptance testing should only be that all the required tests on lower levels (unit, system, system-integration etc.) have passed.
Level 4 – Optimized automated testing
“Linear scripting” within test automation is a concept that each test case is coded individually without reusing any code. It is easy to write and read, but the maintenance is difficult because a single change in the product might require 300 test cases to be updated manually.
The number of automated test cases will grow over time, which will impact the maintainability of the test cases and slow down the time-to-market of the product.
How to deal with it:
“Structured scripting” is about reusing test automation code, by putting it into libraries, e.g., the page object model.
“Data-driven” is about reusing the same code, by only changing the input parameters and expected output. A valuable low hanging fruit, but not always available.
“Keyword-driven” and “process-driven” tries to make “structured scripting” and “data-driven” more readable for non-technicians such as product owners. An example could be Cucumber with “Given”, “When”, “Then”.
“Model-Based-Testing” is even better, since it can create its own test cases, from a specification. For example, to test a range between a and b, then the model-based-testing will automatically create 4 test cases:
- a-1 (below minimum) – this should fail.
- a (minimum) – this should succeed.
- b (maximum) – this should succeed.
- b+1 (above maximum) – this should fail.
This example can be excellent for unit-testing within some programming languages and can merge four unit tests into one.
Level 5 – A test automation product
“Structured scripting”, “Keyword-driven”, “process-driven”, and “Model-Based-Testing” is smart, but also it grows complexity. Changing a single line of code in a library can, for example, influence 300 test cases, so how can we know, that these 300 test cases still work as intended?
We can run the whole test suite to see if any test cases failed in order to fix them. A slow process, because a large test suite can take many hours to run. Imagine:
- fixing a few lines of code.
- Run the complete test suite for 2 hours.
- Fix found errors.
- Rerun the test suite for 2 hours.
- Fix more errors. Rerun the test suite for another 2 hours.
- Etc.
- and the maintenance becomes slow as a nightmare.
But this is not even the biggest problem, since we might have broken some test cases so they always pass. These test cases will never test what they were intended to test and the only way to catch those is to go through all the test cases to verify them manually – which is not an option.
A test suite that cannot be trusted is valueless.
How to deal with it:
A test automation setup is a product itself, and just like any other product, it moves through the layers of complexity. When it is simple, it doesn’t need any documentation. When it becomes complex, it needs its own set of unit-tests, system-tests, end-to-end-tests, etc.
We can end up in a situation, where the product requires test automation, and the test automation requires its own test automation. Each step should reduce the complexity.
Read more at: https://bartek.dk/wordpress/avoiding-complexity-and-chaos-2/
About The Author
