Category: Learn TDD

Introduction

The world changes faster and faster.

New frameworks comes out, old frameworks dies.

Sometimes we just need to change from one to another.

How do we do that?

And how can we keep up?

SpecFlow is dead?

Limited Updates: SpecFlow has seen a reduction in active development and fewer updates, as evidenced by the release history on GitHub, with the last stable release in May 2022 and the last beta version in February 2023:

To Upgrading or not to upgrade?

I asked ChatGPT 4o this question:

SpecFlow seems to be dead.
What would the reasons be to upgrade or not to upgrade to ReqnRoll?
Can you write it as short bullet points?

and got the following answer:

Reasons to Upgrade to ReqnRoll
* Active Development: Regular updates and support.
* New Features: Enhanced functionality and tools.
* Active Community: More resources and community support.
* Better Compatibility: Improved integration with modern tools.
* Performance Boost: Faster and more efficient testing.

Reasons Not to Upgrade to ReqnRoll
* Migration Effort: Time-consuming and costly transition.
* Stability: SpecFlow might still meet current needs.
* Integration Issues: Potential disruptions with existing tools.
* Feature Gaps: ReqnRoll may lack some SpecFlow functionalities.
* Learning Curve: Need for retraining and adjustment.

I would like to add that:

• Security Concerns: A test framework is not in production, so lack of security updates is not as big of a problem.
• Proven Stability: The current version has been working for 2 years, so it will probably work for some time yet.
• ReqnRoll is a new tool: Not all features have been migrated from SpecFlow to ReqnRoll. This is covered partially by “Feature Gaps” point.

So, an upgrade is not urgent.

An important question is: “Why fix it, when it ain’t broken?”

Answer: Because everything else is upgraded, like .NET and dependencies. When something finally breaks and we have waited too long, an upgrade might not be possible anymore, and a lot may need to be re-coded. This is a very high risk, even though it is not yet urgent.

How to upgrade an existing test setup

The following approach can be used for

• SpecFlow → ReqnRoll
• SpecFlow → something else
• Something third →something forth

I asked ChatGPT 4o again:

any suggestions on how to upgrade an existing test suite from one framework
to another? Please answer in steps and each step must be short.

It gave me 16 bullet points:

1. Evaluate New Framework:..., 2. Plan Migration: ..., 
3. Set Up Framework: ...,      4. Analyze Tests: ..., 
5. Map Features: ...,          6. Prioritize Tests: ..., 
7. Create Samples: ...,        8. Automate Migration: ..., 
9. Manual Migration: ...,      10. Validate: ..., 
11. Refactor: ...,             12. Update Docs: ..., 
13. Train Team: ...,           14. Integrate CI/CD: ..., 
15. Retire Old Framework: ..., 16.Monitor: ...

I asked:

other things to remember?

And it gave me another list:

17. Backup Original Test Suite: ..., 18. Version Control: ..., 
19. Consistency: ...,                20. Dependencies: ..., 
21. Error Handling: ...,             22. Performance: ..., 
23. Parallel Execution: ...,         24. Reporting: ..., 
25. Stakeholder Communication: ...,  26. Feedback Loop: ..., 
27. Continuous Improvement: ...,     28. Documentation: ..., 
29. Compliance: ...,                 30. Cross-Platform Testing: ..., 
31. Resource Allocation: ...

I used it as a check list and wrote the next three parts:

1. “How to select a new framework”
2. “How to upgrade from one framework to another”
3. “How to train the test team”

How to select a new framework

The best way to understand a new framework is to actually test it.

Make a test that pass, fails, throws an exception.

Make a data driven test, test with reporting, etc.

I have written an article on how to test a test-framework (it is regarding unit-tests, but it can be used for any other tests). https://bartek.dk/unit-testing-the-unit-testing/

The key point here is, the more we use something, the better we understand it.

So, start with the basics and make a pilot migration on the most critical test cases to experience how they perform.

How to upgrade from one framework to another

When we have an older working test suite, then please keep it and use it.

I have learned from experience that a larger upgrade can end up blocking new tasks. Why? Because all the time is used on making the test suite work again.

When a larger update needs to happen, then create a copy of the test suite (be it a new branch, fork, or a new completely new repository.)

Run both the legacy test suite and the new one.

When something breaks in the legacy test suite, then ask yourself: Can I fix it in max. 60 minutes? If yes, then fix it in the old test suite. If not, then migrate the test case to the new test suite and deprecate it in the legacy test suite.

When the test team has extra time (I know: imaginary thinking 😅), then migrate some test cases to the new test suite (start with the most critical ones).

Evaluate automatic migration vs. manual migration. Sometimes a script can migrate 1000’s of test cases in a short time. Sometimes it is simply cheaper to let people manually copy paste, than to make a script work.

Deprecate the old test suite, when it’s not needed anymore.

How to train the test team

Remember to train the team to use the new test suite.

1. Example test cases

Migrate yourself some of the test cases, so they can be inspired from your code.

2. Unit-test of helper classes as live-documentation

Remember to make unit-tests on your helper classes, to show exampled of how the helper classes are supposed to be used (have to already made unit-tests in the legacy test suite, then it will be easy to recreate the new helper classes in the new test suite).

Proper unit-tests can be used as documentation! And also make it easier to migrate from one framework to another.

3. AI to explain code

You can also use AI to help with the training (which I have written an article about: https://bartek.dk/#ai-learning )

Need help?
Click anyone of them to get a guide, on how AI can help you.
(Opens in a new window)

Conclusion

Selecting a framework is done by testing it out
Changing a framework don’t have to be done at once
Training the test team can be done by examples, unit-tests and AI.

Our world is already changing fast.
AI will make it changer even faster.
AI (with AI-learning) will also help us adapt faster.

Music Intro

Congratulations – Lesson complete!

Introduction on how to write a test before code

It can be very difficult to grasp, on how to write a test case before writing code.
Many ask: “How can you test something that doesn’t exist yet?”

It is because we need to view development from a different angle.
Software is the logic that connects the expected inputs with the expected outputs.

In test driven development (TDD) we take one input and output and write them down as a test case. This test case will fail, since there is no code to support it (❌ RED)
Then we can write the needed code/logic to make the test pass (✅ GREEN)

Then we take another input and output and write it down as another test case.
This test case will most likely also fail (❌ RED).
We will then update the code/logic to make the test pass (✅ GREEN).
Then we need to make sure that all the previous test cases also pass. If not, then we need to refactor the code to make them pass (♻️ Refactor)

This is where
❌ RED → ✅ GREEN → ♻️ Refactor
comes from.

Music Intro

Conclusion

Let’s wrap up our journey into Test-Driven Development (TDD), shall we? 🚀

Introduction to Writing Tests Before Code
We have learned that writing test cases before writing code requires a different perspective, viewing development as the logic connecting expected inputs with expected outputs.

TDD Cycle
We have learned the iterative cycle of TDD: writing a test case (❌ RED), writing code to pass the test (✅ GREEN), and then refactoring the code (♻️ Refactor) to ensure all tests pass.

Comparison with Other Methods
We have learned that TDD offers a balanced approach compared to the Waterfall Method and Cowboy Coding, combining the structured planning of Waterfall with the flexibility of Cowboy Coding.

Creating a Minimum Viable Product (MVP)
We have learned that TDD focuses on creating a minimum viable product by starting with the most critical test cases and progressively adding more, ensuring the product evolves based on real requirements.

Example: Calendar Application
We have learned how to implement a calendar application using TDD, starting with simple date handling and progressively adding more complex requirements like boundary values, negative tests, and special cases like leap years.

Boundary-Value Analysis
We have learned to use boundary-value analysis to test minimum and maximum values, ensuring our application handles edge cases correctly.

Negative Testing
We have learned to extend our tests with negative cases, such as below minimum and above maximum values, to ensure the application correctly handles invalid inputs.

Extending with Exceptions
We have learned to handle special cases, like different month lengths and leap years, by extending our test cases and updating the implementation accordingly.

Critical Considerations
We have learned the importance of identifying and prioritizing critical functionalities, and considering the scope and longevity of the application when deciding which test cases to implement.

Refactoring
We have learned to refactor our code to make it more understandable and maintainable, ensuring all test cases pass after each change.

Ensuring Comprehensive Coverage
We have learned to ask critical questions about the application’s requirements and to extend our tests as necessary, ensuring comprehensive coverage of all important functionalities.

More Features
We have explored additional features, like getting today’s date, based on critical needs.
It’s a risk based testing approach.

Rapid Prototyping
We have learned that knowledge in coding is not binary; it’s a spectrum. TDD allows us to experience and refine our understanding one test at a time, making the development process clearer with each test. This approach can be seen as “Rapid Prototyping,” where each new test case is a tiny prototype that needs validation. Or, as humorously mentioned, “Minimum Viable Waterfalls” 😂😂😂.

Congratulations – Lesson complete!

Introduction to Unit Testing

So, what exactly is Unit Testing?

When we need to develop code, we first determine the inputs our code will receive and the outputs it should produce. Then, we write the required logic to process these inputs into the desired outputs.

A unit test is essentially a description of these inputs and the expected outputs. By writing unit tests, we can verify that the piece of code we are building fulfills these expectations. It’s like setting up mini-experiments to ensure each part of our program behaves correctly.

Unit testing ensures our code is reliable and maintainable, providing a solid foundation for our software projects. It’s like having a safety net that catches errors before they cause trouble, giving us peace of mind and making our coding adventures much smoother!

Music Intro

Needed tools

With Python we can use PyTest or RobotFramework.
With Groovy we can use Spock.
With JavaScript we can use Mocha and Chai.
With C# I have no idea, but ChatGPT suggest: NUnit and xUnit.

The test framework is not that important, the most important part is to use it wisely.

Often it is better to use a simple test-framework and add new features others have developed or develop them yourself.

How to learn a test framework?

Testing is the basis for knowledge and one of the best ways to learn something.
We simply make small tests (experiments) to see what, what succeeds, what fails, and what can be learned from it.

In the next steps I will give you some of these tests, so you can learn the test-framework you want in any coding language you want.

We will take it one test case at a time, just like we would develop with Test Driven Development :-)

I will first show the first unit-test in Python with pytest, and then show how AI can translate it to JavaScript with Mocha and Chai, then into Groovy and Spock.

Test 4: Let’s automate the passed, failed, and crashed test

Testing how something works is a great way to learn.
We test how things work, and how they fail so we can recover better from the fails.

To remember or share our knowledge, we can write it down as documentation.
A great way is to do documentation as unit-tests, because we can run them.

When all the unit-tests passes, then the documentation is up-to-date.
When a unit-test fails/crashes, then the documentation needs to be updated.

This is called live-documentation, because the documentation is alive and evolving with the system under development.

import pytest

def toBeTested_mustPass():
    assert 1+2==3, "this test must pass, but it passed :("

import pytest

def toBeTested_mustPass():
    assert 1+2==3, "this test must pass, but it passed :("

def test_toBeTested_mustPass():
    toBeTested_must_pass()
    assert True

test.py .                                                           [100%]
========================== 1 passed in 0.00s =============================

import pytest

def toBeTested_mustFail():
    assert 1+2==4, "this test must fail, and it did :)"

def test_toBeTested_mustPass():
    toBeTested_mustFail() # we changed this line from pass to fail
    assert True

====================== short test summary info ===========================
FAILED test.py::test_toBeTested_mustPass - AssertionError: this test must fail, and it did :)
========================== 1 failed in 0.01s =============================

import pytest

def toBeTested_mustFail():
    assert 1+2==4, "this test must fail, and it did :)"

def test_toBeTested_mustFail():
    # Given 
    errorMessage = None

    # When 
    try:
        toBeTested_must_fail()
    except AssertionError as e:
        errorMessage = str(e)

    # Then
    assert errorMessage == "this test must fail, and it did :)\nassert (1 + 2) == 4"

test.py .                                                           [100%]
========================== 1 passed in 0.00s =============================

import pytest

def toBeTested_mustFail():
    assert 1+2==4, "this test must fail, and it did :)"

def test_toBeTested_mustFail():
    with pytest.raises(AssertionError, match=r"this test must fail, and it did :\)\nassert \(1 \+ 2\) == 4"):
        toBeTested_must_fail()

When the function is called and expected to fail with an AssertionError
Then the error message must match the expected message

The function is called and expected to fail with an AssertionError, and the exception message should match the expected regular expression

def toBeTested_mustCrash():
    raise RuntimeError("CRASH! ;-)")
    assert 1+2==3, "this must crash, but it failed :("

def test_toBeTested_mustCrash():
    # given
    errorMessage = None

    # when
    try:
        toBeTested_must_crash()
    except RuntimeError as e:
        errorMessage = str(e)
        
    # then
    assert errorMessage == "CRASH! ;-)"

test.py .                                                           [100%]
========================== 1 passed in 0.00s =============================

Data driven testing

Sometimes it is better to have a single test that is data driven, than to have multiple tests.

The balance between them is the readability of the test.

Let’s try to make the 3 first test (pass, fail, crash) data driven.

The original tests looked like this:

import pytest

def test_mustPass():
    assert 1+2==3, "this test must pass, but it passed :("

def test_mustFail():
    assert 1+2==4, "this test must fail, and it did :)"

def test__mustCrash():
    raise RuntimeError("CRASH! ;-)")
    assert 1+2==3, "this must crash, but it failed :("

Which can be made into:

import pytest
from dataclasses import dataclass, field

@dataclass
class TestCase:
    name: str
    input: int
    expected: int
    raisesException: bool

testCases = [
    TestCase(name="1 must_pass", input=1+2, expected=3, raisesException=False),
    TestCase(name="2 must_fail", input=1+2, expected=4, raisesException=False),
    TestCase(name="3 must_crash", input=1+2, expected=4, raisesException=True),
]

ids = [testCase.name for testCase in testCases]
params = [testCase for testCase in testCases]

@pytest.mark.parametrize("testData", params, ids=ids)
def test_cases(testData:TestCase):
     # when
    if testData.raisesException:
        raise RuntimeError("CRASH! ;-)")               
    else:
        assert testData.input==testData.expected, "this test must fail, and it did :)"

====================== short test summary info ===========================
FAILED test.py::test_cases[2 must_fail] - AssertionError: this test must fail, and it did :)
FAILED test.py::test_cases[3 must_crash] - RuntimeError: CRASH! ;-)
===================== 2 failed, 1 passed in 0.02s ========================

I like to use something called @dataclass that we can build test cases from and is supported by autocomplete!

...
from dataclasses import dataclass, field

@dataclass
class TestCase:
    name: str
    input: int
    expected: int
    raisesException: bool
...

Then I can define my test cases as a list of TestCase objects:

...
testCases = [
    TestCase(name="1 must_pass", input=1+2, expected=3, raisesException=False),
    TestCase(name="2 must_fail", input=1+2, expected=4, raisesException=False),
    TestCase(name="3 must_crash", input=1+2, expected=4, raisesException=True),
]
...

Then it transforms the testCases into something pytest understands:

...
ids = [testCase.name for testCase in testCases]
params = [testCase for testCase in testCases]

@pytest.mark.parametrize("testData", params, ids=ids)
def test_cases(testData:TestCase):
    if testData.raisesException:
        raise RuntimeError("CRASH! ;-)")               
    else:
        assert testData.input==testData.expected, "this test must fail, and it did :)"

Making unit-tests understandable

It is really important to try to form the test cases, so it is easy to understand them.

A test case that is not understandable is valueless 
and we would be better without it.

If we take the example from previous chapter:

import pytest
from dataclasses import dataclass, field

@dataclass
class TestCase:
    name: str
    input: int
    expected: int
    raisesException: bool

testCases = [
    TestCase(name="1 must_pass", input=1+2, expected=3, raisesException=False),
    TestCase(name="2 must_fail", input=1+2, expected=4, raisesException=False),
    TestCase(name="3 must_crash", input=1+2, expected=4, raisesException=True),
]

ids = [testCase.name for testCase in testCases]
params = [testCase for testCase in testCases]

@pytest.mark.parametrize("testData", params, ids=ids)
def test_cases(testData:TestCase):
     # when
    if testData.raisesException:
        raise RuntimeError("CRASH! ;-)")               
    else:
        assert testData.input==testData.expected, "this test must fail, and it did :)"

Then it is longer than the following example:

import pytest

@pytest.mark.parametrize(
   "i, e, rte", [
   (1+2, 3, False),  # this test must pass
   (1+2, 4, False),  # this test must fail
   (1+2, 3, True)  # this test must crash
    ],
    ids=["must_pass", "must_fail", "must_crash"]
)
def test_cases(i, e, rte):
    if rte:
        raise RuntimeError("CRASH! ;-)")               
    else:
        assert i==e, "this test must fail, and it did :)"

Except, it is much harder to understand and maintain:

• What does i, e, and rte mean?
• the ids and the comments needs to be paired and updated manually.
• The parameters can figured out, but with 7+ parameters, different value lengths, and 10+ test cases would make this hell to maintain!

So, please use:

• Readable parameter names like: number1 + number2 == result and not n1+n2==r
• Use group parameters into Inputs and Expected, so it is easy to understand what transformation needs to be done (not necessary how it is done) (In Python we can use @dataclasses
• Use test id’s/description to easier navigate which tests has failed.
• Try to create the context the test case needs to work within. Then it will be easier to understand, why something works the way it works.
• Additionally use Skipped/Ignored category in case you want a test case skipped (described in Skipping tests)

Skipping tests

Sometimes we can find a bug, and a test will fail.

There is a dangerous question to ask, that many have opinions about.

Imagine a tests starts to fail, because of a bug. What should we do?

• Fix the bug as soon as possible!
• Let the test fail, until it is fixed.?
• Mark the test with @Ignore and a Jira-bug, so we can fix it soon.

I have tried multiple approaches and all of them has a price.

• When it is not possible to fix all bugs, then we fix only the most critical ones.
• When we are not in production yet, then a bug might be critical, but not urgent.
• When we see a red test report, then we can get used to the red color. Non-critical bugs can make it impossible to see the critical ones.

So, a @Ignore of @skip function can be a good thing, as long as we remember to give it a comment or a link to a jira-story/bug.

In Pytest we can skip tests with:

import pytest


def test_mustPass():
    assert 1+2==3, "this test must pass, but it passed :("

@pytest.mark.skip(reason="Jira: bug-101")
def test_mustFail():
    assert 1+2==4, "this test must fail, and it did :)"

@pytest.mark.skip(reason="Jira: bug-102")
def test_mustCrash():
    raise RuntimeError("CRASH! ;-)")
    assert 1+2==3, "this must crash, but it failed :("

test.py .                                                           [100%]
===================== 1 passed, 2 skipped in 0.01s =======================

Then we can always read about the bug and status in the Jira bug.

A data driven test can be skipped a little differently, by
adding a property to the TestCase called skip (line 10),
assign it to each TestCase (line 13-15)
and then add the skip option to the implementation (line 23-24):

import pytest
from dataclasses import dataclass, field

@dataclass
class TestCase:
    name: str
    input: int
    expected: int
    raisesException: bool
    skip: str

testCases = [
    TestCase(name="1 must_pass", input=1+2, expected=3, raisesException=False, skip=None),
    TestCase(name="2 must_fail", input=1+2, expected=4, raisesException=False, skip="Jira: bug-101"),
    TestCase(name="3 must_crash", input=1+2, expected=4, raisesException=True, skip="Jira: bug-102"),
]

ids = [testCase.name for testCase in testCases]
params = [testCase for testCase in testCases]

@pytest.mark.parametrize("testData", params, ids=ids)
def test_cases(testData:TestCase):
    if(testData.skip):
        pytest.skip(testData.skip)

    if testData.raisesException:
        raise RuntimeError("CRASH! ;-)")               
    else:
        assert testData.input==testData.expected, "this test must fail, and it did :)"

test.py .                                                           [100%]
===================== 1 passed, 2 skipped in 0.02s =======================

Reporting

Sometimes it can be a great idea to make a test report in html.
It can be easier to get an quick overview or navigate through the test cases easier.
For more complex tests it can also create a better overview in a visual day.

A table like this:

TestCase(test=1, product="A", validFrom=today(days=-2), validTo=today(days=-1), expectedValidity=TRUE)
TestCase(test=2, product="B", validFrom=today(days=-2), validTo=today(),        expectedValidity=FALSE)
TestCase(test=3, product="C", validFrom=today(),        validTo=today(),        expectedValidity=FALSE)
TestCase(test=4, product="D", validFrom=today(),        validTo=today(days=2),  expectedValidity=TRUE)
TestCase(test=5, product="E", validFrom=today(days=1),  validTo=today(days=2),  expectedValidity=FALSE)

Can be reformed into a graph like this:

Which is much more readable.
Especially when the complexity grows and we add i.e. timezones.

I will not go much into reporting in this chapter, but will write a separate one, which will contain all kind of good ideas, incl. testing of report templates.

Conclusion

Let’s wrap up our journey into Test-Driven Development (TDD), shall we? 🚀

Needed tools
There are many programming languages and even more test-frameworks. To compare them better, it is recommended to test them out.

How to Learn a Test Framework
We have learned that creating small, incremental tests helps us understand the outcomes and build knowledge of the test framework, similar to the principles of Test-Driven Development (TDD).

Test 1: To Make It Pass
We have learned how to set up a basic test that is designed to pass, involving the installation of the framework, writing a simple test, and ensuring it runs successfully.

Test 2: To Make It Fail
We have learned the importance of including a test case meant to fail, as it helps us understand how the test framework handles and reports failures.

Test 3: To Make It Crash
We have learned to create a test that raises an exception to simulate a crash, which helps distinguish between assertion errors and runtime errors in the test results.

Automating Tests
We have learned to automate tests by renaming methods to avoid automatic execution, using wrapper tests to verify behavior, and ensuring non-crashing tests pass successfully.

Containing Failures and Crashes
We have learned to handle expected errors using try-except blocks (or try-catch in other languages) to manage assertion errors and runtime exceptions effectively.

Data-Driven Testing
We have learned to consolidate multiple tests into a single parameterized test using @dataclass to define test cases, which enhances readability and maintainability.

Making Tests Understandable
We have learned the importance of clear and maintainable tests by using descriptive parameter names, grouping inputs and expected results, and avoiding cryptic variable names.

Skipping Tests
We have learned to mark tests that should not run due to known issues with @skip or @Ignore, and to provide reasons or links to bug tracking systems for reference.

Reporting
We have learned the value of HTML reports for better visual representation and navigation of test results, especially useful in more complex scenarios.

Congratulations – Lesson complete!