Table of Contents
- Musical Intro
- Introduction
- Using AI
- Learn from examples (prompt):
- More examples? (prompt):
- Sometimes you might need to start a new session.
- What usages are there? (prompt):
- Making a BDD scenario from a specific test case (prompt):
- Common mistakes (prompt):
- Training exercises
- Advances analysis (prompt):
- Building examples with the AI
- Example 1: Date ranges
- Example 2: Time (regular, sommer time)
- Example 3: Special characters (formatting)
- Example 4: Color profiles
- Example 5: Repeats (maximum)
- Example 6: Multi-threading (minimum) for performance testing
- Conclusion
- Want more?
Musical Intro
Introduction
Imagine you’re playing Minecraft, and you need to move as close to the edge as possible, to set the next block, without falling off the platform. Boundary-Value Analysis (BVA) in software testing is like that, where we find out where the edge of not-falling and falling is.
Example:
Scenario Outline: Withdraw boundary-adjacent values
Given the user has an active bank account
When the user tries to withdraw <amount> dollars
Then the withdrawal should be <result>
Examples:
| amount | result | comment
| 499 | successful | # Just below maximum boundary
| 500 | successful | # Maximum boundary
| 501 | rejected | # Just above maximum boundaryUsing AI
All this have been created with the cooperation with AI.
Often it can be more useful just to learn, what is needed in the moment, than to learn everything about a topic. This is where AI is perfect.
We can use the free ChatGPT 3.5 or 4o to learn about operators. You are welcome to try other and if you do, please write a comment!
Lets look at essential prompts!
Learn from examples (prompt):
I want to learn about "Boundary-value-analysis".
Can you give me an example written with BDD examples?
Boundary Value Analysis (BVA) is a testing technique used to identify errors at the boundaries of input domains rather than within the ranges. The idea is to focus on the edges of the input ranges where errors are most likely to occur. This is particularly useful in situations where values are input as a range or limit.
To demonstrate Boundary Value Analysis using Behavior-Driven Development (BDD), let's take an example of a banking system that has the following constraints for account balance
Scenario Outline: Withdraw boundary-adjacent values
Given the user has an active bank account
When the user tries to withdraw <amount> dollars
Then the withdrawal should be <result>
Examples:
| amount | result |
| 9 | rejected | # Just below minimum boundary
| 10 | successful | # Minimum boundary
| 11 | successful | # Just above minimum boundary
| 499 | successful | # Just below maximum boundary
| 500 | successful | # Maximum boundary
| 501 | rejected | # Just above maximum boundaryIt is understandable, why we need 9 (below minimum), 10 (minimum), 500 (maximum), and 501 (above maximum). It is not as intuitive to test for 11 (above minimum) and 499 (below maximum).
The ChatGPT 4o, 3.5, and even 4 can’t answer it properly. It just give a lot of explanation that it needs to be, but can’t make a proper why.
An experienced tester would explain it as following:
Example 1, working:
When a code is supposed to be x >= 10, and is coded that way.
Then the test results will be:
Test 1: When x is set to 9, then it must be invalid (passed ✔️)
Test 2: When x is set to 10, then it must be valid (passed ✔️)
Example 2, bug:
When a code is supposed to be x >= 10, but is coded as x>10
Then the test results will be:
Test 1: When x is set to 9, then it must be invalid (passed ✔️)
Test 2: When x is set to 10, then it must be valid (failed ❌)
Example 3, bug:
When a code is supposed to be x >= 10, but is coded as x==10
Then the test results will be:
Test 1: When x is set to 9, then it must be invalid (passed ✔️)
Test 2: When x is set to 10, then it must be valid (passed ✔️)
Test 3: When x is set to 11, then it must be invalid (failed ❌)
Which means we need the 3rd test, in order to catch this kind of bug
More examples? (prompt):
(I have picked the best out)
Can you give more examples?Scenario Outline: Validate username length
Given the system requires a username length between 5 and 15 characters inclusive
When the user enters a username with <userName> characters
Then the validation should be <result>
Examples:
| userName | result | comment
| "use4" | rejected | # Just below minimum boundary
| "user5" | successful | # Minimum boundary
| "userN6" | successful | # Just above minimum boundary
| "userNameoooo14" | successful | # Just below maximum boundary
| "userNameooooo15" | successful | # Maximum boundary
| "userNameoooooo16" | rejected | # Just above maximum boundary
| null | rejected | # outside of string lengthSometimes you might need to start a new session.
Sometimes the AI makes a mistake like writing Given, Then, When (wrong order) and sometimes starting a new session makes it go away.
What usages are there? (prompt):
ChatGPT 3.5, 4, and 4o can have issues, when we ask too much of it.
So, sometimes it is better to ask something more specific (in this case we avoid using the BDD examples, because it loses it’s creativity and most of the examples would be similar about values and not topics.
What can be tested with boundary-value-analysis?Boundary Value Analysis (BVA) is a robust testing technique used to identify errors at the boundaries of input ranges, where most errors tend to occur. It is particularly effective in scenarios where inputs fall into defined ranges or thresholds. Here are some areas where BVA can be applied:
1. Input Fields Validation
Numeric Input Fields: Minimum and maximum allowed values, such as age, quantity, price, or score.
Text Input Fields: Minimum and maximum character lengths for fields like passwords, usernames, descriptions, etc.
2. String lengths...
3. Arrays...
4. File sizes...
5. Date and Time Validations...
6. Special characters...
7. Colors...
8. User Interface Elements...
9. Form Validations...
10. Repeats...
11. Multi-threading... As you can see, this answer gave many more different applications, that the one before.
I took the description from the first one and only the titles from the rest, because I wanted to transform some of them into BDD features.
Making a BDD scenario from a specific test case (prompt):
Can you make a BDD feature of a boundary-value-analysis of Disk Space Utilization: Validating file uploads of files? The max size is 4MB.Scenario Outline: Uploading boundary-adjacent file sizes
Given the user tries to upload a file of size <size> MB
When the file is uploaded
Then the upload should be <result>
Examples:
| size | result | comment
| 3.99 | successful | # Just below maximum valid size
| 4.00 | successful | # Maximum valid size
| 4.01 | rejected | # Just above maximum valid sizeIt wrote more, but it is up to us to filter out, what is important and what isn’t.
Common mistakes (prompt):
What are the common mistakes are there with boundary-value-analysis?Ignoring Edge Cases Outside the Boundaries:
Testing only the boundary values themselves but not the values immediately outside the boundaries.
Example Mistake: Testing only the values 1 and 30 but not testing 0 or 31 for an age input range of 1 to 30.
Not Testing Both Sides of a Boundary:
Focusing only on the lower or upper boundary, but not both.
Example Mistake: Testing the maximum value of 30 but forgetting to test the minimum value of 1.
Overlooking Off-by-One Errors:
Missing off-by-one errors which are common in boundary-related bugs.
Example Mistake: If the boundary condition is <= 30, failing to test with the value 31.
Misinterpreting Boundary Conditions:
Misunderstanding whether the boundary is inclusive or exclusive.
Example Mistake: Assuming a boundary condition is inclusive when it is actually exclusive, leading to incorrect test cases.
Assuming Boundaries Without Verification:
Assuming the boundary values without checking the actual requirements or specifications.
Example Mistake: Assuming an age range is 1 to 30 without confirming with the requirements document.
Not Testing the Exact Boundary Value:
Forgetting to test the exact boundary value itself.
Example Mistake: Testing 29 and 31, but forgetting to test the exact boundary value of 30.
Focusing Solely on Numerical Inputs:
Only applying BVA to numerical inputs and ignoring other types like strings or dates.
Example Mistake: Focusing only on numeric ranges and not testing boundaries for string lengths or date ranges.
Overlooking Interaction Between Multiple Boundaries:
Ignoring the interaction between multiple boundaries when they exist.
Example Mistake: If both age and income need to be tested together, only testing them separately and not considering combined boundary scenarios.
Inadequate Test Data Coverage:
Not providing enough test data to cover all the boundary scenarios adequately.
Example Mistake: Only testing the lower boundary and missing the upper boundary entirely.
Ignoring the Context of the Application:
Not considering the specific context or environment where the boundaries are applied.
Example Mistake: Testing file upload sizes without considering different file formats that might affect the boundary limits.Training exercises
Can you give me some training excesses, so I can train boundary-valuse-analysis and avoid these mistakes? (please write the answers at the end only)Exercise 1: Age Verification
Scenario:
An online registration system only accepts users aged between 18 and 60 inclusive.
Tasks:
1. Identify the boundary values for this input range.
2. Write test cases for:
- The exact boundary values.
- Values just below and above the boundaries.
3. Execute the test cases and note the expected outcomes.
Answers:
- Test cases:
- Age 17 (just below minimum boundary) - Expected result: Rejected
- Age 18 (minimum boundary) - Expected result: Accepted
- Age 19 (just above minimum boundary) - Expected result: Accepted
- Age 59 (just below maximum boundary) - Expected result: Accepted
- Age 60 (maximum boundary) - Expected result: Accepted
- Age 61 (just above maximum boundary) - Expected result: RejectedExercise 2: Password Length Validation
Scenario:
A system requires passwords to be between 8 and 20 characters inclusive.
Tasks:
1. Determine the boundary values for password length.
2. Create test cases for:
- The minimum and maximum length.
- Values just below and above the minimum and maximum length.
3. Execute the test cases and note the expected outcomes.
- Test cases:
- Password length 7 (just below minimum) - Expected result: Rejected
- Password length 8 (minimum) - Expected result: Accepted
- Password length 9 (just above minimum) - Expected result: Accepted
- Password length 19 (just below maximum) - Expected result: Accepted
- Password length 20 (maximum) - Expected result: Accepted
- Password length 21 (just above maximum) - Expected result: RejectedExercise 3: File Upload Size
Scenario:
A web application allows file uploads with sizes between 1MB and 4MB inclusive.
Tasks:
1. Identify the boundary values for the file size.
2. Write test cases for:
- The exact boundary values.
- Values just below and above the boundaries.
3. Execute the test cases and note the expected outcomes.
- Test cases:
- File size 0.9MB (just below minimum) - Expected result: Rejected
- File size 1MB (minimum) - Expected result: Accepted
- File size 1.1MB (just above minimum) - Expected result: Accepted
- File size 3.9MB (just below maximum) - Expected result: Accepted
- File size 4MB (maximum) - Expected result: Accepted
- File size 4.1MB (just above maximum) - Expected result: RejectedExercise 4: Loan Approval Based on Credit Score
Scenario:
A bank approves loans based on credit scores:
- Scores below 600 are rejected.
- Scores between 600 and 749 are conditionally approved.
- Scores 750 and above are fully approved.
Tasks:
1. Determine the boundary values for the credit scores.
2. Create test cases for:
- Values just below and above each boundary.
- The exact boundary values.
3. Execute the test cases and note the expected outcomes.
- Test cases:
- Credit score 599 (just below minimum for conditional approval) - Expected result: Rejected
- Credit score 600 (minimum for conditional approval) - Expected result: Conditionally Approved
- Credit score 601 (just above minimum for conditional approval) - Expected result: Conditionally Approved
- Credit score 749 (just below maximum for conditional approval) - Expected result: Conditionally Approved
- Credit score 750 (minimum for full approval) - Expected result: Fully Approved
- Credit score 751 (just above minimum for full approval) - Expected result: Fully ApprovedExercise 5: Temperature Control
Scenario:
A thermostat allows setting temperatures between 60°F and 90°F inclusive.
Tasks:
1. Identify the boundary values for the temperature setting.
2. Write test cases for:
- The exact boundary values.
- Values just below and above the boundaries.
3. Execute the test cases and note the expected outcomes.
- Test cases:
- Temperature 59°F (just below minimum) - Expected result: Rejected
- Temperature 60°F (minimum) - Expected result: Accepted
- Temperature 61°F (just above minimum) - Expected result: Accepted
- Temperature 89°F (just below maximum) - Expected result: Accepted
- Temperature 90°F (maximum) - Expected result: Accepted
- Temperature 91°F (just above maximum) - Expected result: RejectedExercise 6: Discount Application Based on Purchase Amount
Scenario:
A store offers discounts based on purchase amount:
- Purchases below $50 get no discount.
- Purchases between $50 and $100 get a 10% discount.
- Purchases above $100 get a 20% discount.
Tasks:
1. Determine the boundary values for the purchase amounts.
2. Create test cases for:
- Values just below and above each boundary.
- The exact boundary values.
3. Execute the test cases and note the expected outcomes.
- Test cases:
- Purchase amount $49 (just below minimum for discount) - Expected result: No discount
- Purchase amount $50 (minimum for 10% discount) - Expected result: 10% discount
- Purchase amount $51 (just above minimum for 10% discount) - Expected result: 10% discount
- Purchase amount $99 (just below maximum for 10% discount) - Expected result: 10% discount
- Purchase amount $100 (maximum for 10% discount) - Expected result: 10% discount
- Purchase amount $101 (just above maximum for 10% discount) - Expected result: 20% discountReflection Questions:
1. Did you miss any edge cases outside the boundaries? How can you ensure they are always included in your tests?
2. Were both sides of each boundary adequately tested? Why is it important to test both?
3. Did you encounter any off-by-one errors? How can they be prevented in future tests?
4. Were the boundary conditions interpreted correctly (inclusive vs. exclusive)? How can misinterpretations be avoided?
5. Did you verify the boundary values with the requirements or specifications before testing?Advances analysis (prompt):
Can you make an boundary-value-analysis, for:
We have an app that must show products that have date-ranges.
We may only see products, which date-range include today.
How would we test this? Boundary-Value Test Cases:
1. Exact Match with Today:
Start Date: 2024-05-17, End Date: 2024-05-17
Expected result: Product is displayed.
2. Starts Before, Ends After:
Start Date: 2024-05-16, End Date: 2024-05-18
Expected result: Product is displayed.
3. Starts Today, Ends After:
Start Date: 2024-05-17, End Date: 2024-05-18
Expected result: Product is displayed.
4. Starts Before, Ends Today:
Start Date: 2024-05-16, End Date: 2024-05-17
Expected result: Product is displayed.
5. Starts Before, Ends Before:
Start Date: 2024-05-15, End Date: 2024-05-16
Expected result: Product is not displayed.
6. Starts After, Ends After:
Start Date: 2024-05-18, End Date: 2024-05-19
Expected result: Product is not displayed.
7. Starts Before, Ends Before:
Start Date: 2024-05-15, End Date: 2024-05-16
Expected result: Product is not displayed.arent 5. and 6. the same?You are right, cases 5 and 7 are the same in the initial list. Let's refine the scenarios to ensure all boundary cases are covered distinctly and effectively.
...
7. Product Date Range Completely Before Today:
Start Date: 2024-05-14
End Date: 2024-05-15
Expected result: Product is not displayed.
8. Product Date Range Completely After Today:
Start Date: 2024-05-19
End Date: 2024-05-20
Expected result: Product is not displayed.But it didn’t get the refinement correct, because instead of removing number 7, it changed it and added number 8 (which are covered by 5 and 6)!
Can you rewrite it as a BDD example?
And instead of dates use, today, today +1 day, today -1 day, etc.?Feature: Display Products Based on Date Ranges
As a user of the app
I want to see only the products that are available today
So that I can make purchasing decisions based on current availability
Background:
Given today's date is "2024-05-17"
Scenario Outline: Product date range includes today
Given a product with a start date of "<start_date>" and an end date of "<end_date>"
When I view the list of available products
Then the product should be "<result>"
Examples:
start_date | end_date | result | Comment
today | today | displayed | # Exact match with today
today – 1 day | today + 1 day | displayed | # Starts before, ends after
today | today + 1 day | displayed | # Starts today, ends after
today – 1 day | today | displayed | # Starts before, ends today
today – 2 days | today – 1 day | not displayed | # Ends just before today
today + 1 day | today + 2 days | not displayed | # Starts just after today
today – 3 days | today – 2 days | not displayed | # Completely before today
today + 2 days | today + 3 days | not displayed | # Completely after todayAnd here we can see easier the problem with 7. and 8., since it just added “Completely”, even though it gives no extra data.
Building examples with the AI
AI can do good things, but it is better to build them together, so I did do some examples:
Example 1: Date ranges
Date ranges are even more fun.
Scenario Outline: Product date range includes today
Given a product with a start date of "<start_date>" and an end date of "<end_date>"
When I view the list of available products
Then the product should be "<result>"
Examples:
start_date | end_date | result | Comment
today | today | displayed | # Exact
today –1 day | today + 1 day | displayed | # Before & After
today | today + 1 day | displayed | # Exact & After
today –1 day | today | displayed | # Before & Exact
today –2 days | today – 1 day | not displayed | # Only after
today +1 day | today + 2 days | not displayed | # Only beforeIt is much easier to understand it as a graph:
Example 2: Time (regular, sommer time)
Time ranges are crucial in many applications, such as booking systems. Office meeting rooms, for instance, have specific hours during which they can be booked.
Scenario Outline: An IT-system can only process one event per hour.
Given events are sat in CET/CEST time zone
When an event is tried to be assigned at <event_datetime>
Then it must be <result>
Examples:
event_datetime | result | comment
2024-03-31 01:59:59 | succesful | # Before DST spring forward
2024-03-31 02:00:00 | rejected | # Just during the missing hour
2024-03-31 02:00:01 | rejected | # During the missing hour
2024-03-31 02:59:59 | rejected | # During the missing hour
2024-03-31 03:00:00 | succesful | # Just after the DST change
2024-03-31 03:00:01 | succesful | # After the DST changeExample 3: Special characters (formatting)
Character encoding can affect boundaries.
UTF-8 encoding is a variable-length character encoding for Unicode that uses one to four bytes to encode characters.
Symbols such as A-Z, a-z, 0-9 takes 1 byte per character.
Including a single special character such as ñ, will upgrade the text to 2 bytes per character.
Including a single special character such as 汉, will upgrade the text to 3 bytes per character.
Including a single special character such as 😀, will upgrade the text to 4 bytes per character.
Scenario Outline: Validate input length and encoding boundary-adjacent values
Given the system allows a maximum text size of 12 bytes
When I enter a text <input_text> characters including <character_type>
Then the input should be <result>
Examples:
input_text | result | comment
"a........11" | successful | # Just below maximum byte size
"1.........12" | successful | # maximum byte size
"1..........13" | rejected | # Just above maximum byte size
"ñ...5" | successful | # Just below maximum byte size
"ñ....6" | successful | # maximum byte size
"ñ.....7" | rejected | # Just above maximum byte size
"汉.3" | successful | # Just below maximum byte size
"汉..4" | successful | # maximum byte size
"汉...5" | rejected | # Just above maximum byte size
"😀2" | successful | # Just below maximum byte size
"😀.3" | successful | # maximum byte size
"😀..4" | rejected | # Just above maximum byte size
So, remember that when ever you write a :-) instead of 😀, you actually save 1 byte of data per character in the text you send! Think about the amount of CO2 you might save, by being old school ;-)
Example 4: Color profiles
In 2020, there was a specific wallpaper image that, when set as the background on certain Android phones (especially Samsung devices), caused the phone to crash and get stuck in a boot loop. This issue was related to how the Android system processed the color profile of the image.
The problematic image had a color profile that included colors outside the standard sRGB gamut. When the Android system attempted to convert this color profile to its default display profile (usually sRGB), it encountered colors that were out of range.
Read more
The system should handle out-of-range values by clipping them to the nearest valid value (e.g., setting any value above 255 to 255)
It might be a good idea to test converting images from one profile to another, while giving colors extreme values.
Scenario Outline: Validate handling of out-of-range color values during color profile conversion
Given an image with a <color_profile> color profile
When the image is converted to sRGB
Then any color value above 255 should be rounded down to 255
And the image processing should be <result>
Examples:
color_profile | initial_value | converted_value | comment
Adobe RGB | 259 | 254 | # not rounded
Adobe RGB | 260 | 255 | # not rounded
Adobe RGB | 261 | 255 | # rounded down
ProPhoto RGB | 299 | 254 | # not rounded
ProPhoto RGB | 300 | 255 | # not rounded
ProPhoto RGB | 301 | 255 | # rounded down Example 5: Repeats (maximum)
Login attempts can also be as a boundary-value-analysis, since there is a boundary between “allowed attempt” and “too many attempts”
Scenario Outline: Validate maximum password retry attempts
Given a user is attempting to log in
When the user enters an incorrect password <retry_count> times
Then the system should <result>
Examples:
retry_count | result | comment
2 | retry_allowed | just below max.
3 | retry_allowed | at max.
4 | retry_denied | just above max.Example 6: Multi-threading (minimum) for performance testing
Boundary-Value-Analysis can also be applied to a minimum simultaneous threads.
Scenario Outline: Validate minimum thread handling for concurrent requests
Given the server can handle a minimum of 10 threads
When <thread_count> threads are used to handle concurrent requests
Then the server performance should be <result>
Examples:
threads | result | #comment
9 | good | just below max.
10 | good | at max.
11 | slow but acceptable | just above max.So, BVA is even applied to with Performance testing!
Conclusion
From this lesson, we’ve learned several key aspects of using the Boundary-Value-Analysis (BVA) effectively through structured lessons and examples:
- What BVA is: Like playing Minecraft and move to the edge as close as possible without falling down
- Learn from examples: That we can learn from actual examples.
- Get more examples: That multiple examples can show us more usages.
- Sometimes you might need to start a new session: This point underlines the iterative nature of working with AI.
- What usages are there: That often it is better to ask AI about usages, and not form it into a specific format like BDD scenarios. Then we can get a broader scope like: Input Fields Validation, Range-Based Functionality, Resource Allocation, Financial Transactions, Date and Time Validations, Threshold-Based Alerts, Performance Testing, User Interface Elements, Form Validations, and Configurations and Settings.
- Making a BDD scenario from a specific test case: When we have the broad input, then it is much easier to transform it into a specific format. And that we might have to manually filter the answer, because it can present too much data.
- Common mistakes: Negative scenarios can also give a broader scope, so we also learn of not to do. With BVA it was: Ignoring Edge Cases Outside the Boundaries, Not Testing Both Sides of a Boundary, Overlooking Off-by-One Errors, Misinterpreting Boundary Conditions, Assuming Boundaries Without Verification, Not Testing the Exact Boundary Value, Focusing Solely on Numerical Inputs, Overlooking Interaction Between Multiple Boundaries, Inadequate Test Data Coverage, and Ignoring the Context of the Application.
- Training exercises: To actually create scenarios that have to be solved for Age Verification, Password Length Validation, File Upload Size, Loan Approval Based on Credit Score, Temperature Control, Discount Application Based on Purchase Amount, and Reflection Questions!
- Advances analysis: To let the AI create a boundary-value-analysis for an app, where we only may see products, that are applicable only for today. The AI created one too many scenarios, and even though it admitted it’s fault, it added another one instead of removing the wrong one. The AI could transform it into a BDD scenario, to make it easier for us to filter it ourselves.
AI is indeed a great tool to learn about Boundary-Value-Analysis.
It’s not perfect, but better than some of the free tutorials on the net.
Congratulations – Lesson complete!
Want more?
Check out the comments on Linked.