Testing : Boundary Value Analysis

Gieno    Testing : Boundary Value Analysis

Boundary value analysis (BVA) is perhaps the best-known functional test technique. Historical evidence demonstrates errors tend to occur near the extreme boundaries of input and/or output values of physical variables. The “single fault” assumption in reliability theory states failures are rarely result from the simultaneous occurrence of two or more faults. Traditional boundary value analysis focuses on input or output values at the extreme boundary conditions of independent physical variables. Boundary value testing does not apply to Boolean or logical variables, and is generally not effective in complex combinations.

The minimum set of BVA tests exercises one independent physical variable at the extreme ranges while other independent physical variables are set at a nominal condition. The traditional approach to boundary value analysis discusses the valid input and/or output values for validation test cases. However, in actual practice boundary value testing also includes the invalid values +/- 1 beyond the valid range of values to examine both validation and falsification conditions (n_max, n_max+1, n_max-1, n_min, n_min-1, n_min +1) as well as one test case for nominal values. Including invalid values in the BVA test case suite tests run-time errors that abort normal execution and ensures proper error handling.

For example, X and Y are independent input variables. The minimum physical value of Y is a, and b is the maximum physical value of Y.  The minimum physical value of X is c, and d is the maximum physical value of X. The formula to determine the traditional minimum set of tests is 6n +1, so this example would require 6n +1 = 6(2) + 1 = 13 tests. The set of tests for this example is illustrated in Figure 1 and defined in the table below.

Test case	Values	Test case	Values	Test case	Values	Test case	Values
1	Xnom, Ya - 1	5	Xnom, Yb	9	Ynom, Xc + 1	13	Ynom, Xnom
2	Xnom, Ya	6	Xnom, Yb + 1	10	Ynom, Xd – 1
3	Xnom, Ya + 1	7	Ynom, Xc – 1	11	Ynom, Xd
4	Xnom, Yb - 1	8	Ynom, Xc	12	Ynom, Xd + 1

 

 

 

 

 

Boundary value analysis can also be used to derive a set of “robust” test cases which exercises all possible combinations of the boundary values (minimum – 1, minimum, minimum +1, nominal, maximum -1, maximum, and maximum +1) of two or more independent variables. Jorgenson [1] recommends using “robust” BVA when physical variables interact and function failures could be extremely costly. “Robust” boundary value testing yields a total of 7ⁿ test cases, or as in the previous example 7² = 49 test cases .

In a feature which only includes 2 independent physical variables “robust” boundary value testing is reasonably achievable. However, applying “robust” testing to the Triangle problem in which 3 independent physical variables are

However, numerous case studies at Microsoft and within the industry clearly demonstrate testing all possible combinations is unlikely to reveal any errors or changes in functionality beyond a carefully constructed array of combinations using orthogonal arrays or pair-wise testing analysis and expert knowledge.

Bj’s BVA Reduction Theory

Binder [2] states “Some kinds of externally applied tests may be omitted if equivalent checking is done by assertion.” He also suggests in many cases we can reduce the number of boundary tests “on method arguments that are explicitly checked by pre or post conditions,” and “conditions on looping and recursion with loop invariants and loop variants.”

Bj’s BVA reduction theorem states any values within the minimum and maximum boundary values are inclusive of valid equivalence class of all physical values within those boundary values. This theory asserts in most cases a test exercising the minimum +1, maximum -1, or any nominal value would not produce a different result then a test which exercises the minimum and maximum values of that same physical variable. If the probability of subsequent tests is not likely to produce different results then those subsequent tests are probably redundant.

In general, omitting the nominal and the maximum -1 value tests does not expose any significant risk. However, there are some conditions in which the minimum +1 value should not be omitted from the BVA test suite. Whittaker [3] points out testers should apply inputs that force default values. For example: if a developer specifies a default size of an array, then a size of 0 (not null) is a legal value. Also, a common mistake occurs when developers do not assign variables an initial value. For example:

newBuffer()
newBuffer(0)
newBuffer(1)

where the parameter is the initial buffer size testers should not assume that newBuffer() and newBuffer(0) perform the same, even if the documentation claims they are identical.

REFERENCES

[1] Paul C. Jorgensen, Software Testing: A Craftsman’s Approach
[2] Robert V. Binder, Testing Object-Oriented Systems: Models, Patterns, and Tools
[3] James Whittaker, How to Break Software: A Practical Guide to Testing