Making Change for a $5 Bill: Test Coverage

How many ways are there to “break” a five dollar bill?  Well, you can give the customer five $1 bills.  Or you can give her twenty quarters, or fifty dimes, or a hundred nickels, or five hundred pennies.  Or if you have them, two $2 bills and a dollar coin.  Or various combinations.  Probably lots of combinations; I’ll leave enumerating the various permutations to the reader, as an exercise.  So, if you are testing software that will have to break a five, do you have to test for every possible outcome, meaning every possible combination of bills and coins?  Or do you just have to test the process by which the outcome is produced?

Back in the old days, when we used third generation compiled procedural programming languages, like COBOL and FORTRAN, we tried to structure programs, so the paths through the code were obvious (or at least visible).  In those days, when we talked about testing coverage, we wanted to exercise every likely path through the code.  The theory was that the logic was being tested, so as long as we used valid test data, we should be able to find any problems in the logic.  So we created meticulous scripts, calling for testers to take specific actions in a specific sequence, looking for specific positive results; any deviation was reported as a possible defect.  This focus on positive testing gave rise to the phrase, “The happy path.”  And we’d incorporate bad data to ensure errors would be noted and properly handled; negative testing was more about crap data, or doing things out of sequence.  But things have changed.

Nowadays, most software applications are interpreted, meaning they are “formed” at run time, using the appropriate components and run time libraries.  Differences in versions of the underlying components, such as DLL’s and Java virtual machine and related class libraries, make it nearly impossible to definitively test all possible versions of the runtime, let alone all paths through the software.  In addition, the use of object oriented designs and technologies makes it extremely difficult to be sure you will know what paths will be taken at run time, even with a stable configuration, since an object inherits all behaviors of its parents.  So we have to take a different approach to planning our tests.  We use scenarios, which are general examples of business use cases.

A scenario for testing our example might be, “Make change so that the customer receives the fewest possible coins.”  Or possibly, “Make change so the customer receives eight quarters.”  Or, “Make change using only the bills and coins present in the system, where there are fewer than five $1 bills.”  Note that each of these scenarios describes the outcome, not the process.  Our testing is focused on the business result, not the logic used to arrive at it.  The software is simply a black box, which can be replaced at any time.  Consequently, determining adequate coverage is less about identifying the number of ways to exercise the logic than it is identifying a proper range of outcomes.  Testing is done to ensure that the business decisions made by the application are correct, and use all business rules correctly.  Thus, scenarios should be selected based on the business rules and how they combine to determine a correct result.  Both positive testing and negative testing scenarios should be developed, to ensure that, where two business rules apply, they are applied in priority order.  And since so much of our application software security is role-based, scenarios should be created for all roles.

In the end, software application testing is a component of the total cost of development.  It isn’t cost-effective to “test to perfection” for most business applications, unless they involve safety or other high-risk applications.  The project manager must work with the stakeholders, development team, and management to determine a testing strategy that maximizes quality, within cost and schedule constraints.  And while that means tradeoffs, a clear understanding of the software architecture and the application use cases will allow determination of proper test coverage, and a way to achieve it.