Requires Free Membership to View
When you register, you'll receive targeted emails designed to keep you informed of the most relevant information on Agile development, application security, testing & QA, software requirements, and more.
Hannah Smalltree, Editorial Director
Determining test coverage is best served with three measurable categories, plus a fourth subjective category. First, the easy stuff ... with the right tools, it's quite easy to map requirements to test cases. By so doing, it's easy to report to the customer or business partner what the project status is. If 50% of the test cases are complete, it's reasonable to report that test coverage is at 50%.
Next, test case status -- this is closely related to the above. The entire body of test cases represents near-total (see caveat below) coverage, and test case status is a report on the total case count, count of cases passing, percentage of cases passing, count of cases failing, percentage of cases failing, count of cases blocked, and percentage of cases blocked.
Another way to determine test case coverage is to analyze test coverage using code coverage analysis tools. These are generally used in conjunction with automated tests, eliminating human error as well as getting to the code coverage metric faster. Note that code coverage, while an important metric, is not a panacea to all things testing. Code coverage simply gives a general idea for what areas of the product are missing test coverage. Even if you achieve 100% code coverage, it's possible there are still defects in the code. For instance, you might enter a function with a number outside the expected range -- you could still successfully exit the function, but you may exit with false positives. Code coverage would never catch that.
The final component to test coverage is what I call "gut feel" or "tester's instinct." A seasoned tester simply comes to feel what the project is like -- it's very similar to a golfer reading a green, a cyclist feeling the crowd, or a climber sensing the rock. There are times when all the metrics say a product is OK, but a test manager might still be inclined to call for further testing before signing off. This comes from experience and is something that is nearly impossible to teach another person. I generally look at all the information: test case status, requirements coverage, defect analysis. But in the end, before I sign off on a project I will spend some time letting the project "bake."
Caveat: Testing is a constantly growing process, which means you can never take a snapshot of your test case repository and say you have complete coverage.
This was first published in January 2009