How to determine test coverage?
I love these precise, focused questions! <grin>
Determining test coverage is best served with three measurable categories, plus a fourth subjective category. First, the easy stuff ... with the right tools, it's quite easy to map requirements to test cases. By so doing, it's easy to report to the customer or business partner what the project status is. If 50% of the test cases are complete, it's reasonable to report that test coverage is at 50%.
Next, test case status -- this is closely related to the above. The entire body of test cases represents near-total (see caveat below) coverage, and test case status is a report on the total case count, count of cases passing, percentage of cases passing, count of cases failing, percentage of cases failing, count of cases blocked, and percentage of cases blocked.
Another way to determine test case coverage is to analyze test coverage using code coverage analysis tools. These are generally used in conjunction with automated tests, eliminating human error as well as getting to the code coverage metric faster. Note that code coverage, while an important metric, is not a panacea to all things testing. Code coverage simply gives a general idea for what areas of the product are missing test coverage. Even if you achieve 100% code coverage, it's possible there are still defects in the code. For instance, you might enter a function with a number outside the expected range -- you could still successfully exit the function, but you may exit with false positives. Code coverage would never catch that.
The final component to test coverage is what I call "gut feel" or "tester's instinct." A seasoned tester simply comes to feel what the project is like -- it's very similar to a golfer reading a green, a cyclist feeling the crowd, or a climber sensing the rock. There are times when all the metrics say a product is OK, but a test manager might still be inclined to call for further testing before signing off. This comes from experience and is something that is nearly impossible to teach another person. I generally look at all the information: test case status, requirements coverage, defect analysis. But in the end, before I sign off on a project I will spend some time letting the project "bake."
Caveat: Testing is a constantly growing process, which means you can never take a snapshot of your test case repository and say you have complete coverage.
Dig Deeper on Topics Archive
Related Q&A from John Overbaugh
Learn what's behind AWS outages and how to fix failures before they happen. Continue Reading
Learn strategies for best security test strategies for SaaS cloud. Continue Reading
Expert John Overbaugh identifies the three top concerns of the test manager and offers advice on how to stay ahead of the curve when it comes to ... Continue Reading