How to determine test coverage in a software project

Digg This!     StumbleUpon     Del.icio.us

RELATED CONTENT Software Testing and Quality Assurance Why do performance testers write new scripts so often? How to create performance testing workload models Fixing Web application performance troubleshooting problems Expert advises on implementation of Selenium IDE for effective software testing When should regression testing occur in an automated test plan? Achieving peak performance in integration testing Getting answers about OpenSTA script problems Defining core software regression tests Breaking in functionality on UI application pages Where to find good methodology guides for software testing Software testing and quality assurance (QA) fundamentals How to deal with iteration issues in Agile Five steps to fostering better software tester and QA results Software Testing: New software testing technologies bring new challenges Testing strategies for complex environments Astronaut's STPCon advice: Teamwork delivers "The Right Stuff" How to make your software tamperproof Software consortium seeks standard quality metrics Demo: Using WebGoat, a free software testing tool Seven steps for a quality change and configuration management program Winning responses to "Why is QA always the bottleneck?"

RELATED GLOSSARY TERMS Terms from Whatis.com − the technology online dictionary build  (SearchSoftwareQuality.com) code review  (SearchSoftwareQuality.com) conformance testing  (SearchSoftwareQuality.com) error handling  (SearchSoftwareQuality.com) garbage in, garbage out  (SearchSoftwareQuality.com) load testing  (SearchSoftwareQuality.com) NUnit  (SearchSoftwareQuality.com) quality assurance  (SearchSoftwareQuality.com) stress testing  (SearchSoftwareQuality.com) white box  (SearchSoftwareQuality.com)

RELATED RESOURCES 2020software.com, trial software downloads for accounting software, ERP software, CRM software and business software systems Search Bitpipe.com for the latest white papers and business webcasts Whatis.com, the online computer dictionary

I love these precise, focused questions! <grin>

Determining test coverage is best served with three measurable categories, plus a fourth subjective category. First, the easy stuff ... with the right tools, it's quite easy to map requirements to test cases. By so doing, it's easy to report to the customer or business partner what the project status is. If 50% of the test cases are complete, it's reasonable to report that test coverage is at 50%.

Next, test case status -- this is closely related to the above. The entire body of test cases represents near-total (see caveat below) coverage, and test case status is a report on the total case count, count of cases passing, percentage of cases passing, count of cases failing, percentage of cases failing, count of cases blocked, and percentage of cases blocked.

Another way to determine test case coverage is to analyze test coverage using code coverage analysis tools. These are generally used in conjunction with automated tests, eliminating human error as well as getting to the code coverage metric faster. Note that code coverage, while an important metric, is not a panacea to all things testing. Code coverage simply gives a general idea for what areas of the product are missing test coverage. Even if you achieve 100% code coverage, it's possible there are still defects in the code. For instance, you might enter a function with a number outside the expected range -- you could still successfully exit the function, but you may exit with false positives. Code coverage would never catch that.

The final component to test coverage is what I call "gut feel" or "tester's instinct." A seasoned tester simply comes to feel what the project is like -- it's very similar to a golfer reading a green, a cyclist feeling the crowd, or a climber sensing the rock. There are times when all the metrics say a product is OK, but a test manager might still be inclined to call for further testing before signing off. This comes from experience and is something that is nearly impossible to teach another person. I generally look at all the information: test case status, requirements coverage, defect analysis. But in the end, before I sign off on a project I will spend some time letting the project "bake."

Caveat: Testing is a constantly growing process, which means you can never take a snapshot of your test case repository and say you have complete coverage.