Test coverage: Finding all the defects in your application

Digg This!     StumbleUpon     Del.icio.us

RELATED CONTENT Software Testing and Quality Assurance Why do performance testers write new scripts so often? How to create performance testing workload models Fixing Web application performance troubleshooting problems Expert advises on implementation of Selenium IDE for effective software testing When should regression testing occur in an automated test plan? Achieving peak performance in integration testing Getting answers about OpenSTA script problems Defining core software regression tests Breaking in functionality on UI application pages Where to find good methodology guides for software testing Software testing and quality assurance (QA) fundamentals How to deal with iteration issues in Agile Five steps to fostering better software tester and QA results Software Testing: New software testing technologies bring new challenges Testing strategies for complex environments Astronaut's STPCon advice: Teamwork delivers "The Right Stuff" How to make your software tamperproof Software consortium seeks standard quality metrics Demo: Using WebGoat, a free software testing tool Seven steps for a quality change and configuration management program Winning responses to "Why is QA always the bottleneck?" Software testing tools and frameworks Performance testing tools - Commercial, less expensive and free Software Testing Ezines New IBM Rational, Tivoli integrated tools pair development with IT STPCon: Do reality checks on performance test products, panelists advise Demo: Using WebGoat, a free software testing tool Getting answers about OpenSTA script problems Defining core software regression tests Selecting the best tool for stress and load testing Required prerequisites for performance testing Surgient 7's self-provisioning promises software testers quick IT resource access

RELATED GLOSSARY TERMS Terms from Whatis.com − the technology online dictionary build  (SearchSoftwareQuality.com) code review  (SearchSoftwareQuality.com) conformance testing  (SearchSoftwareQuality.com) error handling  (SearchSoftwareQuality.com) garbage in, garbage out  (SearchSoftwareQuality.com) load testing  (SearchSoftwareQuality.com) NUnit  (SearchSoftwareQuality.com) quality assurance  (SearchSoftwareQuality.com) stress testing  (SearchSoftwareQuality.com) white box  (SearchSoftwareQuality.com)

RELATED RESOURCES 2020software.com, trial software downloads for accounting software, ERP software, CRM software and business software systems Search Bitpipe.com for the latest white papers and business webcasts Whatis.com, the online computer dictionary

The trace matrix is a well-established test coverage tool. Let me offer a quick definition -- the purpose of the trace matrix is to map one or more than one test case to each system requirement, the trace matrix is usually formatted in a table. The fundamental premise is that if one or more than one test case has been mapped to each requirement, then all the requirements of the system must have been tested and therefore the trace matrix proves testing is complete.

I see flaws with this line of reasoning and here are my primary reservations on the over-reliance of the trace matrix:

1. A completed trace matrix is only as valuable as the contents. If the requirements are not complete or clear than the test cases designed and executed might fulfill the requirements but the testing won't have provided what was needed. Conversely if the requirements are clear but the test cases are insufficient then a completed trace matrix still doesn't indicate the testing coverage and confidence that is being sought by a completed table.



2. The trace matrix design relies too stringently on system requirements -- that is the primary design of the trace matrix -- to ensure all system requirements have been tested. But all sorts of defects can be found outside of the system requirements that are still relevant to the application providing a solution for the customer. By looking only at the system requirements and potentially not considering the customers' needs and real life product usage, essential testing could be overlooked. Testing only according to specified requirements may be too narrowly focused to be effective in real life usage -- unless the requirements are exceptionally robust.

So how do you call the end of testing? And how can you assure test coverage?

1. To be able to assure coverage at the end, I'd start with reviewing the beginning -- look at the test planning. Did your test planning include a risk analysis? A risk analysis at the start of a project can provide solid information for your test plan. Host a risk analysis either formally or informally, gather ideas by talking with multiple people. Get different points of view -- from your project stakeholders, talk to your DBAs, your developers, your network staff, and your business analysts. Plan testing based on your risk analysis.



2. As a project continues, shift testing based on the defects found and the product and project as it evolves. Focus on high risk areas. Adapt testing based on you and your testing team's experience with the product. Be willing to adjust your test plan throughout the project.



3. Throughout testing, watch the defects reported. Keep having conversations and debriefs with hands-on testers to understand not just what they've tested but how they feel about the application. Do they have defects they've seen but haven't been able to reproduce? What is their perception of the current state of the application?

Software testing resources: Trace matrix as an indicator of test coverage Web application testing techniques Managing the Testing Process -- Chapter 2, The Test Plan