Do you think tracking pre-production defects is important and if so, why?
The question I have when looking at this idea is, what will we learn from this? What is the reason we have for tracking these defects, or anything else for that matter?
The only reason to consider this, or much else in the realm of software project information gathering, is to help us do our jobs better. If we want to look at the trends found in testing around problem areas, then compare these trends against what occurs in production, we may be able to build better models for testing.
Tracking defects found in testing as a performance metric tends to be demotivating, placing software testers and developers in an adversarial position when they should be in a cooperative position. Instead, if the purpose of tracking defects is to look for patterns in software development, including requirements definition, design and coding, where problems are found on a regular basis, this may provide information around aspects that need improvement in the organization.
Likewise, comparing defects found in testing, fixed or not, with those found in production can show how our models for testing can be improved. As we learn about the product, its intended use and how it actually is used, we can build broader models for testing beyond simple “validate requirements.”
If we limit the view we have in testing, we will limit the types and nature of the defects we detect. These limitations may restrict the information we provide to project stakeholders. By using these detected defects to map what is understood about the software, we can then compare these understandings with the calls to the support center or help desk, whether these result in a defect reported by users or not.
This will help us to do a better job of testing on the next project.
This was first published in July 2012