Software testing offers big ROI |
 |
By Jennette Mullaney, Associate Editor
05 Mar 2008 | SearchSoftwareQuality.com |
|
|
NEW YORK -- Software testing is an expense that ultimately saves an organization a great deal of money. It has a quantifiable return on investment (ROI) and testers can follow a simple series of steps to calculate it, explained Rex Black during a session at the recent FutureTest 2008 conference.
Black, president and principal consultant of RBCS, emphasized the cost-effectiveness of creating quality software. "The money you spend to build it right the first time is always less than the money it costs to fix it," he said. The amount of money saved can be two times, four times or even 32 times the cost of investment, he continued.
Testers concerned about their budgets can cite a wealth of well-established references to justify upfront costs to management, Black said. He listed four ways testing saves money: Finding bugs that get fixed, finding bugs that don't get fixed, running tests that mitigate risks and guiding the project to success through timely, accurate, credible information for project tracking.
The ROI for finding and fixing bugs
When figuring the ROI for finding bugs that are fixed, Black instructed his audience to measure the cost of quality. The cost of quality combines the costs of conformance and nonconformance, he said. Conformance costs are those accrued when an organization creates and tests quality software and can be broken down into prevention and protection costs.
The cost of nonconformance consists of the cost of internal failure before release and external failure after, including the expense of finding and correcting bugs in released software, handling phone calls from angry customers, and losing business due to a damaged reputation.
Black set up equations for a series of hypothetical case studies. He estimated 1,000 "must-fix" bugs occur in a quarterly release. Bugs cost $1 when found during review, $10 when found through developer testing, $100 when found through tester testing, and $1,000 when found by customers.
If an organization conducted no formal testing, developers might find 250 bugs and customers 750, Black hypothesized. Using his figures, the software vendor spends $2,500 for the bugs found by developers and $750,000 for those found by customers for a total of $752,500 for the cost of quality.
If the vendor hires an inferior testing group for $70,000 and that group finds 350 bugs through manual testing, the total cost of quality is greatly reduced, Black told the audience. After paying for the testers, the vendor spends $2,500 for the bugs found by developers, $35,000 for the bugs found by testers and $400,000 for the bugs found by customers. The cost of quality in this case is $507,500, Black found.
In another hypothetical case, Black armed the testing team with automated tools costing $150,000 ($12,500 a quarter for 12 quarters). The testers find 500 bugs and the cost of quality is reduced to $385,000, he calculated. Costs can be cut further if bugs are found earlier, added Black.
"Suppose the test manager realizes it's cheaper to find bugs in requirement and design reviews," he said. If the testers manage to find 150 bugs this way at a price tag of $1, the cost of quality drops to $235,150.
For these hypothetic cases, Black deliberately picked an unexceptional testing team.
"A good team should find 85% or more of defects," he said. "A mediocre job still saves the company a lot of money."
The value of finding bugs that aren't fixed
"The rule of testing is 'find the scary stuff first,'" Black said. Once the "scary stuff" has been fixed, there may be many bugs remaining. However, knowing about these bugs and preparing for them can generate value, he explained. An organization can decide to turn off the feature that contains the bug, warn users of the bug in release notes, or provide technical support staff with a workaround. The workaround provides value by shortening calls to tech support.
Black set up another equation to demonstrate his point. In this case, each software bug initiates five calls, a call for a known bug is 15 minutes shorter than the call for an unknown bug, and each member of technical support is paid $40 an hour. There are 60 minutes in an hour, so put 15 over 60, multiply the fraction by five and by 40. The result is $50 saved.
Convincing management
"The value of information is found in enlightening management and executives," Black said. "We don't do a good job of this."
He encouraged the audience to perform their own calculations.
"What gets measured gets done, what gets valued gets funded," said Black. "What we do as testers has value and we can quantify that value."
|
|
|
|
