All too often, return on investment measures for test automation software purchases are too simplistic, according to consultant Robert Galen. That is dangerous because it "commoditizes the role of the tester" in management's eyes and undermines the value of software testing to the business, he said. "The business case for test automation is always the same: We'll run more tests faster, with fewer people. And the more tests we run, the higher the business value. It's insane."
Test managers who do ROI for test automation wear rose colored glasses.
founder, RGalen Consulting Group
Running more tests, faster, does not produce better software. Better software is the result of running the right tests and continually re-evaluating which tests are the right ones, said Galen, founder of RGalen Consulting Group in NC. To do that you need creative, top-notch testers "who can attack the customer's real problems and design tests accordingly."
In this article, Galen explains why simplistic return on investment (ROI) cases for test automation harm testing organizations, and focuses on what test managers should invest in instead: creating the right tests, and hiring and developing the right testers.
The cost justification for test automation is that manual testing is a slow, labor-intensive, error-prone process, Galen said. ROI projects typically make their case by capturing basic metrics such as: the size of the team, cost per tester, number of tests run, time it takes to run them, and the number of test runs per release, then contrasting those numbers with what would it cost to replace the manual approach with automation. Usually that's done using measures such as the time and cost to establish the automation infrastructure and the automated tests; time costs for ongoing maintenance; costs of the tooling and other factors. "Haven't we matured beyond this simplistic view of ROI?" he wondered.
In Galen's view, this kind of simplistic ROI is a bad idea, regardless of the outcome. Even if management decides to make the purchase, a lot of damage has been done. "You have devalued the test organization, the role of test professionals in management's eyes, and you have committed to cost savings that are unlikely to be realized."
Test automation ROI devalues testers' role
"The business sees testing as a low-skilled activity, and the testers themselves are seen as a cost," Galen said. And that sends the wrong message to management. "You don't want commodity testers. You want the bright folks," noting that you cannot conduct effective automation testing without them. Test automation cost justifications that equate more tests, conducted faster, with the business value of test automation, devalue testers and test organizations, he said.
Simplified ROI measures: Seeing test automation through rose-colored glasses
Simplified ROI measures also ignore the intangible costs associated with moving to test automation, Galen said. "Test managers who do ROI for test automation wear rose-colored glasses." They underestimate the time and effort required to set up new software and customize it to run the right tests, and don't address things like software updates, [and] ongoing support and maintenance of the automation frameworks. "The case for test automation is like the case for offshoring. The numbers look attractive --offshore testers get a dollar an hour and onshore testers make 80 bucks-- but the devil is in the details."
Those details include setting up a test automation framework while simultaneously running your test projects. "You are essentially dealing with parallel projects," Galen said. Unanticipated challenges, such as changes in the technology stack, mean you have to change the automated tests. Even if test automation projects have a lot of funding upfront, the money may run out when it's time to actually test something with the automated framework, Galen said.
Invest time in morphing the test set
Instead of digging themselves into a test automation ROI hole, test managers should invest in tests and in testers, Galen said. A common mistake test organizations make is continually subjecting software to existing tests instead of developing new ones that might uncover new defects. Galen estimated that 95% of tests run are existing tests, and only 5% are new. "This doesn't make sense because applications are evolving all the time, and tests must evolve, too." Testers need to continually ask: "Which tests should I retire? Which tests should I redesign so they have more value?"
Test automation: Invest in testers
Figuring out which tests to redesign and which tests to retire requires top-notch testers, Galen said. "Test managers should attack the evolution of the application with testers' brains." His advice was to hire and develop testers who understand their value is not in testing per se, but in making decisions. "You want testers who ask questions such as: How would the customer use this application? What if we built the wrong application?" Galen said. "You need people who aren't afraid to move to the front of the line -- who aren't afraid to go back and help with requirements."
This is what test managers should invest their time in, not simple ROI cases, Galen said. "Am I opposed to test automation? Am I opposed to return on investment? No and no," he concluded. "But this model of focusing on ROI for test automation, I think it is insane."