We’ve certainly heard a lot from ALM vendors and leaders about the value of automation in Agile ALM, but are CIOs getting the return on investment that they’d hoped for when they went down the automation path? In this article, we will explore what considerations that executives and senior managers should take into account in order to get the best return on their automation investment.
The different types of automation throughout ALM
The first thing to consider is the varying types of automation that can be done throughout the application lifecycle. Often “automation” in ALM is thought to require a large investment in a tool suite, and indeed, many of the ALM tools do tout “automation,” with varying meanings. The automation might refer integration of data from tool to tool or perhaps the generation of reports or dashboards but quite often in ALM, the term, “automation” is referring to test efforts. Even when narrowed down to test automation, the types of test automation can vary greatly.
Having developers automate their unit tests using test-driven development will not require any additional spending and will provide the basis for an automated test regression suite which can be used as part of a continuous integration build. Continuous integration has been shown to allow for quicker builds and deployment at higher quality. This type of automation almost certainly will yield a positive return on investment.
On the other hand, investing in tools that help automate your GUI testing can be very costly, and when a test case “breaks,” often it’s the test case that needs maintenance rather than the test finding a bug in your application software.
For any automation effort, determine what it is costing to do the task manually and how much it would cost to automate that effort. Then determine how much money will be saved, perhaps in time or other resources, by automating that task. What value will be added? When determining costs, it’s important to take into account how much it will cost to maintain your automation and any automation tools.
I’ve seen vendors describe test automation ROI in terms of number of automated tests that can be run compared to how long it would take manual testers to execute the same tests. However, if those tests are not finding any bugs and are costing a lot of money to develop and maintain, your organization is not saving money.
Senior editor and one of the contributors to the book, How to Reduce the Cost of Software Testing, Matt Heusser notes that test automation does not reduce dollars spent. He explains that he doubts automation has broad applicability in decreasing testing costs, though automation may lead to improved quality and more frequent releases. “But in our research we found that in general, test costs – dollars out the door – don’t go down when you implement an automation strategy. There are certain verticals where it does, but not broad horizontally,” Heusser says.
As he notes, though automation is not going to initially reduce your costs, it may indeed provide benefits such as improved quality and improved time-to-market that are hard to quantify when trying to figure ROI. The key is determining the right types of tests to automate.
Consultant Jon Hagar says that, “automation will cost three to five times what doing manual testing takes on the first pass of testing efforts, but if you have regression or O&M efforts, you may see that money back. Automation like all things in testing is justified some places and not others.”
Practitioner Tomas Schweigert brings up the very valid point that one must consider how often a test (or any automation effort for that matter) will be executed in order to determine ROI. He says that automating a test that will only be executed once may not be cost-effective. Yet, he offers, “If the test is executed more than seven times you will save money with a high probability.”
Though Schweigert uses “seven times” as an example, others feel that if you are running the test even twice, it makes sense to automate. Of course, the question to help make this determination would be considering how long it takes to execute the test manually and how long it takes to automate and maintain that automation.
In David W. Johnson’s tip, “How to evaluate testing software and tools,” he steps through a process with specific examples, including an automation strategy, of evaluating tools for ROI. He first recommends doing a needs analysis, identifying candidates, assessing candidates and then implementing a solution. He says, “Following a few simple guidelines and applying a common sense approach to software acquisition and implementation will lead to a successful implementation of the appropriate tool and a real return on investment (ROI).”
With the variety of types of automation available throughout the application lifecycle, it’s impossible to say that every automation effort will yield a positive return on investment. In order to assess ROI, senior managers and decision makers need to understand the needs and carefully consider the costs, including maintenance costs, as well as the hard-to-measure benefits that can be gained from automation.
What ALM automation efforts are you finding are yielding the best return on investment? Let us know by sending an email to firstname.lastname@example.org.