Automated testing in Agile -- and DevOps -- is as important, and elusive, as the Holy Grail. And with good reason: No development shop worth its salt can actually claim to be on the Agile and DevOps path without a healthy dose of automation. But that doesn't mean it's easy. And after all these years of Agile, automation still proves to be quite challenging.
Survey reveals test automation challenges
In 2008, software testing tool provider LogiGear did a survey of Agile practitioners to capture the "state of the union," and the company is currently doing a similar ongoing survey, this time in four parts. Dishearteningly, the results of part one show little to no progress when it comes to automated testing in Agile, according to LogiGear senior vice president Michael Hackett. Ninety percent of testers who responded to the survey said 50% or less of their tests are automated, 27% said just 10% or less of their testing is automated and only 5% said up to 90% of their testing is fully automated; 20% of respondents do all their testing manually.
These numbers don't surprise Steve Elliott, founder and CEO of Agile platform maker AgileCraft. When Elliott visits customers, he might see a lot of sophisticated, high-end automation or absolutely none at all. "It's one of those things that's really all over the map," he said. "Companies just really haven't figured out the automation in Agile piece yet."
But there's no question they need to -- and quickly. "There's even more pressure to be faster than ever before," added Forrester Research analyst Robert Stroud. "The velocity is just going to continue to increase and with it the demands on developers. Things are going to go as fast as the business can let them go." The answer to that -- continuous development/continuous integration (CI/CD) -- can really only happen if automation is in place.
Understanding the Agile testing process
But clearly that's easier said than done in the Agile and DevOps world. Automated testing in Agile has to begin with a complete understanding of the entire development process from start to finish. Companies can't just automate randomly; the process has to work seamlessly. When LogiGear did its survey 10 years ago, the biggest challenge with automation was management's lack of understanding the process, Hackett explained. And unfortunately, the exact same thing is still true today.
"There is a lot of misunderstanding about automation," he said. "So many people are completely unprepared, and if you don't have things already well along the path of significant automation in Agile, you are just going to get steamrolled. The real surprise to me about this survey is how much is still not understood -- like requirements creep and vague or bad user stories. Testers have been complaining about the same things for the last 20 years."
The survey also asked if the move to Agile -- and now DevOps -- has improved the development process. Again, the answers remained little changed over the last 10 years. Back then, about 20% of respondents said moving to Agile was an improvement, and that percentage is the same today.
"There are still a substantial number of people who think that things just aren't getting better," Hackett lamented. "The last two companies I visited had zero automation. It's not going to be good when companies finally wake up and are interested in modernization and digital transformation. It's going to be hard."
What you need to know about Agile at scale
Is Agile still relevant?
Why Agile and DevOps remain a work in progress