Success with continuous delivery (CD) depends on four prerequisites. In the first article in this two-part series,...
experts discussed the importance of a well-established iterative development process. In this article, those experts focus on three practices: automated testing, infrastructure as code and a staging environment.
Software test automation isn't an "all or nothing" affair. Most teams automate some steps and rely on manual testing for others. But continuous delivery demands a deeper commitment to automation. Why? Because, by definition, testing must take place continually, and the only way to manage it cost-effectively is through automation, said Tom Poppendieck, co-author of The Lean Mindset. As each increment of code is released, tests kick off automatically, making sure not only that new code works, but that it integrates properly with the larger codebase. "When software testing is automated at multiple levels, the risk and cost of making changes becomes dramatically cheaper," Tom Poppendieck said.
The more you automate the test process, the easier it is to reach the goal of delivering software continuously, said Carl Caum, a prototype engineer at Puppet Labs, which sells configuration management software. He is not suggesting that software pros eliminate all manual tests, but said they should move steadily toward deeper and deeper levels of automation. "You start with acceptance testing. What defines success in our software? Write down your acceptance criteria first, and then automate that testing process."
The key to success is automating as you go. Often that means conducting manual tests; then, where possible, turning manual steps into automated ones. He offered an example: "You do some manual, exploratory testing. When you come across a workflow that requires the same four steps every time, automate it," he said. "The first time it's exploratory, second time it's a pattern, and the third time it's automated."
Continuous delivery requires automating most test procedures, said Stephen Forte, chief strategy officer for mobile tool maker Telerik. "But at the end of the day, for public-facing applications, you want to have a human being look at the screens. Does the layout work? Are there any spelling mistakes?" These are things automated testing can't catch, he said.
Prerequisite: Infrastructure as code
Just as continuous delivery relies on automated testing, it also depends heavily on a practice known as infrastructure as code. This practice refers to the ability to express the state of the production environment in code, instead of manually determining which servers are running which versions of which software. "In the old days, this [process] was difficult. There were binders with [sticky notes] on them, and they were never up to date," Caum said.
An October 2014 report from Forrester Research, "Competitive Pressures Drive The Business Case For Modern Application Delivery," agreed that expressing infrastructure as code is a key aspect of continuous delivery. "Nonstandard infrastructure is costly: The effort and cost to diagnose production incidents that result from configuration differences between the development, testing and production environments are another form of unnecessary waste. Standardized environment configurations and automated provisioning dramatically reduce or eliminate the problem."
In continuous delivery, expressing the production environment in code is essentially the next step in automating the ongoing test process that takes place behind the scenes, Caum said. "You test the code and it works, but you need to know what happens when it's deployed across 1000 servers."
Prerequisite: A staging area
The final prerequisite for software organizations preparing for continuous delivery is to set up a staging area. "It's an environment that is very close to production environment, but not quite the production environment," Caum said. It has two purposes. The first is really a business purpose, said management consultant Johanna Rothman of the Rothman Consulting Group. "Feature after feature gets integrated into the code and passes all the tests. But that doesn't necessarily mean you're ready to release that feature to the customer," she said. "That is a business decision."
The second reason is that the staging area offers an opportunity to find last-minute errors before the production release, said Litded Davis, director of the project management office at Outsell, a company that offers a SaaS product for the automotive industry. "We release to a staging area every three days," she said. At that point the software has been tested end to end. But holding it in the staging area gives product managers a final opportunity to make sure everything is right. "We look at the flow of the screens to get a true sense of user experience. Sometimes we find error messages that are confusing or inconsistent."
Davis said Outsell is moving toward delivering software updates every day. "It's taken a little bit of time to get there, but we are getting into the rhythm."