How many helpdesk (HD) tickets does your IT team get right after software is released into production? Some organizations’...
post-release HD volume is 50% of the annual total.
Too many organizations today strive to create software quality in development and then are unable to preserve it during release and in production. This problem is common, and one root cause is development and operations teams not working together before a software release. The teams are unclear about what production QA processes are and who should be handling them, according to experts.
Disorganization is widespread when it comes to getting applications into production and maintaining application quality once it’s there, according to Randy Clark, CMO of IT Process Automation software company UC4. He has witnessed the above-mentioned helpdesk ticket explosions.
Pre-deployment DevOps teamwork
“If you have any kind of a back end that's going to be hit by lots of people, the risk of deployment becomes significant,” said Deiner. “That’s why a lot of DevOps teamwork should happen in the cusp right before deployment.”
What goes on when Dev and Ops teams meet? For one thing, Dev and Ops teams collaborate on monitoring strategies before release so Ops knows what is going on with production apps. “That means monitoring key capabilities, some nonfunctional requirements and other operational aspects,” said Howard Deiner.
The development team also provides operations with migration scripts, so Ops can test before production, according to QA director Joe Strazzere. Ops should be briefed on development testing, too. Clear responsibilities should be set. For example, development teams should not be messing with schemas as the app goes into production, as it’s the job of database administrators to monitor them.
Before application release, Strazzere’s operations engineers require the development team to provide preparation assistance. “Production will accept a new or modified application only after being trained and only after a Run Book has been created for the application,” he said. The Run Book explains system details and how production is expected to respond in certain cases.
Pre-deployment and production testing is done by both development and operations teams, but they are testing different things at different times. Production testing centers on processes, like planning, control and automation, said UC4’s Clark. Development teams don’t get involved in processes, but test application code and other product-centric tests.
Strazzere’s production engineers test code during the pre-release and release period. This short-duration, release testing is usually handled only by QA engineers. During this period, Deiner noted, developers move from nitty-gritty tests, like unit tests, to smoke tests, a broad, shallower test format to ensure quality is stable.
Once an application is in production, most organizations move the responsibility for it from the development team to operations. Deiner would like the lines to be blurred a bit and advising extending continuous integration into production application quality processes. “With CI, we can automate our release processes from code check-in to deployment.” Extending CI servers from testing, to release and then into production processes can improve software quality throughout the SDLC.
QA by any other name
As application QA responsibility moves from development to operations, what is the process of production QA and testing called and who’s in charge? SSQ sources call this production QA, production test and ALM test management, and the person in charge is the ALM QA director, project manager and software architect, among others.
Other monikers for the process and manager abound. In a post in the SQA&T blog, a test pro wrote that he and his manager were stumped about what to call production QA in their status reports. Responding, software pros said their organizations used such terms as operational quality, production monitoring, and production support team. One suggested that the process be called Customer Experience Assurance. Others on SQA&T have seen it called production monitoring, production support, sustained engineering and site confidence.
Production software QA approaches
Strazzere’s production QA process is called production monitoring and is done by production engineers who report to the director of production. Production engineers monitor production systems to ensure that time-critical process proceed as expected. They jump in to deal with the occasional interruptions so that our service-level agreements (SLAs) can be met.
There is just a little overlap of production monitoring and development QA in Strazzere’s approach. “We have lower-level testers that check some of the weekly or monthly files before they are moved along to the next part of the system,” he said. “But that ‘QC’ role is diminishing as we automate more and more of the systems.”
Automating production monitoring and quality control provides such benefits as being able to get rid of manual intervention, thereby speeding intervention timeliness and reducing labor and downtime costs. Also, Strazzere said, new systems and enhancements to existing systems can get into production with a smaller investment in infrastructure.
Automation can also solve other problems. For example, manual in-production actions don't scale. “If you have too many of them, you must have a huge team ready to respond at any time,” Strazzere said. “They are best used in smaller doses.”
Automation can also provide an effective feedback loop. Otherwise, the fixes that the production engineers apply won't make their way into a future revision of the code, and manual intervention could be required indefinitely.
While he’s bringing automation into production QA and elsewhere, Strazzere is not assuming that he should automate everything. “Sometimes it makes sense to have real people, instead of automation, as a small part of the system,” he said. “This is a business decision, not always a technology decision.”
Agile methods in production QA
Strazzere’s organization doesn't use any formal Agile methods, but does use Agile’s less formal methods. “For example, we have manual production processes that revolve around an Annual task. When we are doing ‘Annuals,’ we have daily stand-up meetings, and very short -- usually two-day -- time-boxed turnaround for some production tasks of creating output and checking them,” he said. “But we do little that most software groups would term Agile with a capital A.”
Often the impact of Agile is seen in just the way Strazzere describes, Deiner said. Over the years, parts of the methodology have become common practice. Agile’s primary contribution is enabling the DevOps process in the first place, he added. “There were two camps, and both didn’t like each other,” he said. Agile has created a more collaborative environment. Today, he said, developers have more understanding of the operation’s needs and problems, and operations people are more involved in development. The result is better application quality once an application goes into production and better monitoring and quality assurance once software is in production.