Does the DevOps movement shortchange QA? My short answer is no, but the devil is in the details -- and in reality. On the surface, proponents describe DevOps as "global company collaboration." It is basically Agile,
DevOps essentially has the same motto as Agile: to deliver code more quickly, more cheaply, more frequently and with greater reliability. It ventures to include the operations or implementation team with development and quality assurance in a collaborative team so that communication flows not only between development and quality assurance but also to operations.
Teams communicate and collaborate constantly, rather than only when something goes wrong or when a customer implementation fails. In this article, we'll cover three problem areas in adopting DevOps: the merging of roles, the dilution of software testing effectiveness, and the proposed automation of the install process by coding in infrastructure details.
Delivering faster and more reliable products by coding in infrastructure
Coding in the infrastructure enables companies to engineer the entire development application cycle. It allows engineering to control the process more inherently and completely. By developing the install, setup and creation portions that operations teams struggle to cope with in engineering, the bottleneck of broken implementations should be reduced. After all, if you put engineers in charge of coding from top to bottom it seems logical that the workflow becomes more reliable. Engineers develop automated scripts for QA personnel to execute in their test systems first.
In this way, assuming you dedicate enough time for issues, QA will test the install and implementation first. In theory, while testing the new code, QA can also test the setup and implementation, eliminating the need for the team that controls QA environments and saving time, money and frustration. It may eliminate the seemingly ongoing issue that QA environments aren't set up right and testing has to work around, past, over and through continuous setup issues.
If QA tests the code before operations personnel attempt to implement the upgrade or release at a customer site, this would, in theory, also eliminate the errors made by operations personnel. Does it replace the need for training for operations? No. Does it replace the need to provide after-hours support from engineering to operations personnel? Perhaps it won't replace that need, but it may reduce the frustrating late night hours where engineering is trying to find the error or issue in a typically complex setup.
Coding in the infrastructure and testing it prior to attempting to implement it sounds like a logical, time-saving proposition. However, how it works in reality may depend on the complexity of the software system. A large health care software system contains more mission-critical, complex code connections than adding a shopping cart functionality to a Web page. DevOps could save you time and frustration when the infrastructure is coded in and the install and build-process code is tested prior to implementation. The amount it saves you likely depends on the complexity of your software.
Merging roles: QA, developer and operations
Can one person do all three jobs? DevOps merges roles using either pairing or the general creation of a role that combines development, testing and operations into one function, one job, one resource. So, if the theory is that you code in the infrastructure, then test it and implement it -- can one person fulfill all three roles? Although I see the need or desire to affirm that notion, I don't believe it's realistic. If a person is a good engineer, they tend not to be good testers.
More resources on DevOps:
Delivering quality with DevOps as Agile
Move to DevOps to boost ROI and speed deployment
Public cloud influences move to DevOps
Granted, that likely is not always true, but in general it's very difficult to tear apart something you've created. But could an engineer also install and set up the customer implementation in place of operations personnel? Technically, the answer is yes. But in reality, do you really want engineers communicating directly with your customers? When will they have time to develop? If they're outgoing and have customer service skills, then yes, by all means. However, if they are like most coding professionals, making upfront personal contact is not their strongest skill. You may end up with greater problems than just a slow, painful implementation. Operations personnel tend to have stronger people skills than your typical engineer or QA tester.
That being said, having QA, engineering and operations on the same team developing and testing together certainly makes perfect sense. It cannot hurt to have direct communication between all three parties on the team. Not only does it save time, pain and effort, but it also builds support lines. When the operations personnel perform their implementations, they already have team members whom they can contact for assistance. Each team member contributes to the customer's success, and the operation team is not left holding the bag. The team holds the bag. The advantage of communicating with your own team members from start to finish is common sense. It can only provide a more reliable product that works once it's installed. The company suffers less pain and looks good in the eyes of their customers, and everyone wins.
Diluted effectiveness of testing
Can combining the three roles produce a quality product? Or is the testing the piece that gets diluted, removed, abandoned or merged to the point of nonexistence? If you follow DevOps and decide to combine your engineer, tester and operations person into one single resource, is the effectiveness of any one role lost? If so, which one do you think it'll be? I can almost guarantee that testing will be the piece of the puzzle that's diluted to the point of uselessness.
Not that testers must always be separated. But testing is a special skill that engineers who code products don't possess. Yes, operations personnel assist with testing, and even combine the testing role with the operations role. The problem is that the function that ultimately wins out is the one that affects the customer immediately. The need for speed in delivery and implementation can negatively impact testing; for example, if something occurs that slows down development, then testing gets abbreviated so operations has time to install. The need for speed in delivery and implementation, if something occur that slows down development then testing gets shortened or squeezed so operations has time to install.
Even with mission-critical systems, there's a temptation to test while implementing -- you know it will happen. Software more critical to life, property or business operations should not be tested during implementation. In DevOps, the pressure to deliver quickly is the same as all other methodologies, but I think the temptation to combine the roles into one is a mistake that will reduce the effectiveness of testing.
From one perspective, DevOps is the same as Agile, but it doesn't leave operations out of the team. Having each group in the team only makes sense. Making this move alone should save significant time and energy, and provide software that's more fully developed, tested, and reliable for customers. Merging the roles of developer, tester and operations expert into one is not realistic. The skill sets are too different and too critical independently. However, can the roles work together on a team and provide greater communication and value? Definitely. And turning back to the original question: Does DevOps shortchange QA and testing? Only if you merge the roles; each role needs to exist independently and work together as a single team.
This was first published in August 2012