peshkova - Fotolia
LAS VEGAS -- The hard truth about your DevOps initiatives: Tools won't get you there. High-performing DevOps teams resolve to solve their people problems, then augment their capabilities with tools -- not the other way around.
DevOps practitioners and leaders at the DevOps Enterprise Summit here in October discussed success stories, challenges and techniques used in various IT organizations -- and many of those conversations centered on people. What differentiates a DevOps case study from a horror story, many said, was the effectiveness of transformational leadership to get workers across the business to buy in, definitively and continuously.
"Humans are under-served and under-represented, and it is really humans that are going to propel [DevOps] forward," said Jayne Groll, CEO of DevOps Institute, an organization that provides education to the DevOps community. "The companies that are most mature are really the ones that invested in great tools -- but they also invested in great humans."
You can spot a mature DevOps organization by how it performs across various criteria, according to the Accelerate: State of DevOps Report 2019. The yearly study, released by DevOps Research and Assessment, revealed that the percentage of surveyed organizations considered elite DevOps performers nearly tripled, from 7% to 20%, in the last year. Compared with low performers, elite organizations deploy code 208 times more frequently, and they have a lead time from commit to deploy that is 106 times faster. And elite performers don't trade stability for throughput; they also excel on mean time to recovery and lower change failure rate on several orders of magnitude, the report said.
Elite performers overcome the people challenges behind a DevOps transformation to implement the standard hallmarks of viable production-ready development, such as CI/CD, automated testing and thorough monitoring. There's no all-purpose way to deal with roadblocks in DevOps initiatives, but common threads among high performers reveal some trends.
Train up your teams
What differentiates high-performers from less-productive teams? It all starts with a dedicated approach to upskilling team members, Groll said, which can include online resources and other techniques. "Companies that have really adopted a digital approach, an immersive learning approach, are much more successful."
There are several ways for organizations to establish community structures to promote learning, both to identify common internal struggles and be more resilient to personnel or product changes.
According to Accelerate, more than half of elite performers use communities of practice -- small groups of voluntary practitioners -- which was a common thread among attendees at the conference, as well. The report also named bottom-up DevOps initiatives and proofs of concept as common elements among elite performers -- those who nailed DevOps.
Experiential learning helps team members understand each other's roles, Groll said, as internal study groups and newsletters share success stories and help formulate action plans. She added that mature DevOps adopters not only avoid knowledge silos, but also actively improve soft skills like effective sharing, as a means of maintaining open collaboration.
"You really want to commoditize the knowledge across the entire organization so that it's accessible to everybody," she said. "Organizations that are the most successful have done that. They've done that with mentoring, they've done that with soft skills, they've done that with a lot of different approaches."
Step into the dojo
DevOps dojos, in which cross-functional team members gather for hands-on structured training, are popular at large companies. These training sessions, popularized by Target, typically occur in one location and can last several weeks. DevOps dojo cycles enable members from different teams to deliver their work in a more efficient way and gain long-term skills.
Target's not the only major retailer to see the value in DevOps initiatives and discover the need for a human focus to address technological demands. Bryan Finster is a staff software engineer at Walmart Labs, the software development arm of the retail giant, in Bentonville, Ark. Finster facilitates the dojo at Walmart Labs, and is a member of the DevOps Dojo Consortium, a learning community.
In his role at Walmart Labs, Finster deals with the problems that stem from approximately 1,000 globally dispersed teams, some of which work with code created as long as 30 years ago. In his effort to spread DevOps practices throughout the enterprise, Finster has found that his path is to equally educate and evangelize.
"I just really try to understand how people operate, and then give them what they need, or give them information in a way that causes them to adjust," he said. "[We want to] grow other evangelists in the company who will then also push and create those resistance cells you need for real enterprise transformation."
To provide DevOps dojo participants with the best knowledge and experience, Finster said, he must illuminate, communicate and, ultimately, clear challenges for both the tech side and the business side. "It [takes] a little bit of independence to get it done because you have to coach in both directions, and show people pain in both directions," he said.
Finster and the dojo team help Walmart Labs workers assemble and manage tools, deal with edge cases, conceptualize the value stream and, most importantly, construct efficient tests. The QA part of the equation, he said, is often misunderstood or underemphasized in the industry.
"It's layers and layers and layers [of tests], and they have to really understand how to architect that for speed and reliability -- not coverage," he said. "Continuous delivery is testing. So, if you're not focusing on how to get that done right, if you're focusing on speed, you will fail."
Misleading metrics set teams back
Just as tools purport to solve a variety of DevOps challenges, so too do many teams put their faith in overly simplistic performance measurements. "People will see a metric, and they'll think they understand that metric, and then they'll go misuse that metric," Finster said.
For one, DevOps metrics that track speed are misleading, he said. Not all teams can control their deploy frequency -- think of development in regulated industries. Also, if speed comes at the expense of quality, you're not moving faster at all. "Some people will focus on delivery and say, 'Hey, look how fast we're delivering,' but it doesn't matter if it's blowing up all the time," he said.
Finster called velocity, throughput and number of features meaningless metrics. Instead, he said, teams should focus on how quickly they can push code to the master branch of the version-controlled project, and how small they can make their batches of work. These efforts, he said, can reveal waste in the system, such as large stories or task sizes, or poorly understood dependencies.
"We spend a lot of time on educating people on what metrics are used for, how to misuse metrics, how to use them in offsetting ways, and just trying to get people engaged and learning a little bit, so they can make better decisions," Finster said.
But, while enterprise DevOps adopters have rallied around dojos, Finster warns that dojos can become anti-patterns -- as can any DevOps initiative. Knowledge should flow independently of arbiters, as those in the know now are just as prone as siloed teams and old-world leaders to falling behind the trends. Organizations should avoid knowledge gatekeepers and information hierarchies.
"We're not the center of all knowledge; we're just a conduit trying to blast knowledge out as far as we can, every way we can," Finster said. "We're not experts at this, we're just further ahead than [others] are. When we no longer can stay ahead, then we should just go back into product teams."
No pain, no gain
An Agile or DevOps transformation can be a fraught undertaking, but that's a good thing. Organizations must address their people and process challenges head on to fully contextualize a roadmap.
"Predictably, the first thing you're going to have is pain. I don't think people are usually prepared for that," said Jeffrey Fredrick, CTO and head of product and marketing at TIM Group, a financial services company based in London.
When team members hear the word transformation, they immediately equate it to their individual experiences. The result is often mounting pressure, more stress and no relief in sight. Most people, Fredrick said, would rather be competent with an inefficient system than deal with the short-term stress of a new one.
Worse still, many team members endured failed or minimally successful organizational transitions or leadership turnover in the past, which calcifies their resistance to change. No level of pre-planning for a perfect DevOps initiative will alleviate those very real concerns.
"Even with a good plan, you're also doomed to failure most of the time," said Douglas Squirrel, a London-based software consultant. Squirrel and Fredrick collaborate on the Troubleshooting Agile podcast, and co-wrote the book Agile Conversations, which is scheduled to publish in May 2020.
Squirrel and Fredrick say conversation can either confuse or empower, and that too many organizations fail to build trust and alleviate fear in a DevOps initiative. High-performing DevOps teams, they said, power through their communication challenges and actively hunt down ongoing issues.
"The most successful organizations don't hide conflict -- which is, of course, exactly the opposite of what everybody is trained to do," Squirrel said.
Ultimately, some will onboard quickly, and others will need to see a proof of concept. The former group can rely on evangelists to help the latter through their challenges.
Let developers and techs loose
While Finster of Walmart Labs aims to create a shared understanding of how different roles support each other, there's a certain point where you can't hand-hold anymore. Dev teams must try, experiment, fail and repeat.
"Don't solve problems for development teams; give them problems to solve," he said. "Quality comes from ownership."
Just as DevOps centers on continuous -- integration, testing, deployment, delivery and improvement -- so, too, must high-performing teams continuously optimize their performance and processes. Inertia, or a leadership turnover, can derail team progress.
"I think all of us have those challenges," Finster said. "How do we maintain? How do we bake it into the culture, so it's resilient?"
Ultimately, the more support engineers can receive from each other across organizational boundaries, the more everyone can align around vital business goals.
"The people on the front lines are the ones doing it," he said. "All of us are supporting them. They're our direct customers in security. Your customer is the developer on the front line for increasing value. They're not your enemy."
Why one professor believes IT burnout is a dire problem
Ignore the buzz words -- Agile and DevOps are all about outcomes