DevOps principals are well-suited for smaller startups without a lot of legacy cultural processes and technology through which to work. But at the DevOps Enterprise Summit 2017 in San Francisco, Northrop Grumman, a leading government contractor, discussed how their DevOps teams were trained to drive competitive advantage, improve security and empower IT professionals.
Northrop Grumman actively scales Agile and DevOps principles across a $24 billion organization with 65,000 employees. But, like in many large organizations, it took years of hard work and sweeping changes to make that possible.
It began in 2005, when Northrop Grumman Fellow Suzette Johnson was tasked with defining an engineering process for a new mission that had to be delivered quickly. The traditional application development model was not the right answer.
"We felt that it was more risky not to change than to keep doing things the same way," Johnson said. "We took that risk and went with it."
Test early and often
Northrop Grumman quickly scaled one of its core DevOps teams of five members up to 125 members, creating a team that consisted of several smaller teams. But then the company started running into a variety of problems around integrating the code from these teams. Every time one of the DevOps teams made a change, things broke across the group. This lead to the use of continuous integration, which was not a trivial undertaking at this scale. To address this, they started testing earlier and more frequently.
But, as often happens, that caused more changes. The DevOps teams were able to get the software to work on Northrop's infrastructure, but ran into challenges when working with different software from other government contractors.
The DevOps teams now run over 15,000 automated tests per day.
Provide templates for auditable processes
As more program groups at Northrop Grumman began to hear about these early successes, they wanted to start digital transformation initiatives to achieve similar results.
"We needed to start being prepared to provide the infrastructure to prepare for change," Johnson said during her presentation at DOES 2017. "When you start to grow across the organization, you need to think about how we are building that infrastructure."
A key part of this was establishing a foundation of certified tool sets, many of which ran on public or private cloud infrastructures. This made it easy for new DevOps teams to adopt the best tools required to work with their project and workflow requirements.
According to Johnson, "In a larger company, you cannot have just one solution. You need to find out what are different solutions that can meet the majority of [the] needs in the company."
Northrop Grumman also established a center excellence that helps new DevOps teams build out their infrastructure. Part of this includes a focus on auditable processes.
"Those are really important because we are a government contractor and we take audits very seriously," Johnson said.
Train across different roles
It was also important to make sure everyone was trained.
"Building quality was not something that could be tacked on at the end," Johnson said. The company couldn't just say, "go be a DevOps team."
Initially, this training focused on the engineering level, but Northrop Grumman found it also had to start training at the management and leadership levels. The company needed leadership to understand their responsibility in order to provide the kind of support this new model of development required.
"As an organization, if you are only changing at the grassroots level, you don't always have the authority to make changes," Johnson said. "We depend on leadership to help with those kinds of changes." Because of these efforts, Northrop Grumman has now trained over 15,000 people to be part of DevOps teams.