Microsoft has claimed a leadership role in the DevOps space as a big-vendor DevOps tools provider along with competitors, including IBM, CA Technologies and Hewlett Packard Enterprise, in addition to a bunch of smaller companies helping to lead the charge.
A key part of Microsoft's ascension to leadership ranks is Microsoft expert Sam Guckenheimer. Guckenheimer, who has been at Microsoft since 2003, is the product owner of Microsoft Visual Studio Team Services (VSTS) -- the successor to Visual Studio Team System. He came to Microsoft from IBM, which had recently acquired Rational, where Guckenheimer spent several years honing his application lifecycle management (ALM) expertise. He has 30 years of experience as an architect, developer, tester, product manager, project manager and general manager in the software industry.
So, how did Microsoft become a DevOps leader? SearchSoftwareQuality caught up with Guckenheimer for a wide-ranging interview. Nicole Forsgren, CEO and chief scientist of DevOps Research and Assessment, also joined the discussion.
I know Microsoft is using its software development lifecycle/DevOps tools at scale internally, but how are other companies using the Microsoft team development stack at scale?
Sam Guckenheimer: VSTS, which is the hosted service, is now used by about a dozen companies at the above-1,000-user level. We had someone use it for 5,000 users. Most of them have migrated from TFS [Team Foundation Server] on prem, but if you look at our whole population of VSTS users, the minority started with TFS. There's a huge distribution of companies. You'll see companies like Shell totally gung-ho on getting everything into a SaaS and cloud model for their users. You'll see others in a transition. Most of our customers are some place along the transition. We have some smaller customers who are cloud-native; that's where they began. These are typically startups or startup projects within large companies.
What are some of the trends that you're seeing in terms of what companies are looking for -- especially the ones with some level of expertise?
Guckenheimer: One of the most frequent conversations I get pulled into is the DevOps culture conversation and how companies should organize. One of the experiences is that companies start doing continuous integration and continuous delivery, but they still have separate QA departments or separate ops teams. So, they are siloed. They recognize a problem because they have problems collaborating or with handoffs.
I think with that you see a big pattern of people going to continuous delivery and then monitoring results and usage and using that to inform what they do next. This is where it's best to have quantitative and qualitative -- or system and survey -- metrics together, because you can measure patterns. But you also need to ask people what they're doing.
What impact does DevOps have on security?
Guckenheimer: The high [DevOps] performers are shifting security left in a big way. The classic security theory was that it's the CISO's [chief information security officer's] problem -- we scan before release, we generate this big stack of issues that then get triaged out and so forth. The new practice says we want security done from the time the code change is made or the infrastructure change is made. So, we will automatically check security on the open-source packages we consume. We will scan for security errors like credentials in code. We'll do incremental code analysis and work these things as far left into the developer's workflow as we can.
Of course, they continue to protect the perimeter, and they continue to use antivirus and use secure identities. But in addition, they treat security errors like all other quality issues and say, 'We want these handled when the code is fresh, when you're working on it.'
A couple of my favorite partners include WhiteSource, which lets you put into your pipeline an automatic scanning of all the open source that you're using. You get an inventory, and you get notification of any vulnerabilities. And if subsequent vulnerabilities are found in the wild, you get notified that that can break the build. The other partner is Aqua, which is effectively doing that for containers as they're composed by the developer. Those are both small startups that are growing like wildfire because they are hitting a sore spot.
Nicole Forsgren: And what we're seeing is that high performers have 50% less security remediation. By shifting left on security, they're getting higher quality, more secure software. And it's not slowing down speed or affecting stability.
How have the DevOps tools evolved? What needs to come next -- as in, what are some of the pain points that need to be addressed?
Guckenheimer: First, how they have evolved is that we've gone from worrying about continuous integration several years ago to thinking about continuous delivery as something we really need to make easy. And we need to make it easy for all the targets people want to deploy to. So, that means the cloud, on prem or whatever. We also need to work that into the mobile flow, which is newer and messier, because with mobile development you have the different form factors and the app stores in the middle. That's unlike a server or cloud service where you just deploy. With mobile, you get your software onto the device via the app store, and that means you need an intermediate mechanism to get beta distribution to users. And you need different telemetry on it and so forth. So, mobile DevOps is certainly one of the big things.
Then, people have generally gotten pretty good at monitoring for quality of service -- making sure everything's available and doing application performance monitoring. However, there is not right now a great way to take usage patterns and to turn those into hypotheses about what to do next. So, the whole idea of making hypotheses for new development something that is practical is something that is emerging. There's another great company called LaunchDarkly that we work with as a partner that is taking the lead in terms of that workflow. That's something people really want to do a lot more of, and we'll see a lot more of over the next few years.
What's the impact of microservices, containers and serverless on DevOps?
Guckenheimer: There are two directions of impact. One is the general area of helping you go faster. By doing serverless or using containers, you get this very fast flow from development to production. And you eliminate a lot of the complexity of worrying about configuration and environments and so forth. However, you get a second benefit that that whole approach lets you deploy individual services faster and lets you experiment faster. For instance, you can do two versions of an app in parallel and deploy them side by side and use a load balancer to split cohorts to the two. And you can compare what they are doing and compare treatment against the control and make a decision on which one to move forward. That's been a big rising trend and will become more and more how people work.
Can AI be effective in assisting with DevOps?
Guckenheimer: I think AI can be very helpful with production data. For example, for our telemetry across Azure services, we're collecting about 2.5 petabytes a day of data. There is no way that human beings can go through that for anomalies. To look for unusual patterns in that data is certainly something where machine learning helps and can steer you to look at the right stuff.
Where I see a lot of hype from analysts – and I think it's a dead end – is using AI predictively the other way, saying we're going to look at test results and we're going to improve our testing by using AI. What I think people really do is they make it faster, and they instrument production, and they think about what testing is redundant and what testing is missing, and they keep improving it in a loop based on outcomes. You don't really need AI for that. That's oversold.
I think, on the production side, where you have this unpredictable pattern of data, I think that is where we can see a lot more use of AI.
How does Azure Stack play into your DevOps strategy?
Guckenheimer: Azure Stack, at the simplest level, is another deployment target. The value of Azure Stack is that enterprises can effectively deploy a cloud that is private or hybrid or disconnected, and they will then manage that in their data center and be able to get a lot of the benefits of the flexibility of cloud infrastructure, but not have to cross the boundary into the public cloud. It is something that a lot of folks, like government agencies, are very interested in. For us, it's another deployment target.
As a Microsoft expert, can you summarize how far the company has come in using its own development and DevOps tools?
Guckenheimer: The biggest flip was about three years ago. Satya [Nadella, CEO of Microsoft] and the senior leadership team said we are going to reverse a company policy that every division can be responsible for its own engineering system. They specifically said we're going to have One Engineering System -- 1ES, for short – that everyone uses. It will be world-class. It will be the basis for what we then deliver to customers.
Sam GuckenheimerMicrosoft DevOps guru
The vision with that was that anyone in the company could offer to make improvements. The most visible thing we did like that is the announcement and open-sourcing of GVFS, the Git Virtual File System, where we did a 200x to 300x performance improvement on Git for large repositories. That was motivated by WDG -- the Windows and Devices Group -- needing to deal with the size of Windows repositories and not being able to use Git as it was. So, they collaborated with us, and we did this joint development and open-sourced it through the Git community process.
But there are plenty of examples of where engineers at Microsoft, instead of saying, 'Fix this for us,' are now saying, 'Let us contribute.' And the default now is that anyone in the company has access to any of the code in the company, which is 180 degrees from where it was four or five years ago.
So, there was no handing off between divisions?
Guckenheimer: There is this great drawing by an engineer at Google of company org charts, and the one for Microsoft had all the divisions pointing guns at each other. It was really fragmented; people had different processes, different tools. Going back to culture, it was very hard if you were in Windows to move across to Azure or to Office because things were really different and the goal was to make that seamless.
Another culture thing we do that has had a big impact is, instead of doing an annual company meeting, we now do an annual hackathon week. People sign up to join hackathon teams. That is actively promoted as a way to look for your next job. People are encouraged to commingle across org boundaries. It's really great, and there's a ton of energy. A lot of product ideas come out of it. It's really different. Five years ago, giving everyone a week to go off and do something different from what their management chain had said was not what you did.
As a Microsoft expert, would you consider Microsoft a leader in DevOps?
Guckenheimer: Of course. If you look across the tools spectrum, certainly among the cloud providers, I think we have by far the best DevOps tools solution. If you look among the specialists, there are great companies that light up in particular areas, but I don't think they combine the breadth with the capability. We're trying to achieve both best of breed and best coverage, and I would rate ourselves as being at least in the top two of any area you pick.
But you were later to the game.
Guckenheimer: Yes, in some areas. We were by no means the first to offer Kanban boards or what have you, but I think we've caught up with the best Kanban experience and certainly the best Kanban experience with Git workflow. We regularly rate ourselves on that. We weren't the first managed Git provider -- GitHub was. But we are totally competitive now in terms of capability. There are all these companies trying to complement GitHub with continuous integration and continuous delivery; we do both. We don't try to compete with GitHub as being the forge where you put all your open source.
Microsoft partners and DevOps
Windows DevOps tools in the enterprise
Microsoft, Red Hat team up on DevOps