SearchSoftwareQuality.com's recent reader survey found that Agile adoption has slowed significantly amongst readers despite its growth in the industry at large. This apparent decline highlights some of the challenges associated with a transition to Agile, but also reflects a conservative approach to adoption-- which is key, say analysts, to its success.
According to the survey, 34% of respondents currently use Agile, and 12% have plans to implement Agile within the next 12 months. This is down from 57% and 42%, respectively, in 2010. This decline amongst readers differs from what analysts see. “Agile has been in the mainstream for awhile. People understand that it exists. If anything, the rate of adoption has increased,” says Michael Azoff, principal analyst, Ovum. “From our perspective, there’s an increase in adoption and a desire to adopt Agile, whether on an individual project basis or within a whole department.”
Matthew Hotle, vice president and distinguished analyst, Gartner Research, agrees. “The breadth of penetration into our client base is fairly high, but the depth is fairly low,” he says. According to Hotle, upwards of 50% of Gartner’s client base is using Agile, but less than 10% of work being done is actually being done in an Agile way.
Azoff says that many organizations claim they do “Scrum but.” In other words, they do Scrum, but they don’t do all of it. It is common for organizations to only put into practice parts of the methodology. The SearchSoftwareQuality.com survey supports this, with 52% of organizations that are using Agile processes reporting that they are still using waterfall methodologies, as well.
This is a common scenario, and it is possible that respondents to the SearchSoftwareQuality.com reader survey said they were not doing Agile because waterfall practices still play a significant role in their organizations. “Most organizations, at least for the foreseeable future, need to be multilingual. They’re going to have waterfall and iterative stuff,” says Hotle.
When Hotle talks to clients about transitioning to Agile, he recommends a conservative approach and “stacking the deck” in the favor of success. This means choosing a pilot project “that is not unimportant but not mission critical,” that will last three-to-six months. He also recommends putting the right business people and IT people on the project, and hiring a process coach. When the organization has success with one project, they can proceed with another. But in order to see success again, Hotle says the organization must have the same elements in place: the right people, the right project and a process coach. It can take a large organization years to make the changes necessary to become fully Agile, he says.
Ted Young, a development manager for a company in the financial services industry, says his organization has been doing Agile for four years, but it didn’t happen overnight. “It was a gradual process. When I came into the organization, we had a Scrum-like process but it was more ad hoc than anything else,” he says. Young says his organization is now 100% Agile, but offers a disclaimer: “For some definitions of Agile, there are areas where we’re not. We’re a little waterfall-ish. We’re intentionally not waterfall, and we’re intentionally trying to be Agile,” he says.
“Generally speaking, it’s not a smooth transition to Agile,” says Hotle. “It requires quite a dedicated concerted effort, not just by the development team but also by the organization. The transition is a cultural one, and changing an organization’s culture is not easy. You have to set expectations correctly,” says Hotle.
Indeed, these challenges are echoed in the readership survey results with respondents citing as their top Agile challenges resistance to change at 38% and unrealistic expectations at 37%. Both of these challenges may stem from the fact that a good portion of Agile organizations are using it because they were told to. According to the readership survey, 40% of respondents using Agile were mandated by upper management to make the change.
“Agile can create great excitement with so much buzz around the whole paradigm,” says Azoff. “Because there are so many pitfalls around it-- and the cultural ones being quite serious-- the pitfalls are not appreciated or planned for. Moving a waterfall team to Agile and just expecting them to pick it up overnight, that’s unrealistic.”
Changing the company culture was very much a challenge for Haya Rubinstein’s organization, which uses Agile processes for all new development. “First, the move from a hierarchical type of management to a team oriented type; secondly, all the processes were changed to suit Agile development using Agile process management tools such as Jira,” says Rubinstein, who is a test/QA manager in the technology/telecommunications industry.
Some business folks think that the move to Agile will result in huge productivity gains, or they fail to recognize that there is a learning curve, says Hotle. Management thinks that developers should suddenly be able to deliver in one month what took ten months to deliver under waterfall. Rubinstein, who cites lack of expertise as a challenge, echoes this misconception. “The lack of expertise is more an issue of estimation of how much can be accomplished within a sprint and making sure that all issues that are not just development itself, such as QA, are included as backlog items,” says Rubinstein.
Other challenges cited by SearchSoftwareQuality.com readers also point to a lack of expertise; namely, challenges with automation. The need for more automation is cited by 45% of respondents as the third biggest challenge related to change management. Fifty-five percent of respondents cited a lack of automation as their top challenge related to release management.
According to Hotle, automation is not an option when moving from a waterfall to an Agile development process. “There are things that go from should to must, and there are a lot of organizations that either have shelfware or they’re not used to using these products and therefore try to get into an Agile project, and they don’t have the automation to support the very quick delivery that they need,” says Hotle.