| QUESTION & ANSWER |
Agile methods bring improved software quality, but challenges remain |
 |
By Colleen Frye, News Writer
16 Feb 2007 | SearchSoftwareQuality.com |
|
|
Early benchmarking results of Agile development methods such as Extreme Programming (XP) and Scrum projects show productivity and quality improvements, but time/scope pressures remain the biggest challenges to project success. Software metrics and estimation expert Michael Mah sat down with SearchSoftwareQuality.com to discuss the improvements seen and what factors can still negatively affect a project.
|
|
|
| Michael Mah, managing partner, QSM Associates Inc. |
|
Can Agile development methods such as Extreme Programming (XP) and Scrum improve productivity and software quality?
Michael Mah: That's what the data shows so far, but there aren't a lot of quantitative recordings of this yet. What we're seeing, at least on projects I've witnessed, measured, and done some productivity and quality benchmarking on, is that the Scrum and Agile productivity is higher and the defects are lower.
What factors contribute to these results?
Mah: It is this notion of the discipline. In the early days, [with] the Agile Manifesto espousing less documentation and less focus on process, [it] was perceived as an undisciplined approach. If anything, it's highly disciplined when done correctly.
There is a much more proactive attention to requirements in the form of user stories; there's a very active collaboration between people who are saying what they want and communicating with developers who are building what they're being asked to build; and this notion of instantaneous requirements review, and a design review when have you co-located teams where a client is working directly with developers. And then you have an instantaneous code review when you have, say, a pair of programmers in real time checking each other's work. So in opposition to people believing it was a license to hack, it actually turns out when done correctly to be highly disciplined, so you have fewer mistakes.
How does this compare with traditional development projects?
Mah: In the traditional way a software project might unfold, the deadline is set and people aren't sure what the requirements are. Then the requirements are growing and growing, while the deadline and the budget stay fixed. Teams [are] attempting to build more than they can, and then [are] cutting function back when they realize they're not going to make the date. Now that, if anything, is incredibly undisciplined.
With an Agile process, what are some of the ways in which software quality is improved?
Mah: It's when people are very clearly communicating what they want in this immediate dialog and feedback. It's getting the story work done, so user stories are being defined, which is part of the requirements activity, and then making sure there's active listening and communication in a direct feedback loop even before they're coding.
When they're coding, there's a direct feedback loop as the code is being built. Then perhaps when they're running tests, that's the third opportunity to catch whether or not what's been designed and implemented is correct, so early detection in the way it's going, and not waiting until testing.
Do Agile development and process improvement methods have similar goals around quality and productivity?
Mah: The Carnegie Melon-SEI framework [Capability Maturity Model (CMM)] was all about trying to create a discipline that didn't result in your waiting to test bugs out after the fact. The whole idea about high-level design reviews and code review always existed in good software development practices. Perhaps the SEI in many ways took the step of creating maturity levels to define how well people adhered to those design principles. I think Agile development is another way of implementing design principles. [SEI and Agile] are both trying to achieve the same thing. They're trying to develop higher-quality code faster and at lower cost.
Why are process-improvement advocates and Agile advocates sometimes perceived as at odds?
Mah: I think the marketing of a new approach always takes the CNN Crossfire approach of disparaging what came before them. When the whole Agile movement came it was initially framed as something like a rebellion against heavy processes, and they tagged the SEI maturity framework as a heavy process. You have to be mindful that you can have good outcomes with both approaches and bad outcomes with both approaches. It depends on how you implement the approach.
|
|
|
|
|
[SEI and Agile] are both trying to achieve the same thing. They're trying to develop higher-quality code faster and at lower cost.
Michael Mah
managing partner, QSM Associates Inc.
|
|
|
|
|
|
|
|
|
|
You've done research on XP projects and came up with metrics to determine their success. Can you share some of your findings?
Mah: We're finding defects 20-50% lower. One company we recently benchmarked had half the bugs compared to industry trends, and yet they still achieved a pretty fast schedule, anywhere from 20-40% faster. The cost was about a million less per project per release compared to the average. So in that case they were really firing on all cylinders, but it didn't happen overnight. It took a considerable retooling of the development environment.
They had a whole different way of establishing their teams, a different way of working with clients, a Ping-Pong table in the corner for regular breaks, 40-hour maximum work weeks. The executive who had the vision to implement it said it took years to fully make this transition. There's a cultural shift, but that also is true of process improvement in any discipline. When we looked at productivity, quality and time-to-market improvements on moving from SEI Level 1 to 2 it typically would take 1.5 to 2 years for an organization to move up one level across a division.
What kinds of things impact software quality and whether a project is considered successful?
Mah: [For this company], it varied with some of the nuances, release to release -- how they decided to deal with time-to-market pressures, how they decided to deal with the demand for functionality within a given deadline. Those things ultimately are very large drivers of the outcome from a defect perspective -- how you manage the time/scope pressure. It always comes down to how management responds to that constraint.
In this one organization they had varying degrees of scope/deadline pressure. Not every project will come out to exactly the same behavior as the previous project because things change -- the complexities of what people are building shift, you might have a release that has more complex stories. You also have different mixes of team skills. There might be a natural staff turnover where the makeup of the team in one release may be different from the makeup of the team of the prior release. And you may have learning curve issues.
Have you also benchmarked Scrum projects?
Mah: A large financial services organization implemented the Scrum approach, and we're seeing some of the disciplines in an agile Scrum approach also filtering their way into outcomes of those projects, too. Interestingly, we saw a contrast of large Scrum teams vs. small Scrum teams, and here's where the time/scope and team size dynamic played out.
We saw projects that had very aggressive deadlines with a very ambitious ramp-up of large teams; we'll call that Scrum A. They had a higher number of defects than the Scrum B approach, which did not have as much time pressure. They used a smaller team and still managed to develop within a credible schedule, just not an insane death march. And they had fewer defects. Now both were Scrum, but in one case you had more defects and in the other case you had fewer defects, so it's not just the Scrum factor. It had to do with how management dealt with the larger dynamic pressure, which was the scope/deadline tradeoff. The process is not the silver bullet.
What's the biggest challenge to managing this scope/deadline tradeoff?
Mah: The biggest challenge in that particular organization came back to this: How do we estimate projects regardless of whether they use a Scrum approach or not? Do we make reasonable commitments regarding a deadline for the amount of scope we take on? Do we make reasonable commitments so that we have a high level of quality? To some extent some of the practices in an Agile approach will actually tilt the organization to having a higher chance of making reasonable commitments, but that doesn't always happen.
You can still have, for example, regulatory requirements suddenly being forced upon the financial services industry that say you have to have this reporting by this date, otherwise you're not in compliance. Sometimes those market demands are beyond your control and then you simply have to do what you can. Regardless of whether you use Scrum, XP or SEI Level 3 or 4, you may or may not get good outcomes.
Time pressure is ingrained in our work culture. You've been writing about "optimal friction." Can you explain?
Mah: I'm glad you asked; that term was actually coined by my colleague Sean Callaghan in a discussion we had about the pressures at a client we were working with. Part of what I'm trying to look at is when time pressure hits a sweet spot, which produces enough motivation and a good outcome for a project to hit its target, compared to when it crosses a line where that time/pressure falls out of the optimal zone and winds up being counterproductive and creates destructive chaos. That's part of the thinking behind what I call "optimal friction." There's a way of managing that time pressure so that it serves a useful purpose, but not where it goes beyond a certain point where it then creates pressures on a project, pressures on a team, where things completely break down. Much of my work involves helping software developers understand what optimal friction is and exploring ways to achieve it.
Michael Mah is a managing partner at QSM Associates Inc. (Quantitative Software Management) in Pittsfield, Mass., and a senior consultant with Cutter Consortium. He is a recognized expert on practical applications of software metrics, project estimation/control, and IT productivity benchmarking. His pre-published ideas of the upcoming book, Optimal Friction: People Dynamics at Work in the Information Age, focus on how technology projects behave under pressure, the interpersonal dynamics and the management of internal conflicts.
|
|
|
|
