Fixing software defects, particularly those created by changing requirements, drives up the cost of Agile software development projects, according to a new survey by voke inc. Respondents reported software project increases of $3.2 million in 2008 to $3.4 million in 2012. Overall, the survey revealed that Agile is delivering increased customer satisfaction for more than 40% of companies but isn't doing as well on making development less expensive.
Voke surveyed technology leaders of 200 geographically-dispersed, technology and non-technology companies to create two reports: "Market Snapshot Report: Agile Realities" and "Strategic Brief on the Cost of Rework for Agile and Non-Agile Projects." Voke's research on the cost of fixing defects, or rework, in software projects is explored here, as are voke's and Agile users' advice on reducing Agile costs. The survey's revelations on development leaders' misconceptions about Agile, as well as experts' views on correcting them, are covered in a companion piece, The Agile method remains confusing for software professionals.
The increase in project costs is surprising in light of significant drops in software development team size and project duration, said Theresa Lanowitz, voke's founder and a former Gartner Research analyst and Sun and Borland technologist. Average team sizes dropped from 75 people in 2008 to 30 people in 2012 and project duration from 17.2 months in 2008 to 11 months in 2012. Overall, respondents assumed that these cuts would reduce project costs, but they did not fully evaluate the costs of fixing defects and changes in requirements.
"So, 44% of the people surveyed had no idea what their rework was on a per-sprint basis," said Lanowitz. Rework -- defect removal – is always part of the cost for quality in software development.
"Given that 44% of the people do not really understand what their rework is, how can they effectively measure what the cost of the software is that they're actually building?"
One tenet of the cost of quality, a fairly well-known concept, is that the later in the lifecycle a defect is found, the more expensive it is to resolve or remediate the issue, said Lanowitz. Generally, the more software requirements are identified and clarified prior to beginning development, the less quality costs.
Voke's analysis of rework costs in non-Agile and various types of Agile projects shows the cost of rework is fundamentally different in Agile. Agile project consultants that voke surveyed reported needing customers to have flexible budgets because the end date of a project is unknown when using Agile practices, largely because requirements are not set in place initially. "The practice of analyzing and refining requirements in source code is an expensive practice and could lead to significant schedule delays," said voke's Brief.
Organizations surveyed showed little understanding of how the cost of rework manifests differently in Agile projects, said Lisa Dronzek, co-founder of voke. "In effect, Agile embraces the fundamental realignment of business ownership of requirements to developer ownership of requirements through frequent changes in source code," she said. By moving the requirements analysis and discovery process into source code, organizations no longer benefit from inexpensive requirements changes early in the lifecycle.
The fundamental nature of the Agile movement is welcoming changing requirements even late in development, Dronzek said. Forty-eight percent of survey participants reported continuously changing requirements based on customer feedback in Agile projects versus 18% for non-Agile projects. Also, 46% reported continuously changing requirements based on new business ideas for Agile projects, compared to 15% for non-Agile projects. Additionally, 30% reported requirements changing within a sprint. Survey participants said that Agile itself adds to an increased level of requirements changes.
One survey participant noted that Agile has been, but should not be, used as an excuse to promote a lack of discipline in agreement on requirements. Effective requirements are the secret ingredient to success of any software project, Lanowitz said. Organizations should not overemphasize a perceived goal of Agile, or any process, in lieu of leveraging the most effective approaches or tools to identify Agile requirements.
Overall, a scenario for hidden software costs is created when adding Agile development's requirements-as-you-go process with the business side's lack of clarity about how to determine the cost of quality estimates. "If you're participating in an Agile project, you really have to understand and proactively manage and measure your rework cost," said Lanowitz.
The cost of QA timing and developer turnover
Developer turnover is another cost-increasing element not considered very often in quality cost analysis of Agile development, Dronzek and Lanowitz said. How does turnover impact the cost of defects?
"Agile values documentation less while advocating self-organizing teams and a constant pace of development," said Dronzek. "This creates a huge problem for ongoing maintenance when individuals or teams move on to a new project and are unavailable to help support a prior project that has little or no documentation."
In effect, Agile embraces the fundamental realignment of business ownership of requirements to developer ownership of requirements through frequent changes in source code.
Lisa Dronzek, voke co-founder
Organizations need to evaluate their own organization to assess how their budget, project duration, team composition and team size has varied over time. To forestall the impact of developer turnover, organizations need to determine whether a software project is disposable, such as for a campaign or a one-time event, or if the organization must maintain the project over time, Dronzek said.
Another key factor in cost of quality is how early QA is started in a development project. In the voke survey, most enterprises (71%) reported not starting QA testing at a project's beginning, during requirements definition. About twice as many technology companies started QA early. Three-quarters of enterprises and over half of the tech firms did not have QA working concurrently with developers during sprints. Overall, said Lanowitz, doing QA from the get-go keeps development and production maintenance costs low.
Mistaken notions about Agile
The voke survey reveals that Agile isn't being practiced well by most voke survey respondents, Agile consultants said. For example, the majority of respondents don't include business, IT and development in one project team or do testing in early stages.
"Agile does not realign ownership of requirements from business to development," said Nari Kannan, CEO of appsparq Inc., a Louisville, Ky.-based software consulting company. Instead, Agile calls for IT and business to provide an earlier validation of whether development has understood requirements more properly than waterfall development does. "In the earlier Waterfall models, there was no way of doing this. If a business signed off on requirements, that's exactly what they got!"
By allowing business and IT to change requirements during development, Agile reduces the rework costs that come from requirements changes being made after an application has been fully developed, a situation that happens frequently in Waterfall development, Kannan said.
Good Agile teams also recognize the value of doing some but not all requirements work up front. "Good product management has requirements gathering as a continuous process throughout the project lifecycle," said Scott Sehlhorst, product management and strategy consultant for Tyner Blain. "What Agile does do differently, however, is bias the organization towards a process that embraces making changes mid-stream versus one that works to minimize those changes."
Agile coach Lisa Crispin agrees. Her Agile project teams don't gather all-encompassing requirements up front because requirements are almost always changed by business and IT during development. Some requirements are only uncovered by building features and getting feedback, she said. Crispin is a veteran Agile and QA/test practitioner and co-author, with Janet Gregory, of Agile Testing: A Practical Guide for Testers and Agile Teams (Addison-Wesley, 2009).
Sehlhorst added that software use practices and products change rapidly, even as projects are in process. "So some requirements cannot be discovered up front because they don't yet exist." he said.
The best Agile teams use continuous integration, or automated testing, to discover "broken stuff" from the beginning of projects, said Sehlhorst. They encompass QA in short iterations, uncovering problems before they get too big to fix, said Sehlhorst. Getting feedback from customers mid-cycle helps Agile teams uncover "wrong stuff" and "missing stuff" earlier, which reduces the risk (and cost) of going to market with the wrong product.
Using a product manager or stakeholder is a rework-cost-reduction practice voke recommends. It's also one that Crispin's teams employ. Her projects include a product manager, who is charged with defining business and IT requirements. "We get the whole team -- business, IT and development -- together with the product owner and go through all the stories."
Software requirements and Agile
Some organizations are interpreting Agile as a call for throwing out successful development practices, such as using cross-functional project teams (business, IT and development) and defining and testing requirements throughout development. This is misguided and costly, according to the voke survey and Agile experts.
"The importance of laser focus on requirements is what comes out of this survey," Lanowitz said. "Requirements are at the center of every successful or unsuccessful software project, regardless of the style of development."