Why divide software development up into iterations? And if an iteration is supposed to result in an increment, what do you want in the increment? The answer is likely to be quite different depending on which stakeholder you ask. Someone in a business-facing role may answer "everything," where a more development-centered individual may answer "the technically interesting bit."
Iterations vs. increments
When discussing iterative and incremental development, the terms iteration and increment are often used freely and interchangeably. They are not, however, synonyms. Iteration refers to the cyclic nature of a process in which activities are repeated in a structured manner. And increment refers to the quantifiable outcome of each iteration.
Iterations can offer a development process two key things: iterative refinement, where the process improves what already exists and is being done, and incremental development, where the process results in progress against project objectives.
Additionally, the term increment has the obvious implication that there should be more of something at the end of an iteration than there was at the start. Without a clear notion of an increment, iterations are likely to just go in circles — "Oh no, not again," to quote the unfortunate bowl of petunias in Douglas Adams' Hitchhiker's Guide to the Galaxy.
Planning vs. plans
In his 1990 article, "Life-Cycle Approaches," Edward Berard noted: "Walking on water and developing software from a specification are easy if both are frozen." And this is only part of the picture: Software development is buffeted by many kinds of change, not just those that affect specification. That does not mean, however, that all is lost and that there is no sense in trying to envision, estimate and plan. What it does call for is a clear-headed appreciation of the difference between a plan and the act of planning.
The centerpiece of managing a sequential development process, such as the waterfall lifecycle, is the plan. Such lifecycles are founded on the premise that we are able to predict how complex and nondeterministic systems will interact and grow over long periods of time.
Some things are easy to predict with reasonable confidence: the time taken for an apple to fall from a tree to the ground, the path of the stars across the night sky, the typical time taken for a professional athlete to run a given distance, the excitement of my 5-year-old son first thing in the morning on Christmas day. However, when we are dealing with systems as complex as markets, businesses, development teams and software, reasoned prediction often crosses the line into surrealist fiction. Not only are these systems individually nontrivial, but what we are trying to predict is how they interact and influence one another.
To avoid the pejorative connotations that waterfall development has acquired, and to create a more positive branding, the term plan-driven development is being used increasingly to refer to sequential development processes. This term is intended to counterpoint agile development with the implication that agile projects do not operate according to plan. However, the role that plans play in agile development is as an updateable snapshot of a more continuous approach to planning. Agile development is indeed not plan-driven development; it is planning-driven development. And just to be clear: Continuous planning is not the same as continuous monitoring of conformance to a plan.
The purpose of a plan is to offer not so much a roadmap as a route. To be sure that the best route is the one we're on — or to know it's one we're not on — we need timely feedback. Iterations need to include activities that make such feedback explicit and bring planning into the present rather than the past tense: a kick-off meeting to establish and clarify what needs to be done and what can be done in an iteration; daily stand-up meetings (15 minutes and you're done); available and accessible information on the current state of system integration; a demo at the end of an iteration to make progress (or lack of progress) visible and concrete; a heartbeat retrospective at the end of the iteration to reflect on what worked in the iteration and what needs attention in the next.
Iterative refinement is about feedback-based improvement. Among other things, developing a software system is an act of learning, which means that we have a better idea of what we should have built when we're finished than when we started. Unless we want to be stuck with the decisions we made at the point of greatest ignorance — the beginning of a project — we need to refine as we go. Such refinement should be continuous and it should apply to both the code and the process.
Refactoring offers a stable vehicle for modifying and improving the quality of code while preserving its functional behavior. Defects (yes, let's give bugs their proper name) should be fixed as we go, not in a phase squeezed in as an afterthought just before delivery. Such a phase is not simply late; it divorces the act of correction from the act of creation and is then — to add insult to injury — mislabeled testing.
Iterative refinement is also applicable to the process itself. Holding a retrospective at the end of a project can bring closure, but why wait until the end to find out what works and what doesn't? The end of an iteration offers a useful point to check out how things are going.
Along with running a demo attended by stakeholders, a heartbeat retrospective makes the end of an iteration a real event, as opposed to a virtual and easily ignored event in an online calendar. But eliciting feedback is only half of the equation. To be of value, the feedback has to be acted on. This is not necessarily easy, but it is necessary. Without responding to feedback with action, what role does feedback play?
One interpretation of the term iterative development is that it is nothing more than iterative refinement in the large, in other words "keep developing until you get it right." Nowadays this interpretation is much less common than it once was. Iterations are assumed to be associated with increments whose content is chosen based on business value or technical risk.
Unlike material products, software has no physical extent by which you can measure progress — the number of lines of code is neither a measure of completion nor a measure of quality (Stalin's observation that "quantity has its own quality" aside). That software has more degrees of freedom in design is both a boon and a burden. In principle anything is possible, but we can no longer rely on our physical senses for estimation and tracking.
It is in this capacity that increments make the otherwise invisible act of software development visible. This means that instead of measuring progress in terms of technical artifacts, such as lines of code or number of classes, progress is best demonstrated through completed functionality. Completed use cases offer one useful measure of progress, tests offer support for this view, and demonstrations to stakeholders offer a reality check.
Visible progress is a key benefit of development structured in terms of increments of actual functionality. However, invisible regress can be an issue if quality is not also perceived and treated as an asset. Indeed, these days the problem is not that some practitioners assume that iterative development is only about iterative refinement, but that they many assume it is only about incremental development. Without refinement the development process ceases to be an adaptive and learning process. The pipeline of increments starts to sputter and falter through the accumulation of technical debt — defect drag, code rot, architectural decay, etc.
About the author: Kevlin Henney is an independent consultant and trainer based in the UK. His work focuses on software architecture, patterns, development process and programming languages. He is a coauthor of A Pattern Language for Distributed Computing and On Patterns and Pattern Languages, two recent volumes in the Pattern-Oriented Software Architecture series. You may contact him at firstname.lastname@example.org.