Adapting over Conforming
A traditional project manager focuses on following the plan with minimal changes, whereas an agile leader focuses on adapting successfully to inevitable changes.
Traditional managers view the plan as the goal, whereas agile leaders view customer value as the goal. If you doubt the former, just look at the deﬁnition of "success" from the Standish Group, who has published success (and failure) rates of software projects over a long period of time. Success, per the Standish Group is "the project is completed on time and on budget, with all the features and functions originally speciﬁed."1 This is not a value-based deﬁnition but a constraint-based one. Using this deﬁnition, then, managers focus on following the plan with minimal changes. Colleague Rob Austin would classify this as a dysfunctional measurement (Austin 1996)—one that leads to the opposite behavior of what was intended.
When customer value and quality are the goals, then a plan becomes a means to achieve those goals, not the goal itself. The constraints embedded in those plans are still important; they still guide the project; we still want to understand variations from the plans, but—and this is a big but—plans are not sacrosanct; they are meant to be ﬂexible; they are meant to be guides, not straightjackets.
1 Standish Group. Chaos Reports (http://www.standishgroup.com/chaos_resources/chronicles.php).
Adaptive Principle Statements
Both traditional and agile leaders plan, and both spend a fair amount of time planning. But they view plans in radically different ways. They both believe in plans as baselines, but traditional managers are constantly trying to "correct" actual results to that baseline. In the PMBOK2, for example, the relevant activity is described as "corrective action" to guide the team back to the plan. In agile project management, we use "adaptive action" to describe which course of action to take (and one of those actions may be to correct to the plan).
The agile principles documents—the Agile Manifesto and the Declaration of Interdependence—contain ﬁve principle statements about adaptation, as shown in Figure 4-1.
These principles could be summarized as follows:
- We expect change (uncertainty) and respond accordingly rather than follow outdated plans.
- We adapt our processes and practices as necessary.
The ability to respond to change drives competitive advantage. Think of the possibilities (not the problems) of being able to release a new version of a product weekly. Think of the competitive advantage of being able to pack
2 The Project Management Institute's Project Management Body of Knowledge, known as the PMBOK.
age features so customers feel they have software speciﬁcally customized for them (and the cost to maintain the software remains low).
Teams must adapt, but they can't lose track of the ultimate goals of the project. Teams should constantly evaluate progress, whether adapting or anticipating, by asking these four questions:
- Is value, in the form of a releasable product, being delivered?
- Is the quality goal of building a reliable, adaptable product being met?
- Is the project progressing satisfactorily within acceptable constraints?
- Is the team adapting effectively to changes imposed by management, customers, or technology?
The dictionary deﬁnes change as: "To cause to be different, to give a completely different form or appearance to." It deﬁnes adapt as: "To make suitable to or ﬁt for a speciﬁc use or situation." Changing and adapting are not the same and the difference between them is important. There is no goal inherent in change—as the quip says, "stuff happens." Adaptation, on the other hand, is directed towards a goal (suitability). Change is mindless; adaptation is mindful.
Adaptation can be considered a mindful response to change.
The Science of Adaptation
Former Visa International CEO Dee Hock (1999) coined the word "chaordic" to describe both the world around us and his approach to managing a far-ﬂung enterprise—balanced on the precipice between chaos and order. Our sense of the world dictates management style. If the world is perceived as static, then production-style management practices will dominate. If the world is perceived as dynamic, however, then exploration-style management practices will come to the fore. Of course, it's not that simple— there is always a blend of static and dynamic, which means that managers must always perform a delicate balancing act.
In the last two decades a vanguard of scientists and managers have articulated a profound shift in their view about how organisms and organizations evolve, respond to change, and manage their growth. Scientists' ﬁndings about the tipping points of chemical reactions and the "swarm" behavior of ants have given organizational researchers insights into what makes successful companies and successful managers. Practitioners have studied how innovative groups work most effectively.
As quantum physics changed our notions of predictability and Darwin changed our perspective on evolution, complex adaptive systems (CAS) theory has reshaped scientiﬁc and management thinking. In an era of rapid change, we need better ways of making sense of the world around us. Just as biologists now study ecosystems as well as species, executives and managers need to understand the global economic and political ecosystems in which their companies compete.
"A complex adaptive system, be it biologic or economic below, is an
ensemble of independent agents
- Who interact to create an ecosystem
- Whose interaction is deﬁned by the exchange of information
- Whose individual actions are based on some system of internal rules
- Who self-organize in nonlinear ways to produce emergent results
- Who exhibit characteristics of both order and chaos
- Who evolve over time" (Highsmith 2000).
For an agile project, the ensemble includes core team members, customers, suppliers, executives, and other participants who interact with each other in various ways. It is these interactions, and the tacit and explicit information exchanges that occur within them, that project management practices need to facilitate.
The individual agent's actions are driven by a set of internal rules—the core ideology and values of APM, for example. Both scientiﬁc and management researchers have shown that a simple set of rules can generate complex behaviors and outcomes, whether in ant colonies or project teams. Complex rules, on the other hand, often become bureaucratic. How these rules are formulated has a signiﬁcant impact on how the complex system operates.
Newtonian approaches predict results. CAS approaches create emergent results. "Emergence is a property of complex adaptive systems that creates some greater property of the whole (system behavior) from the interaction of the parts (self-organizing agent behavior). Emergent results cannot be predicted in the normal sense of cause and effect relationships, but they can be anticipated by creating patterns that have previously produced similar results" (Highsmith 2000). Creativity and innovation are the emergent results of well functioning agile teams.
An adaptive development process has a different character from an optimizing one. Optimizing reﬂects a basic prescriptive Plan-Design-Build lifecycle. Adapting reﬂects an organic, evolutionary Envision-Explore-Adapt lifecycle. An adaptive approach begins not with a single solution, but with multiple potential solutions (experiments). It explores and selects the best by applying a series of ﬁtness tests (actual product features or simulations subjected to acceptance tests) and then adapting to feedback. When uncertainty is low, adaptive approaches run the risk of higher costs. When uncertainty is high, optimizing approaches run the risk of settling too early on a particular solution and stiﬂing innovation. The salient point is that these two fundamental approaches to development are very different, and they require different processes, different management approaches, and different measurements of success.
Newtonian versus quantum, predictability versus ﬂexibility, optimization versus adaptation, efﬁciency versus innovation—all these dichotomies reﬂect a fundamentally different way of making sense about the world and how to manage effectively within it. Because of high iteration costs, the traditional perspective was predictive and change averse, and deterministic processes arose to support this traditional viewpoint. But our viewpoint needs to change. Executives, project leaders, and development teams must embrace a different view of the new product development world, one that not only recognizes change in the business world, but also understands the power of driving down iteration costs to enable experimentation and emergent processes. Understanding these differences and how they affect product development is key to understanding APM.
Agility is the ability to both create and respond to change in order to proﬁt in a turbulent business environment (from Chapter 1). The ability to respond to change is good. The ability to create change for competitors is even better. When you create change you are on the competitive offensive. When you respond to competitors' changes you are on the defensive. When you can respond to change at any point in the development lifecycle, even late, then you have a distinct advantage.
Adaptation needs to exceed the rate of market changes.
But change is hard. Although agile values tell us that responding to change is more important than following a plan, and that embracing rather than resisting change leads to better products, working in a high-change environment can be nerve-wracking for team members. Exploration is difﬁcult; it raises anxiety, trepidation, and sometimes even a little fear. Agile project leaders need to encourage and inspire team members to work through the difﬁculties of a high-change environment. Remaining calm themselves, encouraging experimentation, learning through both successes and mistakes, and helping team members understand the vision are all part of this encouragement. Good leaders create a safe environment in which people can voice outlandish ideas, some of which turn out not to be so outlandish after all. External encouragement and inspiration help teams build internal motivation.
Great explorations ﬂow from inspirational leaders. Cook, Magellan, Shackleton, and Columbus were inspirational leaders with vision. They persevered in the face of monumental obstacles, not the least of which was fear of the unknown. Magellan, after years of dealing with the entrenched Spanish bureaucracy trying to scuttle his plans, launched his ﬁve-ship ﬂeet on October 3, 1519. On September 6, 1522, the Victoria, last of the ships, sailed into port without Magellan, who had died in the Philippines after completing the most treacherous part of the journey. The expedition established a route around Cape Horn and sailed across the vast Paciﬁc Ocean for the ﬁrst time (Joyner 1992).
Great explorers articulate goals that inspire people—goals that get people excited such that they inspire themselves. These goals or visions serve as a unifying focal point of effort, galvanizing people and creating an esprit de corps among the team. Inspirational goals need to be energizing, compelling, clear, and feasible, but just barely. Inspirational goals tap into a team's passion.
Encouraging leaders also know the difference between good goals and bad ones. We all know of egocentric managers who point to some mountain and say, "Let's get up there, team," when everyone else is thinking, "Who is he kidding? There's not a snowball's chance in the hot place that we can carry that off." "Bad BHAGs [Big Hairy Audacious Goals], it turns out, are set with bravado; good BHAGs are set with understanding," says Jim Collins (2001). Inspirational leaders know that setting a vision for the product is a team effort, one based on analysis, understanding, and realistic risk assessment, combined with a sprinkle of adventure.
Innovative product development teams are led, not managed. They allow their leaders to be inspirational. They internalize the leader's encouragement. Great new products, outstanding enhancements to existing products, and creative new business initiatives are driven by passion and inspiration. Project managers who focus on network diagrams, cost budgets, and resource histograms are dooming their teams to mediocrity.3, 4
Leaders help articulate the goals; teams internalize them and motivate themselves. This internal motivation enables exploration. We don't arrive at something new, better, and different without trial and error, launching off in multiple new directions to ﬁnd the one that seems promising. Magellan and his ships spent 38 days covering the 334 miles of the straits that bear his name. In the vast expanse of islands and peninsulas, they explored many dead ends before ﬁnding the correct passages (Kelley 2001).
Magellan's ship Victoria sailed nearly 1,000 miles, back and forth—up estuaries that dead-ended and back out—time and time again. Magellan (his crew, actually) was the ﬁrst to circumnavigate the globe. But Magellan would probably have driven a production-style project manager or executive a little crazy, because he surely didn't follow a plan. But then, any detailed plan would have been foolish—no one even knew whether ships could get around Cape Horn; none had found the way when Magellan launched. No one knew how large the Paciﬁc Ocean was, and even the best guestimates turned out to be thousands of miles short. His vision never changed, but his "execution" changed every day based on new information. Teams need a shared purpose and goal, but they also need encouragement to adapt—to experiment, explore, make mistakes, regroup, and forge ahead again.
Responding to Change
We expect change (uncertainty) and respond accordingly rather than follow outdated plans.
This statement reﬂects the agile viewpoint characterized further by
- Envision-Explore versus Plan-Do
- Adapting versus anticipating
In Artful Making, Harvard Business School professor and colleague Rob Austin and his coauthor Lee Devin (2003) discuss a $125 million IT project disaster in which the company refused to improvise and change from the detailed plan set down prior to the project's start. "'Plan the work and work the plan' was their implicit mantra," they write. "And it led them directly to a costly and destructive course of action.…We'd all like to believe that this kind of problem is rare in business. It's not."
Every project has knowns and unknowns, certainties and uncertainties, and therefore every project has to balance planning and adapting. Balancing is require
d because projects also run the gamut from production-style ones in which uncertainty is low, to exploration-style ones in which uncertainty is high. Exploration-style projects, similar to development of the Sketchbook Pro© product introduced in Chapter 1, require a process that emphasizes envisioning and then exploring into that vision rather than detailed planning and relatively strict execution of tasks. It's not that one is right and the other wrong, but that each style is more or less applicable to a particular project type.
Another factor that impacts project management style is the cost of an iteration; that is, the cost of experimenting. Even if the need for innovation is great, high iteration costs may dictate a process with greater anticipatory work. Low-cost iterations, like those mentioned earlier, enable an adaptive style of development in which plans, architectures, and designs evolve concurrently with the actual product.
Product, Process, People
Faced with major redirection in release 2.0 of their product, the Sketchbook Pro© team introduced in Chapter 1 delivered the revised product in 42 days. As I quip in workshops, "I know teams who would have complained for 42 days with comments such as: "They don't know what they want. They are always changing their minds." Adaptability has three components—product, process, and people. You need to have a gung-ho agile team with the right attitude about change. You need processes and practices that allow the team to adapt to circumstances. And you need high quality code with automated tests. You can have pristine code and a non-agile team and change will be difﬁcult. All three are required to have an agile, adaptable environment.
The barrier to agility in many software organizations is their failure to deal with the technical debt in legacy code. The failure is understandable because the solution can be costly and time consuming. However, failure to address this signiﬁcant barrier keeps many organizations from realizing their agile potential. It took years for legacy code to degenerate; it will take significant time to revitalize the code. It requires a systematic investment in refactoring and automated testing—over several release cycles—to begin to solve the problems of years of neglect.
Barriers or Opportunities.
One of the constant excuses, complaints, or rationalizations about some agile practices are "They would take too much time," or "They would cost too much." This has been said about short iterations, frequent database updates, continuous integration, automated testing, and a host of other agile practices. All too often companies succumb to what colleague Israel Gat calls the "new toy" syndrome—placing all their emphasis on new development and ignoring legacy code. Things like messy old code then become excuses and barriers to change. Some activities certainly are cost-prohibitive, but many of these are artiﬁcial barriers that people voice. Experienced agilists turn these barriers into opportunities. They ask, "What would be the beneﬁt if we could do this?"
Working with a large company—and a very large program team (multiple projects, one integrated product suite) of something over 500 people— several years ago we wanted them to do a complete multi-project code integration at the end of every couple of iterations. The reply was, "We can't do that, it would take multiple people and several weeks of time out of development." This was from a group who had experienced severe problems in prior releases when they integrated products very late in the release cycle. Our response was, "What would the beneﬁt be if you could do the integration quickly and at low cost?" and, "You don't have a choice; if you want to be agile you must integrate across the entire product suite early and often." Grumbling, they managed the ﬁrst integration with signiﬁcant effort, but with far less time than they anticipated. By the time 3–4 integrations had occurred, they had ﬁgured out how to do them in a few days with limited personnel. The beneﬁts from frequent integration were signiﬁcant, eliminating many problems that previously would have lingered, unfound, until close to release date.
Most, but not all, of the time perceived barriers to change (it costs too much) really point out inefﬁciencies—opportunities to streamline the process and enhance the organization's ability to adapt. Agile development demands short-cycle iterations. Doing short-cycle iterations demands ﬁnding ways to do repetitive things quickly and inexpensively. Doing things quickly and inexpensively enables teams to respond to changes in ways they never anticipated previously. Doing things quickly and inexpensively fosters innovation because it encourages teams to experiment. These innovations ripple out into other parts of the organization. Lowering the cost of change enables companies to rethink their business models.
Reliable, Not Repeatable
Note that the word "repeatable" isn't in the agile lexicon. Implementing repeatable processes has been the goal of many companies, but in product development, repeatability turns out to be the wrong goal; in fact, it turns out to be an extremely counterproductive goal. Repeatable means doing the same thing in the same way to product the same results. Reliable means meeting targets regardless of the impediments thrown in your way—it means constantly adapting to meet a goal.
Repeatable processes reduce variability through measurement and constant process correction. The term originated in manufacturing, where results were well deﬁned and repeatability meant that if a process had consistent inputs, then deﬁned outputs would be produced. Repeatable means that the conversion of inputs to outputs can be replicated with little variation. It implies that no new information can be generated during the process because we have to know all the information in advance to predict the output results accurately. Repeatable processes are not effective for product development projects because precise results are rarely predictable, inputs vary considerably from project to project, and the input-to-output conversions themselves are highly variable.
Reliable processes focus on outputs, not inputs. Using a reliable process, team members ﬁgure out ways to consistently achieve a given goal even though the inputs vary dramatically. Because of the input variations, the team may not use the same processes or practices from one project, or even one iteration, to the next. Reliability is results driven. Repeatability is input driven. The irony is that if every project process was somehow made repeatable, the project would be extremely unstable because of input and transformation variations. Even those organizations that purport to have repeatable processes are often successful not because of those processes, but because of the adaptability of the people who are using those processes.
At best, a repeatable process can deliver only what was speciﬁed in the beginning. A reliable, emergent process can actually deliver a better result than anyone ever conceived in the beginning. An emergent process can produce what you wish you had thought about at the start if only you had been smart and prescient enough.
Herein lies a deﬁnitional issue with project scope. With production-style projects, those amenable to repeatable processes, scope is considered to be the deﬁned requirements. But in product development, requirements evolve and change over the life of the project, so "scope" can never be precisely deﬁned in the beginning. Therefore, the correct scope to consider for agile projects isn't deﬁned requirements but the articulated product vision—a releasable product. Product managers may be worried about speciﬁc requirements, but executives are concerned about the product as a whole— Does it meet the vision of the customer? When management asks the ever-popular question, "Did the project meet its scope, schedule, and cost targets?" The answer should be evaluated according to the vision, value, and totality of the capabilities delivered. That is, the evaluation of success can be encapsulated in the question "Do we have a releasable product?" not on whether the set of speciﬁc features deﬁned at the start of the project was produced.
Agile Project Management is both reliable and predictable: It can deliver products that meet the customer's evolving needs within the boundary constraints better than any other process for a given level of uncertainty. Why does this happen? Not because some project manager speciﬁed detailed tasks and micromanaged them, but because an agile leader established an environment in which people wanted to excel and meet their goals.Although APM is reliable, it is not infallible, and it cannot eliminate the vagaries of uncertainty, nor the surprises encountered while exploring. APM can shift the odds toward success. If executives expect projects to deliver on the product vision, within the constraints of speciﬁed time and cost, every time, without fail, then they should be running an assembly line, not developing products.
Reﬂection and Retrospective
Adapting requires both a certain mindset and a set of skills. If we are to be adaptable, then we must be willing to seriously and critically evaluate our performance as individual contributors and as teams. Effective teams cover four key subject areas in their retrospectives: product from both the customer's perspective and a technical quality perspective; process, as in how well the processes and practices being used by the team are working; team, as in how well the group is working as a high-performance team; and project, as in how the project is progressing according to plan. Feedback in each of these areas—at the end of each iteration and at the end of the project—leads to adaptations that improve performance. The "how to" of retrospectives and reﬂection is covered in Chapter 10, "The Adapt and Close Phases." Principles to Practices
We adapt our processes and practices as necessary.
Ultimately, what people do, how they behave, is what creates great products. Principles and practices are guides; they help identify and reinforce certain behaviors. Although principles guide agile teams, speciﬁc practices are necessary to actually accomplish work. A process structure and speciﬁc practices form a minimal, ﬂexible framework for self-organizing teams. In an agile project there must be both anticipatory and adaptive processes and practices. Release planning uses known information to "anticipate" the future. Refactoring uses information found later in the project to "adapt" code. Ron Jeffries once said, "I have more conﬁdence in my ability to adapt than in my ability to plan." Agilists do anticipate, but they always try to understand the limits of anticipation and try to err on the side of less of it.
More Chapter Excerpts on Agile
Developing great products requires exploration, not tracking against a plan. Exploring and adapting are two behavioral traits required to innovate— having the courage to explore into the unknown and having the humility to recognize mistakes and adapt to the situation. Magellan had a vision, a goal, and some general ideas about sailing from Spain down the coast of South America, avoiding Portuguese ships if at all possible, exploring dead end after dead end to ﬁnding a way around Cape Horn, then tracking across the Paciﬁc to once-again known territory in the Southeast Asia archipelagoes. Great new products come from similarly audacious goals and rough plans that often include large gaps in which "miracles happen," much like the miracle of ﬁnding the Straits of Magellan.
Publisher Disclaimer: "This chapter is an excerpt from the new 2nd Ed. of the book, Agile Project Management: Creating Innovative Products, by Jim Highsmith, published July 2009, ISBN 0321658396 by Addison-Wesley Professional as part of its Agile Software Development Series, Copyright 2010 Pearson Education, Inc." (publisher: http://www.informit.com/store/product.aspx?isbn=0321658396) (Safari Books Online: http://safari.informit.com/9780321659200)