Thirty-five years ago Fred Brooks wrote the software engineering classic Mythical ManMonth. Fifteen years ago he...
added four chapters, creating the Silver Anniversary Edition, including the incredibly popular essay "No Silver Bullets: Accident and Essence in Software Engineering ", originally published in 1986. For the silver anniversary he also published the reply "No Silver Bullets: Refired." Perhaps it's time to reconsider. Have we indeed found the silver bullet no one thought possible?
I think it's fair to say that any technology book older than most of its readers, yet still in print is likely a classic of the literature. Brook's main argument was that development consists of a large number of activities, such as requirements, high level design, low level design, coding, and so on. Each of these activities, including writing code, only takes up a small percentage of the overall total.
This means that any 'silver bullet', like object oriented development or test automation, that promises hundreds of percent of performance improvement could, at best, have a marginal impact on the project's total cost. For example, if writing code takes 25% of the time, and you totally eliminate it through Object Oriented reuse magic don't laugh, people were talking about that with third-generation event-driven programming in 1986, then the best improvement you could hope for would be a 25% overall reduction in cost.
Brooks was arguing that these promises "silver bullets", like SQL or Visual Basic (or the rage at the time, Computer Aided Software Engineering), could eliminate some of the accidental work, but the overall essence would still be pretty expensive. That as an industry we were seeking "silver bullets" to slay the mythic beast of spiraling development cost, but we should really be looking for a series of small, bronze bullets that might add up to, say, a 50% reduction in development costs.
It's been another fifteen years since "No Silver Bullet: Refired", and I believe we've learned a thing or two.
Forget accident; it's time to change the essence
Notice that I'm not rejecting Brooks' premise, that to have a silver bullet, we'd need to change the essence of the work itself. Nor am I proposing some pie in the sky idea like transformational specification, where we express our requirements in English and working code magically pops out. It's worth noting that some people have tried that; SQL was initially SeQL, Structured English Query Language, which was supposed to be done by an analyst or manager with no need for a programmer. Likewise, Business Intelligence software was supposed to make report writing so simple that end users would service themselves. While we have had some strides in that direction, I wouldn't say it qualifies as a Silver Bullet, at least not yet.
No, I am suggesting we completely change the way we look at the software delivery process.
Consider the typical big project of the 1970's, say IBM's OS/360. Sure, you had requirements work, creating the specification, high level design, low level design, code, unit test, system test, and deploy. But take a minute and reimage that waterfall. What were the people at each step actually doing?
I think they were translating.
The requirements analyst translated a vague idea into something on paper. The functional spec translated the needs of the customer ('why') into a promise of features ('what'). The high level design took the 'what' and turned it into a high level 'how'; the low level design translated that into pseudocode. Regular code was translated from pseudocode; unit tests intersected the code and the high level design, and system tests translated the needs of the customer into the language of testing.
Now keep thinking about that. The software wasn't just designed and built once, it was designed and build six or seven different times, each in a different language.
What if all those activities weren't really value-add? What if it were possible to skip three-fourths of them without negative impact?
Dare I say that, if we could skip three-fourths of those documentation and translation activities, then, well, those activities were essentially waste?
Suddenly we've got a new Silver Bullet: eliminate redundant translation from the software development process. More about how to do that in a moment.
But second, my other proposal. Consider the lot of the poor customer, who has to go to the effort of making a spec or requirements document. At the very beginning of the process, when they know the least, we ask the customers to make this list of what the software will do. Also, they generally do it without any idea of what the features will cost, or ability to trade off; we plan to write the specification first, estimate later.
This stands in direct contrast to how a craftsman or artisan works. The master craftsman will ask a bunch of questions, create a quick prototype, ask "is that what you want?" then use that feedback to create the next prototype, repeating the process until he's got something useful.
Can we weave these two ideas together to create a silver bullet? I think so.
Agile, lean and craftsmanship
Consider a typical Extreme Programming team. Instead of having one group create a specification, and then the other group read it, they create lightweight specifications collaboratively, through an exploration process. The team produces no formal design documents of any kind; tests are either executable code, examples that help extend the spec, or exploratory tests, which are extremely light and cheap to produce. There is no design documentation and nothing to keep "up to date" except a "big visible chart" or two, and those can be generated from a spreadsheet once a week with a few clicks.
Now, compared to the series of translation activities I outlined above, Extreme Programming is positively lightweight; the developers produce code and tests. Yes, the team might produce documents and charts, but those are done because the team itself finds the documents valuable, not to satisfy some external entity. And yes, management can require user manuals and guides and so forth, and the team can create them, but the cost is explicit.
Finally, the team has a method of fighting through the "what are we going to build?" analysis paralysis by actually talking to the customer, developing mockups, proof of concept applications, and gathering feedback. Instead of trying to build the entire application, the team will try to get something, a real feature, maybe two or three, to done, actually done-done, within a week or two. They might not ship that feature, but the software will be good enough to show to the customer.
And when the team does show the software to the customer and he smacks his forehead and says "This is exactly what I asked for, but not what I need. Now I know what I need!" Well, it only took two weeks to get to that place, not two years.
I'm sure many of you are familiar with these ideas, but please, take a moment to consider how this fundamentally alters the essence of software development. It may not actually slay the beast of software, but in terms of meeting Dr. Brook's definition, it's is amazingly close.
Introducing lean software development
While some software teams recognize (and are even doing!) Agile Software Development, I still see some confused looks and doubt about Lean software, so let's start there. If I wanted to boil down Lean to one thing, it would be 'drive out waste.' Of course, I've yet to meet anyone who was actually for waste, so let me be more specific.
Here's the magic of lean software development: The team gets to define what waste is. That means that if the team doesn't find value in the Vice President's all hands meeting, well, then, it's waste. If the team finds that no one ever refers to certain documentation ever, well, then, the documentation is waste. (And, yes, if the team believes that coffee breaks are valuable for discussion, then they are not waste.) While Lean thinking encourages standards, the only standards that can be 'codified' are those that the team decides are valuable. Team decisions on waste are made by consensus, not management decision.
Of course, it is always possible that someone else insists on certain policies and practices for accountability or oversight, but Lean Thinking still looks at these as waste, asking "How can we comply with this requirement with the minimum effort possible?
It's been my experience that managers and business people tend to "get" and intuitively support "Lean" efforts, where they might oppose or be scared of words like "Agile" and "Scrum." The good news is this means you can get started on a Lean initiative tomorrow. Just talk to the boss about it, schedule a brownbag with the team, and send them a link to this article, and ask "what wasteful or redundant processes do we have that we can streamline or eliminate?" Bring a pen and paper; you might just need several sheets.
It's hard to reduce the software craftsmanship movement to a few paragraphs; Pete McBreen published an entire book on it . Let's start by looking at software as a wicked problem, which is a problem that can't even be understood until you make an initial, wrong attempt at a solution. That means the process of creating software is one of quick cycles and feedback. That building software is more about defining what is possible than "gathering" "requirements."
Software Craft doesn't start with gathering requirements; it starts with getting the key players in the project (yes, including the testers) to sit down and discuss what to build. The team could use a formal story card process, like in Extreme Programming, or you could just spend a day at a whiteboard, designing and refining prototypes. While craft in development means a specific set of disciplines, in test it means focusing on what test ideas to run, and what they tell us, instead of trying to enshrine test strategy in a set of "Test Cases" or a "Test Plan." Two places to go to study test methods are James Bach's Rapid Software Testing Course (with free materials online) or the freely available Black Box Software Testing (BBST) materials available from Florida Tech. If you'd like more formal study from home, the Association For Software Testing regularly offers a home study BBST course free to its members.
All of these techniques, from prototyping to rapid software testing, are designed to keep things moving instead of "codifying" the current thinking into a document. When the thinking changes, the documents either need to be updated, which is expensive, or become out of date, which is misleading. The answer isn't to reject all documentation, nor is it to document everything; instead, it's to find out how much documentation is "just enough", and how much is waste.
The three-legged stool
Hopefully you've agreed that eliminating the translation work of the waterfall, eliminating waste, and kick starting the development and test process with prototypes can change the essence of the software work. The next thing I typically hear is "...but Matt, my boss will never go for that!"
Well, I have two suggestions for you. First, yes, schedule the brown bag and talk about lean. Start talking about testing as a craft activity, and that a joint application development team might be able to get the requirements down more quickly than traditional methods. Find a challenged project that is approaching crisis, and suggest that the organization use it as a pilot for a project that is actually Agile . Short of regulatory or legal concerns, technical teams simply do not need permission to do the best job they possibly can, so my advice for all of these things is 'just do it.'
Of course, it's possible the boss says 'No', that the business insists on its documentation, or that some ninny in "process improvement" insists that the customer create a "complete, consistent, correct" specification before those dirty technical people are allowed to see it.
And when the project is late, or doesn't actually meet customer needs, or is over budget, and the demand comes from on high to explain the problem, you've got a readymade answer: You offered a silver bullet to the management team, and they chose not to pursue it. Now, politely and respectfully ask: Can we try it this time?
About the author: Matt Heusser is a technical staff member of SocialText, which he joined in 2008 after 11 years of developing, testing and/or managing software projects. He teaches informaton systems courses at Calvin College and is the original lead organizer of the Great Lakes Software Excellence Conference, now in it's fourth year. He writes about the dynamics of testing and development on his blog,Creative Chaos.