COLUMN

The importance of testing software requirements

By Dr. Joe Marasco 08 Oct 2007 | SearchSoftwareQuality.com Software quality news and advice Digg This!     StumbleUpon     Del.icio.us

Dr. Joe Marasco

Software that performs well is useless if it ultimately fails to meet business user requirements. But while research study after research study shows that requirements errors are the number one cause of software project failures (costing U.S. companies approximately $30 billion a year in total¹), many organizations continue to deliver requirements that are unclear, ambiguous or incomplete. These erroneous requirements then go on to be tracked, traced and perfectly executed, resulting in a system that "does what was required" but unfortunately doesn't do what business users thought they asked for.

The problem is that organizations fail to apply the same rigorous testing processes to requirements development as they do to software development. This is a mistake, as the cost of identifying critical errors during requirements development is exponentially less than errors that are uncovered after coding begins.²

Why is it so difficult to get requirements right?

How often have stakeholders in your organization signed off on use cases they haven't read because there was simply no time to delve into a document the size of an average phone book? What about those marathon meetings where everybody hammers out requirements definitions in one room only to come out even more confused about what's really needed because developers see the world one way and business users another? For too long now, rigid, disjointed and time-consuming processes for requirements development have left little room for effective collaboration between business and IT stakeholders, or to accommodate the inevitable changes that crop up.

The body of best practices associated with software testing is comprehensive -- there's even an ISO standard associated with software evaluation. In contrast, requirements testing is often carried out in an ad-hoc manner, using outdated or impractical methods that leave a lot of room for error and miscommunication. That has to change. Following are some key "best practices" for ensuring that requirements are as complete and accurate as possible, at all stages of development:

Know where your problem is
The requirements development lifecycle consists of several important, yet different processes -- elicitation (gathering requirements), specification and analysis (putting requirements into a formal model or document, such as a use case, and inviting stakeholder feedback) and validation (making sure everyone understands and agrees on the requirements put forth, and that they are realistic and precise). Validation is the step where the majority of requirements errors will be uncovered, but unfortunately, with so much time and energy spent on elicitation and specification, it is also the most commonly overlooked.

To mitigate these costs, requirements should be tested for precision and accuracy both before coding begins and as part of an iterative process that runs parallel with ongoing development.

Furthermore, validation is also frequently misunderstood. For example, a number of organizations confuse the idea of requirements "management" with validation. A recent Forrester report cogently noted that the focus on -- and availability of -- tools that manage requirements during the software development process is detracting attention from the far larger problem of whether or not requirements are accurate in the first place. Simply managing requirements -- no matter how well this is done -- will still result in the age-old problem of "garbage in -- garbage out."³

The same goes for UI prototyping and simulation tools. Also mistaken for methods of requirements validation, these tools help stakeholders "see" the finished user interface, serving up a visual preview of how users will navigate the system. While prototyping addresses the question of "How will this look once we get it done?" true requirements validation answers the question: "What do we want the software to do?"

An iterative process is best
Testing is an ongoing and integral part of the software development life cycle. In contrast, the process of validating requirements is often viewed as a one-time event -- with no one going back to reevaluate the specs during different stages of development.

That is problematic because most development today is iterative -- i.e., based on the idea of building software incrementally. Lessons learned during development are thus integrated into the project during the beginning of every new "iteration," with the goal being an end product that is more stable and resilient. As a result, it is wise to assume that requirements will change as the project moves through development, for at least two reasons: one, the process of doing iterative development allows us to learn about the system as we built it, thereby refining our notion of what the requirements should have been, and, two, the outside world is changing concurrently, imposing adjustments that can not be ignored. Thus, a continuous and cumulative cycle of ongoing requirements validation is critical to maintaining quality.

Bridge the communication gap
For a great majority of organizations, the most fundamental requirements pain point can be summed up in those immortal words from "Cool Hand Luke" -- "What we have here is a failure to communicate." The goal of software development is to solve business problems. But in the chain of project stakeholders, starting with the business person and ending with the programmer, there can be a great deal of variability in how each individual taking part in an initiative views the problem at hand. Ultimately, all these people are trying to solve the same issue, but each of them sees it differently, and each of them describes it in a dialect that they feel comfortable with. Thus, effective communication must be the driving force behind requirements validation -- getting people across the business and IT sides of the fence to agree on a single version of reality.

Use visualization to "see what was said"
Different stakeholders prefer to specify and analyze requirements with different tools. Domain experts and business analysts favor English uses cases, whereas software architects and programmers prefer models and views. One group tends to work in text and the other in diagrams, which is a problem when it comes to validation. Translating the English language use cases into models is often a slow, painstaking process that opens the door for errors to creep in.

As a result, a relatively small number of use cases are actually translated into models, for reasons of cost, convenience and time. That is a shame because the translation of textual use cases into visual models, properly done, is an excellent way to bridge the communication gaps that are inherent in requirements validation.

Automation can play an important role in this respect, serving as a "neutral" and consistent translator of words into visuals. The ability to automatically generate visual models from English use cases can help bring all stakeholders into the process -- from the domain expert on through to the software engineer -- while still allowing them to work in a language that is most comfortable to each.

Visual representations, for example, can help business analysts "see" errors and inconsistencies that are difficult to spot in text form. At the same time, automation relieves software engineers of having to slog through hundreds of pages of textual requirements. Linking text to the visual is key here, allowing words and pictures work with, rather than against, each other.

Respect the business analyst
The business analyst plays a critical role in the requirements validation process, as this is the individual most suited to serve as the middleman between business and IT. Because business analysts are closely in touch with the people and processes that drive an organization's operations, they can help IT more clearly understand where and how technology will most effectively support business objectives. Most important, the business analyst should have insight into how the organization's users tend to interact with technology, and thus be able to steer developers away from applications that may be too complex and towards solutions that will enable and empower end users.

But validating requirements is just a small part of the business analyst's job. It is important to recognize that the business analyst is a highly valuable, yet highly constrained, resource. Accordingly, this individual must be armed with tools that make the validation process less cumbersome, more efficient and easier to track.

Track all communication and feedback
Finally, in an environment where requirements are being changed and reevaluated continually, a formal process for documenting and tracking the various iterations of the validation process is paramount. This is where Web 2.0 technologies like social networking can play a valuable role.

Imagine a "My Space"-style application that brings business stakeholders and developers together, providing a single place where everyone involved in requirements validation can go to review and comment on the specification draft. Now imagine that every comment, change and requirement iteration is tracked and archived in this single repository.

It is easy to see the enormous improvements to accuracy, transparency and efficiency that this can bring to the requirements process, as everybody can easily see the entire history, who had a finger on which part, and where various comments rate in importance on the requirements "food chain." The last thing anyone wants to have to do when the project clock is ticking is rummage around in vain for that critical Post-It that was attached to page 326 of a 500-page document or search through thousands of emails to find the change suggested by a key business stakeholder.

Furthermore, the right collaboration tool can also help project teams maximize the time of constrained resources like subject matter experts (SMEs). For instance, if the vice president of customer service can devote only six hours to a specific IT project, it is important that this contributor is able to provide his input quickly and easily and that other project stakeholders can call up this input just as quickly and easily when they need to refer back to it.

Requirements errors cost organizations a great deal in terms of dollars, resources and time. To mitigate these costs, requirements should be tested for precision and accuracy both before coding begins and as part of an iterative process that runs parallel with ongoing development. Requirements validation is critical to achieving better software quality, but all too often it is a process that gets rushed or overlooked. Taking the time to "get it right" -- by addressing communication issues, involving all stakeholders and applying consistent, reliable best practices for validation -- should not be viewed as an unwelcome delay, but rather as a necessary part of development that will ultimately offer the best chance for improving time-to-market and end results.

-----------------------------------------
About the author: A 35-year industry veteran, Dr. Joe Marasco has been involved in nearly every aspect of creating and marketing software, from product development and sales, to marketing and operations. His recent book, The Software Development Edge: Essays on Managing Successful Projects, published in 2005 by Addison-Wesley, delivers "in the trenches" expertise on managing software projects. Throughout his career, Dr. Marasco has been cognizant of the importance of improving requirements quality early in the development process, and his current leadership role at Ravenflow is focused on revolutionizing the way that software requirements and business processes are developed and validated.

End notes

¹ National Institute of Standards & Technology, U.S. Department of Commerce, The Economic Impacts of Inadequate Infrastructure for Software Testing, 2002
² Dean Leffingwell, Managing Software Requirements, Addison-Wesley, 2003
³ "The Root of the Problem: Poor Requirements" by Carey Schwaber with Gene Leganza and Megan Daniels, Forrester Research, Sept. 1, 2006

Tags: Software requirements techniques (Prototyping, Storyboards, Modeling, State transitions),  VIEW ALL TAGS

Digg This!     StumbleUpon     Del.icio.us