Software product managers produce requirements documents. Software developers produce code. Software testers? They...
produce data. The role of the test manager is to transform that data into business information. So how do we, as test managers, contribute to sound business decisions with meaningful test reporting?
Identifying relevant business needs
Before we can write a report that addresses a business need or answers a business question, we need to identify what that business need or question is. Once we know the question we are answering, we can provide information that helps the business make that decision. Identifying the business need or question is a two-phase process: recognizing the need and understanding the need.
The first phase is identifying that a need exists at all. Sometimes, someone on the business side (such as a marketer, a project manager, a product manager, or a sales engineer) will come to me and ask for something. This is the easiest way to identify a need-- it's direct and generally explicit. However, sometimes needs aren't as explicit. The need exists, but either the business hasn't yet asked you for it or they don't know that the test organization has information relevant to the need. Identifying these implicit needs is more difficult. Look at the events occurring between now and the next release, including events that are business events, not just technical or product events. For example, the end of a fiscal quarter is a business event. Software release is a business event as well as a technical event. A trade show is also a business event. Even if the test organization is not directly involved, I can provide value.
The second phase is to take the need that we have identified and distill it into an actionable-- and answerable-- question or decision. We know who has the need-- marketing, or sales or upper management-- so we can now go offer our services. Go have a conversation, point out the general need and say, "Can you please give me some examples of what might meet this need?" For example, if there is an upcoming trade show, there is a need for our product and our organization to look better than the competition. "Looking better" isn't something we can do, though; rather, "looking better" is an overall effect. We need to make this far more concrete, and no one has asked us a question. So what are ways in which we might "look better"? This is a good time to go to someone in marketing and ask the question; you'll get lots of ideas: great new feature, better performance, increased data safety, etc. Now you have questions and needs. You have identified your relevant business information: "Is the product faster? If so, how much?", "Is our data safer than it was?", "What can we do with the product now that we couldn't last year?" We won't get all of the examples, but we can probably help with at least one.
Translate from data to information
Once we have a business decision or question, we need to turn our test data into business information. We have to turn, for example, "150000 IOPS/s mean throughput with fast-lane/slow-lane implementation" to "150% faster."
Business information is:
- targeted at a decision, question or need
- understandable to a broad audience
- devoid of irrelevant details or data
Test information is:
- a mountain of data, some relevant, some not
- highly granular and specific
Generally, a single test tells us very little; it's one piece of data. These pieces of data gain value when they are placed in context with all the other data generated by the other tests. Knowing that a bug exists in the SOAP module isn't very informative. Knowing that there are more severe bugs in the SOAP module than any other area, can tell you a lot about the stability of the module.
We have our questions, so sift the data seeking tidbits related to that question. If, for example, the question we're looking at is "Is our performance better than it was last year?” go through your test data looking for anything related to performance. From there, compare it to the same data from last year. Is it faster? Is some of it faster and some of it not? Find the pattern. If there's a pattern or a configuration or a use in which the software is faster than it was a year ago (or than the marketing figures were a year ago), great! Now you can write up the speed increase along with the factors that caused that increase (performance improvement, more parallel configuration, whatever it is), and you have some valuable business information.
The key is to look at your test data for information relating to the question.You're seeking to answer the question given the data before you. Often, the answer will be in combinations of data: trends, patterns, curves, etc. From there, it's simply a matter of writing down the answer in a way that the business person asking the question can understand.
Tips for writing business-oriented reports
When writing a test report for business users, keep the following tips in mind:
- Use marketing terms, not technical names for products and features.
- Identify the question or need in the introduction.
- Someone reading the first half page should be able to get the important parts of the information, so put the conclusion right after the introduction.
- Describe where to get more information (e.g., a wiki, a technical report, etc.)
A test team has a wealth of data, and it exists to provide information for sound business decisions. Every test report is an opportunity to answer a question, to influence a decision, and to fill a need. Identify the need, find the data, and translate it to actionable business information.
Catherine Powell is a principal at Abakas, a software consulting company. Catherine has been testing and managing testers for about ten years. She has worked with a broad range of software, including an enterprise storage system, a web-based healthcare system, data synchronization applications on mobile devices, and web apps of various flavors. She is an author and a formal mentor to testers and test managers. Catherine focuses primarily on the realities of shipping software in small and mid-size companies. Catherine's published works, blog, and contact information can be found at www.abakas.com.