You've been here before: there's a need for a testing cycle, including regression testing, and the schedule doesn't allow for the coverage you'd like to achieve, or that some Very Important Persons have mandated. You know there are going to be tradeoffs in the breadth and depth of coverage you can achieve. How will you make them? How will you manage and communicate those tradeoffs? This tutorial will describe a risk-focused approach for planning, managing, and communicating breadth and depth of coverage in the face of a tight schedule and constrained resources.
Table of Contents
The app testing scenario we know too well
Planning application risk reduction
Tight deadline app testing approaches
Communicating coverage breadth and depth to management
The app testing scenario we know too well
On a recent project, I was working with a short schedule to regression test a Web-based application. The company was simultaneously upgrading to a new full version of browser, new full version of desktop OS, and new full version of a third application that was a mission-critical point of integration with the Web-based application. Given the scope of the changes there was an argument to be made for "full regression," but the schedule only allowed for two weeks of testing (plus about two weeks prior to that for organizing a team and creating test scripts).
Another important element of this project was that I wouldn't be doing any testing myself. The business was to provide people who used the system to perform the testing. On one hand, this has the great advantage of meaning that people intimately familiar with their area of the program were going to have their eyes on it. They would quickly detect even subtle changes from the norm.
On the other hand, these were people who had full-time responsibilities during the day. They would only be able to contribute quarter-time availability at best, probably much less. Any crisis in their specific duties could pull them away completely.
The challenge was to come up with a strategy that would ensure that the business testers could identify the most important things to test first, going deeper if time permitted. At the same time we'd need to avoid getting silo’d on depth of coverage in specific areas and losing needed breadth of coverage. Whatever approach we used needed to be as efficient and effective as possible, since we really didn't know what resource availability would be like and how much coverage we'd actually achieve. The approach would need to degrade gracefully in the face of these kinds of changes. Finally, we'd need to be able to communicate breadth and depth of coverage as time was running out to help the business make the go/no-go decision to deploy.
Professional testers often manage this kind of tradeoff between risk and breadth and depth of coverage almost unconsciously, but how do you guide non-testers to this same behavior? How do you maintain visibility into their progress and coverage so you can help re-direct when needed?
Planning application risk reduction approaches
The first priority was to have a discussion with the project lead, development team and other key stakeholders to talk about risk. The goal was to understand where the big-picture technical and project risks were. We determined that we could safely eliminate "full regression" as the requirement based on those discussions, and identified key functional areas to focus on. Users from the business were identified and selected based on their expertise in the target areas.
The next task was to figure out how to communicate what we wanted to achieve. I was concerned that this could prove challenging. In my experience, non-testers generally fall into two camps when asked to test software: the "Test Everything!" camp (remembers every error and edge case scenario since the dawn of time and wants to retest them all) and the "I Logged In and It Worked" camp (who can't imagine a test more complicated than a single happiest-of-happy-paths test). Both of these mindsets would be ill-suited to the project.
To preempt this, I felt I needed to be able to give the business a framework to focus thinking around risk and coverage. To leverage their strength as business experts, I turned to a generic risk list and asked the testers to think about their tasks in terms of:
- Things they did very frequently
- Things that were mission-critical, even if they didn't do them frequently
- Things that could cost the company money
- Things that could get the company sued
- Things I couldn't live without
- Things that have been error-prone or buggy in the past
- Things with precise calculations
This was a good list in the context of the project, inspired and adapted from pieces of James Bach's "Heuristic Risk-Based Testing" and Scott Barber's FIBLOTS heuristic. You probably have your own go-to risks for your context.
Tight deadline app testing approaches
Next, pretend your boss walks up to you and tells you that a code change is being made; but you've got only X amount of time to test and give a reason to not deploy, or it's being put into production. What do you test? Here is the full breakdown of approaches I’d take when testing in this scenario, ranging from critical to good:
If you could only test one thing around area, feature, or function X to try to prove it was okay to use in production, what would that be? Come up with a workflow/scenario or small series of workflows/scenarios (<= 15 minutes total to run) that steps through a high-frequency-use and/or critical feature or process.
If you only had one hour to test area, feature, or function X to try to prove it was okay to use in production, what would that be? Come up with a series of tests that take about an hour to execute that represent important real-world scenarios from that area of the program. The goal here is to try to exercise common and/or critical tasks that the Critical/One Thing test was too brief to include.
If you only had four hours to test area, feature, or function X to try to prove it was okay to use in production, what would that be? Come up with a series of tests that take about four to execute that represent important real-world scenarios from that area of the program. The goal here is to try to exercise the balance of common tasks that the Important/One Hour test was too brief to include, and perhaps add interesting data combinations or edge scenarios if time permits.
I sized the minutes/hours associated with each level of testing to something that was appropriate for the amount of time I thought the testers would have available for this project; again, it was only two weeks with part-time resources. The time buckets are small but contextually appropriate for our particular time-constrained situation. For a larger project, I’d be apt to drop the time constraints associated with the three levels.
Communicating coverage breadth and depth
By having the testers keep their tests and scenarios granular and labeled with one of the three importance designations above, it would be easy to understand what kind of risk-based depth of coverage the business had in mind. With our selection of representatives from different areas of the business, we had already made sure that we were set up for the breadth of coverage we wanted. The next thing we needed was a way to understand how that breadth and depth mapped to availability, time remaining, and the risks we were addressing.
To help visualize this, conceptualize the entire program as a box (Figure 1).
Next, visualize each of the features/components/operational areas as vertical slices of the program (Figure 2). (Operational areas made sense for our project, but any classification approach that achieves the level of granularity you want is fine.)
Next, visualize the Critical/Important/Good tests as horizontal slices of the program (Figure 3). To keep with the connotation of "depth of coverage" the riskiest portions of each vertical slice can be conceptualized at the top of the vertical slices. Since the approach for this project also attached time as a factor in determining test importance, the horizontal slices for our diagram get larger as you move down from the top. The absence of any label for the implied bottom horizontal slice also helps underscore that not everything is being tested.
Marking each box with the number of tests planned in each vertical slice starts to draw a picture of the depth of coverage in each section (Figure 4).
Now a couple of things start to come together. You've got a quick way to assess coverage across the different features and compare it to your own assessment of risk and coverage, or compare between vertical slices for inconsistencies. Based on the observer’s knowledge about the vertical slices, questions that this chart might elicit include:
- Vertical C seems more risky to me than Vertical E, why does E have so much more coverage?
- Shouldn't C have more coverage?
- Why does Vertical D have so few Critical tests?
- How is it that Vertical B is almost all critical tests? Is there more risk there than we thought? Does the tester need additional coaching on the meaning of the different levels?
If time or resources are against you as testing progresses, it's easier to understand what kind of depth and breadth is being achieved and where tradeoffs might make sense.
- Do we feel okay if we only achieve Critical coverage for Vertical A? Should we look for additional resources to try to get some Important coverage too?
- Should we take time away from Vertical D, giving up "Good" and maybe some "Important" coverage to get more Critical coverage in one of the verticals that's running behind?
The chart doesn't answer these questions, but it puts you in a position to identify patterns and know to ask the questions.
Running tests in order of criticality helps ensure you get the most important things tested earliest in the cycle. Being able to see your coverage makes it easier to have discussions about risk and options when time or resources are in short supply. For our project, we were fortunate enough to be able to execute all three levels of testing across each operational area. We still got value from knowing which tests were blocked down the stretch (a few Critical ones, a few Important ones) to be able to speak intelligently about the risks still on the table and the coverage already achieved.
I enjoyed the way this approach let me talk about the test strategy, assess coverage goals, monitor test script creation and execution, and communicate with testers and stakeholders. I look forward to using the approach in the future to see how well it scales to larger projects.
About the Author
Rick Grey is an Associate Partner for software testing and test consulting at DeveloperTown, a venture development firm. He’s a seven year software testing veteran with eleven years of experience overall in software. He's a subscriber to the principles of the context-driven school of testing. You can reach him at firstname.lastname@example.org.
On communicating with the box drawing
The figures used for this article were created in Word with tables, and I think they’re (hopefully) good enough to get the concept across. During the project I tried creating the visual representation of this approach with Word and Excel. Word was cleaner, while Excel offered more ability to automate some special effects around coverage achieved.
But I found the most effective way to share this information with others was with a white board. Seeing the diagram take shape, seeing areas get shaded or marked to represent risk, and getting a more interactive experience really seemed to be the best way to help others understand the concept. In the future, I hope to be able to reserve a white board as a permanent home for the diagram, updating it as the project unfolds.