Tip

Finding software flaws with error-guessing tours

In Rex Stout's series of mystery novels, the great detective Nero Wolfe never leaves his New York brownstone apartment. Instead, he sends his "leg man" Archie Goodwin to investigate crime scenes, perform interviews, tail suspects, and such, and solves the mystery without ever leaving his home. Archie often asks Wolfe for detailed instructions, to which Wolfe often replies: "Use your intelligence guided by your experience."

Intelligence guided by experience is the heart of the test technique known as "error guessing." Let us classify error guessing into two broad categories: local error guessing and error guessing tours.

In general, testers know approaches that are likely to turn up errors in any application: extremely large values; small values; unusual sets of data; unusual combinations of data; flipping back and forth and back and forth between pages; performing multiple actions simultaneously; providing unexpected input at unexpected times; and so on. The general term for such institutional testing approaches is "heuristics." Successful error guessing is often about choosing the correct heuristic for the situation at hand.

Local error guessing

Besides general error guessing approaches, an intelligent, experienced tester will be aware of error guessing approaches to the specific applications he or she tests for a living. For example, if your application has handled Japanese characters poorly in the past, you can

    Requires Free Membership to View

guess that using Swedish characters as input might also cause a problem. If you find an error doing image handling, you can suspect that there are other image-handling errors lurking nearby. In my experience, handling images well is hard; if I find one imaging-handling error, I know I can find more. Again, that's my intelligence guided by my experience.

Sometimes physical clues might cause you to guess that you are on the track of an error. A page might take too long to load, making you guess that if you concentrate on and around that page, you might cause an error.

Note: You might notice the screen flicker too much when you do some action, making you guess that if you can find the right moves, you can turn the flicker into fireworks. I have written elsewhere about having testers read the source code for the application they are testing.

Reading the source code is a great way to guess about possible errors. If you find sloppy code or confusing comments, there is a good chance that there are errors lurking in that part of the application.

Using error guessing with tours

One of the criticisms of error guessing as a test technique is that it lacks discipline, that it is an unreliable shotgun approach to finding issues.

Related Content:
Budget-friendly Web app performance testing, monitoring tips
Instead of skipping software performance testing and monitoring in a budget crunch, use these low-cost methods and tools.

Cut software performance testing costs with built-in measurements
Software testers can cut performance testing costs and improve application performance by using tools to gather metrics during each testing phase.

There has recently been quite a lot of discussion in the testing world about the concept of a "tour." The idea is closely related to the idea of a testing "checklist" and is in opposition to "scripted" testing. One of the most well-known set of testing tours comes from Mike Kelly with his FCC CUTS VIDS tours. Later on James Whitaker had quite a lot to say about tours. Again, reading about testing tours will improve your testing intelligence.

Note:The purpose of a tour is to examine in a particular way and sometimes in a particular order, all of the places in an application where something could go wrong. It is a disciplined way of approaching the testing of a particular application. Designing a good set of tours certainly takes some intelligence, but tours are a lot more valuable when coupled with excellent test experience.

And this is where error guessing comes in. An intelligent, experienced tester will take the opportunity to do error guessing at each stop on the tour. And keep in mind that tours are flexible; side trips in the service of error guessing are encouraged; this is one of the aspects of tours that distinguish them from test scripts, which are by definition inflexible and invariable.

Take it on tour

Notice that this combination of error guessing and test tours cannot be automated. Of all the possible kinds of defects that software may manifest, test automation is capable of finding only a small subset. To find the rest of the software errors requires real human beings.

But real human beings are a wildly inconsistent bunch. Using error guessing in combination with well-designed testing tours is a powerful combination of techniques that maximizes the chance of finding the bugs that only human beings can find. And using this combination of techniques can only increase a tester's intelligence and experience.

So take your error guessing on tour, using your intelligence guided by your experience.

Further study on testing heuristics

There is quite a lot of information about testing heuristics available. If you want to jump-start your ability to do error guessing using test heuristics, print off a copy of Elisabeth Hendrickson's "Test Heuristics Cheat Sheet". The test techniques listed on those pages will certainly increase your intelligence; increasing your experience using them is up to you.


About the author: Chris McMahon is a software tester and former professional bass player. His background in software testing is both deep and wide, having tested systems from mainframes to web apps, from the deepest telecom layers and life-critical software to the frothiest eye candy. Chris has been part of the greater public software testing community since about 2004, both writing about the industry and contributing to open source projects like Watir, Selenium, and FreeBSD. His recent work has been to start the process of prying software development from the cold, dead hands of manufacturing and engineering into the warm light of artistic performance. A dedicated agile telecommuter on distributed teams, Chris lives deep in the remote Four Corners area of the U.S. Luckily, he has email: christopher.mcmahon@gmail.com.

This was first published in January 2010

There are Comments. Add yours.

 
TIP: Want to include a code block in your comment? Use <pre> or <code> tags around the desired text. Ex: <code>insert code</code>

REGISTER or login:

Forgot Password?
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy
Sort by: OldestNewest

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to:

Disclaimer: Our Tips Exchange is a forum for you to share technical advice and expertise with your peers and to learn from other enterprise IT professionals. TechTarget provides the infrastructure to facilitate this sharing of information. However, we cannot guarantee the accuracy or validity of the material submitted. You agree that your use of the Ask The Expert services and your reliance on any questions, answers, information or other materials received through this Web site is at your own risk.