"What Makes a Good Test" (PDF) is reprinted by permission of Dorset House Publishing from G.M. Weinberg, Perfect Software, pp. 67-73. Copyright
Requires Free Membership to View
When you register, you'll receive targeted emails designed to keep you informed of the most relevant information on Agile development, application security, testing & QA, software requirements, and more.
Hannah Smalltree, Editorial Director
(c) 2008. All rights reserved.
"There is nothing either good or bad,
but thinking makes it so."
-- William Shakespeare, English Dramatist and Poet
(1564--1616), Hamlet, Act 2, Scene 2.
What makes a good test? The question can be answered at a highly technical level, but that's not our goal here. In fact, it's helpful to step back from that debate and look at the question from a management point of view. How can you know if testing is being done well? How much credence can you put in test results?
You can never know for sure.
You may or may not agree with Hamlet that there is nothing either good or bad. For the sake of argument, though, let's suppose there is such a thing as a "good" test and ask the question, How can one know whether a particular test (or set of tests) is, indeed, good? Let's start with something even better than "good" by looking at a definition of "perfect." A perfect set of tests would have the following characteristics:
a. It would detect every bug in a system.
b. It would never detect a non-bug as a bug.
c. It would give us complete confidence that it has done a and b.
d. It would accomplish a, b, and c quickly and cheaply
enough for our needs.
Now consider a system under test. In the simplest case, if that system were perfectly bug free (a situation likely to exist only in our dreams), then any test that finds no bugs meets condition a. Some tests that we would consider really lousy could meet that condition as well, but when run against a bug-free, perfect system, they would look pretty good.
Or would they? We don't know in advance whether we're testing a bug-free system or a lousy system. (If we did, why would we need to test?) So, imagine two sets of tests: perfect tests and lousy tests. When run against our bug-free, perfect system, both sets of tests reveal no bugs. So, on the basis of their bugfinding alone, we couldn't tell the difference between a perfect test and a lousy test.
In fact, what might be an adequate test for one implementation of a system might be a lousy test for another implementation of the same system. In other words, "goodness" cannot be a property of a test, but only a property of the relationship between a test and an implementation.
Going one step further, the same test of the same implementation might be adequate for one organization but lousy for another. For example, a test that is adequate for a system to be used internally by a single organization might be totally inadequate if the same implementation were sold as a software product to a thousand organizations. In other words, "goodness" cannot be a property of tests and implementations, but only a property of the relationship among tests, implementations, and situations.
So, you can never tell for sure, and you never can tell by looking at a test in isolation, whether a test is good -- but you do have many ways to tell whether a test is likely to be bad. Metatests play an important role. Later, in Chapter 9, we examine some indicators of "bad" tests.
You can assess goodness only after the fact.
If you knew how many bugs were in a system, you could at least begin to assess the goodness, or not-badness, of a set of tests. For instance, after a system has been in use for a few years, a prudent manager will have statistics on how many bugs were shipped with the system. By keeping track of what bugs turn up in use, then analyzing them to see what they're like, you will have at least some kinds of information, such as,
- how good your testing was, and in what ways
- how testing might be improved in the future
- what kinds of bugs your testing characteristically missed
Knowing such information allows you to make better estimates in the future, even if you don't improve your testing process. Such information may also be used to improve the development process -- although in this regard, most likely I'm dreaming again.
Download the full chapter for more ways to assess the quality of your tests and a list of common testing mistakes.
This was first published in March 2009