"What Makes a Good Test" (PDF) is reprinted by permission of Dorset House Publishing from G.M. Weinberg, Perfect Software, pp. 67-73. Copyright (c) 2008. All rights reserved.
Download the full chapter.
Buy Perfect Software.
"There is nothing either good or bad,
but thinking makes it so."
-- William Shakespeare, English Dramatist and Poet
(1564--1616), Hamlet, Act 2, Scene 2.
What makes a good test? The question can be answered at a highly
technical level, but that's not our goal here. In fact, it's helpful to
step back from that debate and look at the question from a
management point of view. How can you know if testing is being
done well? How much credence can you put in test results?
You can never know for sure.
You may or may not agree with Hamlet that there is nothing either
good or bad. For the sake of argument, though, let's suppose there
is such a thing as a "good" test and ask the question, How can one
know whether a particular test (or set of tests) is, indeed, good?
Let's start with something even better than "good" by
looking at a definition of "perfect." A perfect set of tests would
have the following characteristics:
a. It would detect every bug in a system.
b. It would never detect a non-bug as a bug.
c. It would give us complete confidence that it has done a and b.
d. It would accomplish a, b, and c quickly and cheaply
enough for our needs.
Now consider a system under test. In the simplest case, if that
system were perfectly bug free (a situation likely to exist only in
our dreams), then any test that finds no bugs meets condition a.
Some tests that we would consider really lousy could meet that
condition as well, but when run against a bug-free, perfect system,
they would look pretty good.
Or would they? We don't know in advance whether we're
testing a bug-free system or a lousy system. (If we did, why
would we need to test?) So, imagine two sets of tests: perfect tests
and lousy tests. When run against our bug-free, perfect system,
both sets of tests reveal no bugs. So, on the basis of their bugfinding
alone, we couldn't tell the difference between a perfect test
and a lousy test.
In fact, what might be an adequate test for one implementation
of a system might be a lousy test for another implementation
of the same system. In other words, "goodness" cannot be a property
of a test, but only a property of the relationship between a test
and an implementation.
Going one step further, the same test of the same implementation
might be adequate for one organization but lousy for
another. For example, a test that is adequate for a system to be
used internally by a single organization might be totally inadequate
if the same implementation were sold as a software product
to a thousand organizations. In other words, "goodness" cannot be
a property of tests and implementations, but only a property of the
relationship among tests, implementations, and situations.
So, you can never tell for sure, and you never can tell by
looking at a test in isolation, whether a test is good -- but you do
have many ways to tell whether a test is likely to be bad. Metatests
play an important role. Later, in Chapter 9, we examine some
indicators of "bad" tests.
You can assess goodness only after the fact.
If you knew how many bugs were in a system, you could at least
begin to assess the goodness, or not-badness, of a set of tests. For
instance, after a system has been in use for a few years, a prudent
manager will have statistics on how many bugs were shipped with
the system. By keeping track of what bugs turn up in use, then analyzing them to see what they're like, you will have at least some kinds of information, such as,
- how good your testing was, and in what ways
- how testing might be improved in the future
- what kinds of bugs your testing characteristically missed
Knowing such information allows you to make better estimates in
the future, even if you don't improve your testing process. Such
information may also be used to improve the development
process -- although in this regard, most likely I'm dreaming again.
Download the full chapter for more ways to assess the quality of your tests and a list of common testing mistakes.
