News Stay informed about the latest enterprise technology news and product updates.

Pairwise testing at STPCon

This year’s STPCon had something a little bit different – hands-on sessions where not only did the presenter actually test software — the participants did as well.

The first session of the conference was Justin Hunter’s “Let’s Test Together,” which promised to not only introduce a new test design method, but to change the way we (the audience) think about software testing.

It was an interesting session. While I’m afraid I have neither a time machine nor did I get permission to video-tape the session, I do have the next best thing: Hunter’s Slides plus a summary.

The first thing Hunter did was hand out a series of ‘spools,’ representing the requirements for mortgage-origination software. The software had inputs allowing five types of credit rating, five ranges of for income, six property types, and six different locations in the United States the mortgage could originate in (slide five). Hunter asked us to look at each range (not boundaries), and come up with a number of suggested test cases; a simple 6x5x6x6 yields 1,080 test ideas — and real software of this type would have far more than six inputs.

Next Hunter pointed out the assumption behind the 1,080 test cases was that some magical combination of things were wrong. So he pulled out some historical data from the National Institute of Standards and Technology, pointing out that most bugs are tripped by either a single condition (all people in income range number one) or a combination of two conditions – say all people income range one and credit score three. Thus, based on historical data, if you “just” ran all combination of tests, you would find something like 85% of the bugs in ten or eleven test ideas.

To prove it, Hunter ran a computer program to generate those ten tests, then create a board with all 23 (6+5+6+6) options.  He had an audience member throw darts onto two elements of the grid, and, yes, everything the young gent threw was covered in those ten cases.

We call this solution “All-Pairs”, or pairwise testing.  After explaining the technique, Hunter drew a chart, from high-value to low-value, diagramming the kinds of problems pairwise testing was effective at (limiting configuration or input types), as well as the types if had little or no application for, such as error and exception handling.

It was a neat session and his passion came through.  While Justin Hunter did not invent the pairwise idea, you might say it runs in the family:  William G. Hunter, his father, was a professor of statistics and contributor to the design of experiments movement in the 1980’s.

But wait, there’s more

It turns out that doing the fancy math to generate the table, to figure out those ten test cases, is non-trivial.  And, while you could buy a statistics book with pre-cooked templates that can be adapted, that takes a fair bit of work as well. 

Hunter has software, Hexawise, that participants could use to speed up the process. Instead of doing the manual work, you let the computer generate the tables; you can also ask if the next ten, or the next ten, on and on, of test ideas sorted by highest probability of detection.

Yes, it does get better than this — you could win the lotto.  I’m just afraid the odds on that are a little longer than 1,080 to ten.

But hey, we were in Vegas.

A 14-day trial of Hexawise is available here.

Join the conversation


Send me notifications when other members comment.

Please create a username to comment.

Matthew, It was great to see you at STPCon. I was extremely impressed with how Abbie and the STPCon team organized the event and that you organized the "Rebel Alliance" activities at the conference. Thank you for taking the time to write about my hands on testing techniques session. Two clarifications: Firstly, there's no reason why you should no this of course, but just for the record, we're changing our pricing policies. Previously, we have had a free 14-day trial period. We haven't announced it officially yet, but for a limited time we're going to give away several hundred permanent free licenses away to the first five users at every organization provided users share some feedback with us and/or their colleagues and friends. Secondly, I rushed through the dart throwing exercise so fast that I fear I lost you and others with the specific details. You described it accurately except that there weren't 1,028 possible tests represented by the test inputs on the dart board; there were [ULIST] 72 billion[/ULIST] ! See slide 5 of this presentation (or, better yet, by referring to the sample Google Maps illustrative example in the Hexawise tool itself). Here's what the dart board exercise was meant to demonstrate: With all of the test inputs selected (slide 4 from contains a lot of them), there were 72 billion possible tests (which is clearly far too many to realistically test). So how can a tester select a manageable subset? That is what the Hexawise test case generating tool is designed to do quickly and easily. Hitting the "Create Tests" button in Hexawise results in 35 very powerful tests being created. These 35 tests are selected (in a matter of a couple seconds) from the 72 billion possible tests to not only make sure each test input gets tested once but that each test input is tested together with every other test input in the plan in at least one test case. In other words, every single pair of test inputs gets tested at least one time in only 35 tests (from 72 billion possible tests). The participant threw several pairs of darts at the dart board (with me telling him that, believe it or not, the odds are that the pair of test inputs he selects with his darts will be tested by one of the first TEN Hexawise-generated tests). He threw 2 darts three times to identify three different pairs of test input values. The fist pair of test input values he identified with the darts were covered together within the first Hexawise 10 tests (selected from 72 billion). The next pair of test inputs he identified were tested together in the very first Hexawise test (selected from the same 72 billion). The third pair of test inputs he identified were tested together in test case number 13 (again, out of 72 billion). While it is at first shocking that this could possibly be true, the coverage of all possible pairs of test inputs can be seen on the coverage chart below. Anyone wanting to recreate the dart game exercise at home could do so by selecting any two test inputs they wanted from the Google Maps example in Hexawise (on the Define Inputs screen), then hitting the "Create Tests" icon and searching for where that pair of values gets tested in the Hexawise 2-way plan. There are 4 out of 5 odds that a randomly selected pair of test inputs will be tested within the first 12 test cases (again, from the overwhelming 72 billion tests that would have been required to test all the combinations comprehensively). [IMG src="" alt="Hexawise pairwise testing coverage chart" /] Thanks again. See you around on the testing conference circuit! - Justin
The math is not the hard part, compared with the analysis - first assuring you have all the contributing factors to the tests, and last weeding out the impossible combinations. The first step is making sure you have all the interdependent factors involved in the test. In the example given there are 4. The second step is setting up the grid of tests. In the cases where you have factors that are either on or off, this should be easy for computer people. You just count in binary. With 0 as off and 1 as on, the test cases are 0000 0001 0010 etc. for the 16 (2^4) combinations. The complexity comes if you have values or ranges of values to test. That's where such a tool can be a tremendous help. The last step is to eliminate the impossible cases. Suppose that all factors can never be all off. You have just eliminated the case 0000. Sometimes this analysis is not so easy. I have been using this method, where it can be applied, for decades of testing software, long before pair-wise testing became the buzz word. What I mean by "where it can be applied" means that pair-wise should not be the only tool in your design toolkit. Someimes use cases or flow charts suit the problem better. But pair-waise has been invaluable in achievig feature and integration coverage quickly and easily. Perhaps the simple, "Poor man's" example above can get people started to use the approach.