Manage Learn to apply best practices and optimize your operations.

Exploratory vs. planned testing: Which yields better results?

Software test expert Chris McMahon tests in an exploratory manner side-by-side a colleague who tests with a methodical, planned manner, comparing the rate of defects found over time. The results may surprise you.

Measuring a software testing effort is a difficult thing. There is a lot of literature about the abuse of test...

metrics, but surprisingly little about test metrics done well. This is a story from early in my career, when I was first learning how to do software testing, from a time when the testing world itself looked very different than it does today. A colleague and I devised an interesting way to measure our work, and we learned something important. That lesson still informs my work today.

Exploratory testing and test planning
Exploratory testing (ET) is a style of software testing that I enjoy very much, and I have become quite good at it. But to someone looking over my shoulder while I work, what I do may seem disorganized or chaotic. What is happening is that I am creating a mental model of the system, and in order to do that, I often jump around in the application in ways that could very well confuse an outside observer. I know from long experience that it is an effective way for me to work, but not everyone enjoys working like this.

My colleague at the time was a very methodical and conscientious tester. She always had well-designed test plans in place, and she would work through well-documented test cases in an orderly fashion until she completed the test plan. She was quite good at her work and we enjoyed working together.

We always had a lot of respect for each other, but at one point we wondered whether one of our work styles was more effective than the other's. So we came up with a project to find out.

What to measure?
We knew in advance that we were both going to be working on a specific project with very clear requirements to be completed in a particular amount of time. We would get code to be tested dropped in the test environment on a specific day, and the project was required to be complete a significant time later. We agreed that we each would work only in our own style. I would do ET while she carefully and conscientiously executed the test plan. At the end of the project, we would count the number of issues/defects reported, and we would also measure the rate at which each of us reported these issues.

The results were fascinating.

The results come in
Early in the project my issue-reporting rate was significantly higher than hers. I had this wild upward spike of bug reports, finding many issues very quickly, while her issue-reporting rate gradually increased to a certain level and then remained stable.

But in the middle of the project, my issue-reporting rate began to decline. I found all the easy bugs fast, and it was taking longer and longer to find the trickier bugs as the project progressed and the code base matured. Her issue-reporting rate stayed consistent as she methodically worked through well-designed test cases.

Past the halfway point of the project, my issue-reporting rate dipped below hers and eventually fell essentially to zero before the project was complete. Toward the end of the project, her issue-reporting rate slowed gradually and only approached zero very close to the end of the project.

At the end of the project we discovered that we had each reported almost exactly the same number of issues over the length of the entire project. But most of mine had come early in the project while hers had been reported systematically at a regular rate throughout the project. At the end of the exercise, we agreed that neither style of working was significantly better than the other.

What now?
The results of the test metrics exercise allowed us to plan our work much more effectively. I was usually in charge of testing on shorter projects with critical release dates, because I was better at finding more bugs more quickly. For longer more complex projects that required more planning and foresight, she was usually in charge.

I am a much better tester today than I was then, but this project showed me some of my own strengths and weaknesses, and even though this test metrics project happened more than a decade ago, the results still inform my work. I remain a strong exploratory tester, and while my test planning skills are much improved, I still defer such work to people better than I am whenever I have such colleagues.

This was last published in September 2010

Dig Deeper on Exploratory Software Testing

Start the conversation

Send me notifications when other members comment.

By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy

Please create a username to comment.