Exploratory vs. planned testing: Which yields better results?

Exploratory vs. planned testing: Which yields better results?

Measuring a software testing effort is a difficult thing. There is a lot of literature about the abuse of test metrics, but surprisingly little about test metrics done well. This is a story from early in my career, when I was first learning how to do software testing, from a time when the testing world itself looked very different than it does today. A colleague and I devised an interesting way to measure our work, and we learned something important. That lesson still informs my work today.

Exploratory testing and test planning
Exploratory testing (ET) is a style of software testing that I enjoy very much, and I have become quite good at it. But to someone looking over my shoulder while I work, what I do may seem disorganized or chaotic. What is happening is that I am creating a mental model of the system, and in order to do that, I often jump around in the application in ways that could very well confuse an outside observer. I know from long experience that it is an effective way for me to work, but not everyone enjoys working like this.

My colleague at the time was a very methodical and conscientious tester. She always had well-designed test plans in place, and she would work through well-documented test cases in an orderly fashion until she completed the test plan. She was quite good at her work and we enjoyed working together.

We always had a lot of respect for each other, but at one point we wondered whether one of our work styles was more effective

    Requires Free Membership to View

    Login

    When you register, you'll receive targeted emails designed to keep you informed of the most relevant information on Agile development, application security, testing & QA, software requirements, and more.

    Hannah Smalltree, Editorial Director

    By submitting your registration information to SearchSoftwareQuality.com you agree to receive email communications from TechTarget and TechTarget partners. We encourage you to read our Privacy Policy which contains important disclosures about how we collect and use your registration and other information. If you reside outside of the United States, by submitting this registration information you consent to having your personal data transferred to and processed in the United States. Your use of SearchSoftwareQuality.com is governed by our Terms of Use. You may contact us at webmaster@TechTarget.com.

than the other's. So we came up with a project to find out.

What to measure?
We knew in advance that we were both going to be working on a specific project with very clear requirements to be completed in a particular amount of time. We would get code to be tested dropped in the test environment on a specific day, and the project was required to be complete a significant time later. We agreed that we each would work only in our own style. I would do ET while she carefully and conscientiously executed the test plan. At the end of the project, we would count the number of issues/defects reported, and we would also measure the rate at which each of us reported these issues.

The results were fascinating.

The results come in
Early in the project my issue-reporting rate was significantly higher than hers. I had this wild upward spike of bug reports, finding many issues very quickly, while her issue-reporting rate gradually increased to a certain level and then remained stable.

But in the middle of the project, my issue-reporting rate began to decline. I found all the easy bugs fast, and it was taking longer and longer to find the trickier bugs as the project progressed and the code base matured. Her issue-reporting rate stayed consistent as she methodically worked through well-designed test cases.

Past the halfway point of the project, my issue-reporting rate dipped below hers and eventually fell essentially to zero before the project was complete. Toward the end of the project, her issue-reporting rate slowed gradually and only approached zero very close to the end of the project.

At the end of the project we discovered that we had each reported almost exactly the same number of issues over the length of the entire project. But most of mine had come early in the project while hers had been reported systematically at a regular rate throughout the project. At the end of the exercise, we agreed that neither style of working was significantly better than the other.

What now?
The results of the test metrics exercise allowed us to plan our work much more effectively. I was usually in charge of testing on shorter projects with critical release dates, because I was better at finding more bugs more quickly. For longer more complex projects that required more planning and foresight, she was usually in charge.

I am a much better tester today than I was then, but this project showed me some of my own strengths and weaknesses, and even though this test metrics project happened more than a decade ago, the results still inform my work. I remain a strong exploratory tester, and while my test planning skills are much improved, I still defer such work to people better than I am whenever I have such colleagues.


This was first published in September 2010

Join the conversationComment

Share
Comments

    Results

    Contribute to the conversation

    All fields are required. Comments will appear at the bottom of the article.

    Disclaimer: Our Tips Exchange is a forum for you to share technical advice and expertise with your peers and to learn from other enterprise IT professionals. TechTarget provides the infrastructure to facilitate this sharing of information. However, we cannot guarantee the accuracy or validity of the material submitted. You agree that your use of the Ask The Expert services and your reliance on any questions, answers, information or other materials received through this Web site is at your own risk.

    Back to top