Q&A: Exploratory testing, regression testing and automation

Find out Elisabeth Hendrickson's thoughts on exploratory testing and take a look at how it compares to regression testing, both manual and automated, in Agile environments.

Where does exploratory testing fit in an Agile environment and how do you balance that with manual regression testing?...

Elisabeth Hendrickson, an active speaker and leader in the Agile community, will be presenting “Exploratory Testing in an Agile Context” at the Agile 2011 conference in August. In this interview, I got a chance to talk with Elizabeth about some of her thoughts about exploratory testing and get a sneak peek into her presentation.

Lisa Crispin: I’m excited to see your Agile 2011 session. There are lots of places to learn about exploratory testing, but I think it’s important for Agile teams to understand why they need to do this type of testing and where it fits on an Agile project. Would you please share a couple of your observations about the intersection of exploratory testing and Agile software development.

Elisabeth Hendrickson: Great question! I think it’s important too. Exploratory testing is how we discover risks and vulnerabilities that we didn’t even think about when we were fleshing out the expectations for a given feature or story. Since we can’t think of everything in advance, I don’t consider a story done until it’s been implemented, checked, and explored.

There are two big questions I tend to hear over and over again about exploratory testing in Agile: “When should we start exploring?” and “What’s the right balance between exploring and manual regression testing?”

My answer to the first question, when do we start, is “now.” When I’m on a team, I start exploring immediately. It doesn’t matter if a feature isn’t done. If there is code checked into source control then there is something I can explore. I think Ward Cunningham said it best:

Agile programs are more subject to unintended consequences of choices simply because choices happen so much faster. This is where exploratory testing saves the day. Because the program always runs, it is always ready to be explored.

Note that one of the key reasons the program always runs is that high functioning Agile teams automate so many tests. If a change results in something breaking, the team finds out right away.

That relates to the second question: “What’s the right balance between exploratory testing and manual regression testing?”

Usually teams that ask me about how to balance exploring with manual regression testing are stuck in a situation where they have very little automation at a system or business-facing level. Because the regression burden is already so high, they see exploratory testing as just one more thing they have to do added on top of a very long to-do list.

So when teams ask me about balancing the two, I think they’re asking the wrong question. Instead of asking how to split their time between manual regression and exploration, I would rather have them ask themselves why their regression testing is manual and address that problem.

In fact, this relates to your first book, Testing Extreme Programming. I remember giving you a lot of grief for Chapter 19, the infamous test automation chapter when we first met. When I read that chapter, I disagreed strongly. I had been burned by the high cost of test automation so many times. I just didn’t see how all regression tests could be automated. But I’ve now been on enough projects where we automated all the regression testing. I not only see that it works but also see that it’s necessary. Agile teams that don’t automate their regression tests are forever playing catch up and often find that they don’t have enough time for exploring even though it’s important.

Crispin: Your session description mentions that participants will get to do exploratory testing of a hand-held electronic game. What can participants expect to take back to their real jobs, in terms of practical application?

Hendrickson: It might seem like testing a little toy is nothing like testing real, serious software. But it turns out that software is software. These games can be just as vulnerable to issues related to inputs and outputs, states and transitions, sequences and timing, and configuration as any other software.

So in the session, we’ll use heuristics to discover the capabilities and limitations of the devices. We’ll come up with charters to guide our exploration. And we’ll practice debriefing test sessions to share information.

Back in the real world, participants will be able to use those same techniques in exploring and reporting on the software they work on day-to-day.

The conversation between Crispin and Hendrickson continues with Developers testing software: Q&A with an Agile guru.


Elisabeth Hendrickson is the founder and president of Quality Tree Software, Inc., a consulting and training company dedicated to helping software teams deliver working solutions consistently and sustainably. She also created Entaggle, a community-based site for giving and getting professional recognition. And she founded Agilistry Studio, a practice space for Agile software development in Pleasanton, CA.

A software professional for over 20 years, Elisabeth has been a member of the Agile community since 2003. She served on the board of directors of the Agile Alliance from 2006 - 2007 and is one of the co-organizers of the Agile Alliance Functional Testing Tools program. Elisabeth splits her time between teaching, speaking, writing, programming, and working on Agile teams with test-infected programmers who value her obsession with testing. You can find her on Twitter as @testobsessed.

Dig Deeper on Topics Archive