First off, let's get on the same page about what exploratory testing is. Generally, it's considered a mode of simultaneous learning, test design and test execution. It's a method of testing that values the tester as an integral part of the test process, and my experience has been that it's an approach that shares the same values as the Agile Manifesto:
- Individuals and interactions over processes and tools
- Working software over comprehensive documentation
- Customer collaboration over contract negotiation
- Responding to change over following a plan
Unlike traditional test methods where the process might be phased, exploratory testing is more of a real-time process. Instead of test design and execution being separated by iteration or team, with each test executed the tester learns more about the product and uses that information to design the next best test to be executed. It's a process that allows the tester to respond to each test in a way that should maximize focus on the most relevant risks.
Why choose exploratory testing if you're working on an agile project?
I recently asked some testers who work on agile teams where they struggle. David Christiansen, a senior developer for the Collaborative Software Initiative, said that for him it's the pace. "Keeping up with the ever-changing context of the project is very critical in adding value as a tester who is not also writing the code. It's a bit embarrassing (and discrediting) to file a bug report only to have a developer say, 'That's because we haven't done the story on that bit yet.'" Most of the responses I received had some variation on the challenge of keeping up with the rest of the team.
To deal with the constant changes on an agile project, exploratory testers need to work to make the testing relevant and coherent to their project team. That means they need to understand which risks are important to the team. They need to understand where they should focus test coverage and work with the team to make daily decisions about what they will work on and how they will structure that work.
The real-time nature of exploratory testing can also help testers deal with the time pressures of agile projects. Exploratory testers need to have frameworks and techniques for structuring and managing their ideas. They have to be experts at generating test ideas while testing and tying those ideas to test techniques. That process starts with developing the ability to identify missing information and considering multiple possibilities and probabilities for explaining what they've found based on the information they have. It requires that they identify the relevant dimensions, variables and dynamics and use those to build and work with mental models of the software they are exploring.
When alternating between learning, test design and test execution, testers are switching among different activities and perspectives. Like the programmers on the team, they are working quickly, focused on testing the most important features at that moment and building out coverage as they go. As they work they continuously take feedback and use it to extend, refine, refactor and correct their testing. They have tools (mental and physical) that allow them to produce, manage, abandon and recover test ideas. This allows them to work fast with confidence.
Exploratory testers are also skilled at controlling how they interact with the world. They know that collaboration can be more productive than working in isolation. They are good at obtaining the tools and information needed to support their testing efforts. Above all, they are continually focused on improving their ability to interact with the software they test: understanding its business purpose, the technology and how it's configured, and establishing procedures for better control of experimental conditions. They collect different kinds of data and look at different aspects of the software than the programming staff might.
Where does exploratory testing fit in your agile project?
When I talk about where exploratory testing might fit in an agile project, I like to borrow heavily from Brian Marick's model for agile testing directions. In that model, Marick outlines two reasons why people on agile teams might desire tests. The first is in support of programming; these include the tests that support writing code (unit tests) or provide an indication of when the programmer might be finished (acceptance tests). The second reason is to critique the product; these tests "look at a finished product with the intent of discovering inadequacies." It's here, in the area of critiquing the product, where exploratory testing can play a major role in your agile project.
In most of the agile teams I've worked with, tests that support programming have been done by the programmers and are almost always automated in some fashion. You can think of these tests as asking the question, "Are we done?" Exploratory testing asks a different question: "Is there a problem here?" Testers working on agile teams bring value by looking for possible problems. The risks they focus on are informed by all the other testing already being done on the project; based on that, they focus their testing efforts on those areas where the existing automated tests might come up short.
Automated tests are important, but they can't always address the complexity of finding potential problems related to usability, reliability, performance, compatibility and other quality criteria. Automated tests can't ensure the documentation matches the final product. They don't typically cover test scenarios past common use (like disfavored use or extreme use). Automated tests are great for testing specific aspects of functionality, but for complex applications they aren't so great at testing the platform dependencies (external hardware or software, deployment configurations). They don't often address issues related to operations (dealing with how data ages, alarming/alerting, logging, etc.). And automated tests aren't often focused on relationships between the product and time (concurrency, changing rates or transactions, or delays on input).
Effective exploratory testers working on agile projects use the tactics of exploratory testing to help inform the project team about possible issues with the product. Their testing can be unstructured and freestyle or be managed using charters and test sessions. Exploratory testers often make use of performance, runtime analysis and other types of testing tools. When looking at introducing exploratory testing into your project, start by finding a skilled tester who's comfortable working in a rapidly changing environment, and have them work with the programming team on the next iteration of your project.
More on exploratory testing
For more information on the topic of exploratory testing in an agile environment, take a look at the following links:
- You can find a more detailed look at exploratory testing in James Bach's "Exploratory Testing Explained."
- "Exploratory Testing Dynamics" offers more information on the skills and tactics employed by successful exploratory testers.
- Take a look at Brian Marick's series of blog posts on agile testing directions.
- For some examples of using exploratory testing to identify possible issues, take a look at Michael Bolton's "Is there a problem here," a short, informal tutorial by James Bach, or Ben Simo's blog on the topic.