Chartering exploratory test efforts in Agile

Charters outline specific test routes taken when exploratory testing an application. Each charter identifies what is being tested and which system is being subjected to the test. Many agile teams have adopted this method of exploring applications for unexpected behavior, flaws in the source code and other oddities throughout the interface. Learn how to adopt and implement exploratory testing in this expert tip.

Chris McMahon
In recent years the consensus among agile practitioners is that exploratory testing (ET) is an activity that is required on agile teams. There are a number of guidelines for doing ET well: heuristics to keep in mind for all kinds of testing; failure modes of various kinds of systems; and charters to guide the scope of the testing. In this tip, we examine the use of charters to help guide exploratory test efforts.

Charters define scope
ET is a skilled and disciplined approach to testing. And one of the skills exploratory testers master is the ability to manage the scope of testing so that the software is tested in a thorough and appropriate manner. Testers manage the scope of exploratory testing using a concept called a "charter."

Charters are statements of what aspects of the system are to be tested. Charters, unlike what are called "scripts" in scripted testing, do not specify how the system is to be tested, only that some aspect of the system is to be tested. For example, a charter might say "test the login functionality" where a script might say "Type John Smith into the User field, and "[email protected]" into the Password field." A charter leaves the actual steps of the testing up to the skilled and disciplined tester. The reason for this is that a skilled and disciplined tester might notice that an extra character at the end of the password still lets the user log on, where a scripted test would never expose such an error.

Designing charters for exploratory testing is one of the most difficult aspects of the work. It is hard to know how much testing is enough, or what aspects of the system need more coverage, or how long a tester should spend examining any particular aspect of the system. However, agile software projects often produce exactly the charters that an exploratory tester needs to work.

Tracking Agile work
A typical agile project will start with a backlog of stories. The stories in the backlog are a list of all the changes that the business might want to make to the system. The backlog is changing all the time as the priorities of the business change. Stories are managed in various ways. Some teams start with very high level descriptions of big features (often called "epics") and then create smaller subsidiary stories that describe granular, more easily handled aspects of the epic story. Some teams make a dedicated effort to have only small stories in the backlog.

For every iteration, the team creates a set of tasks related to each story. If Story A is in the iteration, tasks for Story A might be: "code X"; "code Y"; "code Z"; "integrate X and Z"; "write user documentation"; "do testing." When all the tasks are completed, the story is considered done, and is ready for release to production.

Related Content:
From STAREast: Is exploratory testing crucial or optional?
Although proving vital, exploratory testing is often forced to take a backseat to numerous application testing initiatives.

Six tours for exploratory testing the business district of your application
Exploratory testing is sometimes thought to be "directionless testing", but James Whitaker's new book suggests that exploratory testing, is more structured than it is given credit for.

The status of stories and tasks changes throughout the iteration, and many agile teams track this status by moving the cards that represent the stories and tasks in and out of columns on a board or wall. Picture a whiteboard with columns that might be marked something like "Not Started," "In Development," "In Review," "In Test," and "Done." Story and task cards move across the board from left to right, sometimes going backward if a problem of some sort arises in conjunction with one of the stories or tasks.

The status column marked "In Test" is not what many testers might think. Moving a card into this column is not any kind of formal handoff or certification that the software is somehow fit for some purpose. For an agile team, a card with status "In Test" means that someone on the team would like a particular kind of attention paid to this particular card. And this is where ET comes in.

Stories and tasks as charters for ET
ET tasks come in every size, and this is what makes agile story and task cards ideal as charters for ET. From the example above:

When the card saying "code X" enters the "In Test" status, an exploratory tester may treat that as a charter. The tester might start by reading the source code, looking for issues in the code itself, and looking for how that code is physically connected to the rest of the system. Then the tester might exercise that code, either through the User Interface (UI), or directly by means of some behind-the-UI sort of access. Throughout the process, the tester is using skill and experience to search for issues that could potentially have a negative effect on the users, the business, or the team itself.

And when the "write user documentation" card enters the status "In Test", the tester takes up a charter to examine the user documentation. The tester is looking for problems with the language of the documentation, but the tester is also checking that the description of the system matches the actual behavior of the system, that nothing in the documentation is false or misleading, and that someone who reads the documentation will not be surprised by the behavior of the system being described.

Those are the small, granular tasks that an exploratory tester can use as charters for close, detailed work. But at some point, the larger epic story enters the status "In Test," and the exploratory tester can use that as a charter to do whole-system and end-to-end tests, "soap opera" tests (where everything in the system goes wrong at once), performance testing, or whatever other sorts of large-scale test activity is appropriate for the system.

Scope, focus and time
Charters define the scope of testing; if the charter is to test the coding of X, then testing the coding of Z is outside the scope of this particular charter. Testing for the "code X" task card suggests that the tester exercise a very narrow focus, concentrating closely on X to the exclusion of other examination.

In an agile team, the exploratory tester will know that behind the task for "code X" is another task for "integrate X and Z." So the tester can manage his time testing X because the tester knows there is another charter coming that will demand a wider focus than just a focus on X.

Particular charters are often tested in sessions, and sessions are often done for a set length of time. The test charter for X might specify 30 minutes and the test charter for the integration of X and Z might specify 60 minutes. At the end of the test session, it is good practice to create some sort of test report, whether that means creating a set of defect reports, closing a set of fixed defect reports, or creating a statement of what was examined and in what way.

Another approach to managing time for testing charters is to manage the number of charters "In Test" at any given time. For example, the team may allow only five charters "In Test" at any given time, and only allow a new charter in when an existing charter is retired.

Move those cards
Every aspect of an agile development process may serve as a charter for exploratory testing. As those cards move across that board, the work of the exploratory tester is constantly changing. One charter may have the tester examining the code itself or individual columns in database records, while another charter may have the tester examining the entire system from end to end. Move every card through the "In Test" column. Doing that guarantees that the team will have as much information about the system as possible.

Dig Deeper on Topics Archive