Tip

Exploratory testing: Misunderstood concepts and common challenges addressed

Exploratory testing is probably the most common testing technique for software. Though widely employed, it is often misused and misunderstood by testing organizations and testers. Exploratory testing can harvest significant returns in terms of detected defects and overall software quality when employed appropriately. Combined with other testing techniques and technologies, it can become a significant testing force multiplier

    Requires Free Membership to View

Table of contents:
What is exploratory testing?
When to employ exploratory testing
Exploratory testing tutorial, part 2

What is exploratory testing?

Exploratory testing is often misconstrued to be the equivalent of undisciplined random testing of the application space, much in the same way that Agile development is often twisted to mean a very loose and undisciplined approach to software development. Neither definition is the intent of the discipline or technique, but it can certainly become the real-world implementation. For our purposes, let us define exploratory testing as structured ad hoc unscripted testing of an application space where the tester simultaneously learns the application space, evolves test cases, performs test execution and publishes defects.

Exploratory testing is really a way of thinking about testing and the application space under test. The definition speaks to a state of mind rather than a formal methodology. Our definition includes two key elements that are not often stressed in discussions on or about exploratory testing: structure and publication of defects.

Exploratory testing should be structured in the sense that it should be context driven, with the context determined by the goals of the current testing effort. The context of any exploratory testing effort is situational, but there are several common points of reference:

Functional area

  • Focus on a particular aspect of the application space (i.e., billing).
Production issues
  • Focus on a problematic area of the application identified by production issues.
Tester training/evaluation
  • Evaluation of the testing abilities or aptitude of a tester through paired testing. This can also be used as a mutual training exercise between testing peers.
New functionality or release
  • Focus on learning a new functionality while evolving test cases and identifying defects. This is the most common context for exploratory testing.
The primary value of any testing effort, including exploratory testing, is the detection and remediation of software defects -- to publish defects in a manner that expedites the remediation process. The same level of discipline as in any other testing effort should be applied when publishing defects detected with exploratory testing. The published defect must contain:

Defect name

  • The name or title is the essence of the defect, including the functional area and nature of the defect.
Defect description
  • The description clearly states what sequence of events leads to the defect, and includes a screen snapshot or transcriptions of any error messages.
How to replicate
  • The defect description provides sufficient detail for the triage team and the developer fixing the defect to duplicate the defect.
Defect severity
  • The severity assigned to a defect is dependent on the impact of the defect on the testing effort, and the risk the defect presents to the business if the defect is rolled out into production.
Impacted area
  • The impacted area can be referenced by the functional component or area of the system.
Exploratory testing is part of a testing continuum that goes from very informal exploratory testing to extremely rigid scripted test cases that capture every aspect of the test. All have their place in the overall testing process. The ability to leverage the appropriate approach at the appropriate time will yield the highest return for your testing investment.

To the top

When to employ exploratory testing

If we think of testing as a continuum that goes from completely freestyle exploratory testing to extremely rigid and structured scripted test cases, then where and when should we use structured ad hoc exploratory testing? The answer should be when it derives the most value. But how do you determine when that is? There are several common triggers that would move the focus of your testing efforts toward exploratory testing:

  • Quality concerns about a particular functional area
  • Production issues
  • Tester training and evaluation
  • New functionality or release

    In each case the trigger could come from the team as a whole or from a particular tester whose concerns are based on one or more of these common triggers. (Keep in mind that this is far from an exhaustive list.)

    There are two additional triggers for moving toward exploratory testing that are difficult to quantify but equally important: tester fatigue and target fixation.

    Exploratory Testing Myths
    Exploratory testing equates to random navigation through the application space.
    • Exploratory testing is a context-driven approach that requires a disciplined and skilled tester to detect anomalies within the application space.
    Exploratory testing can be done by any tester.
    • Exploratory testing requires experienced and skilled testers who are experts at their craft. They must be able to detect and then explore suspicious areas of the application in a fashion that detects as many defects as possible.
    Exploratory testing cannot coexist with other forms of testing.
    • Exploratory testing can, and essentially always does, coexist with other forms of testing. All experienced testers explore an application before formulating test cases, which is essentially exploratory testing.
    Exploratory testing is a less disciplined approach to testing.
    • Exploratory testing is a people-driven process that places an emphasis on context-driven testing. This does not imply that it is any less disciplined than other more scripted approaches.

    In a situation where most if not all testing is manual, tester fatigue can set in. Manual testing following a predefined set of scripts with known inputs and outputs does become a very mind-numbing activity. Eventually the tester begins to see what they expect to see, leading to detectable defects reaching production. Moving back and forth between manual scripted testing and unscripted structured exploratory testing will reinvigorate the testers, lead to new defects being discovered and improve the overall quality of the end product.

    Once again, in a situation where most if not all testing is predetermined scripted manual test scripts or even automated test scripts, target fixation can set in. Target fixation occurs when the tester becomes fixated on one aspect of the application to the exclusion of all other aspects of the application space. Moving toward exploratory testing and encouraging the exploration of other aspects of the application space will reinvigorate testers, allow them to refocus their efforts, lead to new defects being discovered and improve the overall quality of the end product. In all cases, leveraging exploratory testing in combination with other techniques and tools (i.e., test automation) will increase both job satisfaction and product quality.

    To the top


    David W. Johnson (DJ), is a senior test architect with more than 23 years of experience in information technology across several industries. He has played key roles in business needs analysis, software design, software development, testing, training, implementation, organizational assessments and support of business solutions. Johnson has developed specific expertise over the past 15 years on implementing "test ware," including test strategies, test planning, test automation (functional and performance) and test management solutions.

    This was first published in December 2009

  • There are Comments. Add yours.

     
    TIP: Want to include a code block in your comment? Use <pre> or <code> tags around the desired text. Ex: <code>insert code</code>

    REGISTER or login:

    Forgot Password?
    By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy
    Sort by: OldestNewest

    Forgot Password?

    No problem! Submit your e-mail address below. We'll send you an email containing your password.

    Your password has been sent to:

    Disclaimer: Our Tips Exchange is a forum for you to share technical advice and expertise with your peers and to learn from other enterprise IT professionals. TechTarget provides the infrastructure to facilitate this sharing of information. However, we cannot guarantee the accuracy or validity of the material submitted. You agree that your use of the Ask The Expert services and your reliance on any questions, answers, information or other materials received through this Web site is at your own risk.