I recollect using exploratory test techniques to uncover hidden defects for a software product organization, in my role as a tester 12 years ago, despite lacking formal training or awareness. It was my urge to uncover hidden defects which drove me to adopt exploratory test methods. Today, even as software testing gurus continue to debate the merits and demerits of exploratory testing, its awareness and practice, with the exception of some software product development organizations, continues to shrink rapidly. When I started thinking about the possible reasons why the majority of the testers today fail to adopt exploratory testing, I realized that the gap really lies in the design of the overall test strategy. Most of the test strategies today heavily focus on scripted test methods with limited or no scope for the usage of exploratory test methods . There is no doubt that exploratory testing does have certain advantages, but in order for us to leverage the same, we need to revisit our application testing strategies and ensure a fine balance between scripted and exploratory test methods and tools. This balance would ensure an increase in the effectiveness of the testing function without compromising testing efficiencies.
What is the Current Role of Scripted Testing Vs. Exploratory Testing?
In the last 15 years, as testing methodologies have undergone several changes, scripted testing has taken center stage. QA teams currently focus on ensuring the application under development complies with the stated requirement specifications, rather than trying to evaluate the behavior of the application under development through:
- Extensive application and business knowledge.
- Innovative and creative methods simulating possible user behavior.
- Ensuring thorough evaluation while managing time.
Some of the possible explanations/situations that led to the industry-wide adoption of scripted testing:
- Pace of technology advancement (from colossal mainframes to the iPad) has significantly driven change across IT systems. In turn, this has increased the volume of testing required drastically.
- Insufficient availability of SMEs has made it difficult to keep up with volume of testing.
- Rising costs and need to improve efficiency necessitated commoditization of the testing function. This resulted in reduced involvement of SMEs in the testing process.
- Improved IT project management practices pushed the need for better control on budgets and the outcome of the quality to be quantified. This has significantly pushed scripted testing in comparison to exploratory testing.
- Requirements management continues to be a pain point in many IT organizations resulting in a push to strengthen traceability of requirement to test cases.
- As the size of applications increased and business process flows started to span multiple applications, it became necessary to adopt a more process-oriented approach to test these complex systems. This is precisely what scripted testing offered.
Why we need exploratory testing and its significance in today’s world
While IT organizations across industry verticals continue to improve the quality of their product, according to my experience, the defect removal efficiency of the overall IT organizations has not improved beyond 85%. However, a majority of the testing groups within the IT organizations have transformed themselves and become more efficient, reporting defect removal efficiencies of 95% to 98%. Most IT process gurus are analyzing this situation and trying to devise strategies to address this big quality gap that exists between what testing groups are claiming and what the overall IT application quality really is. It is very clear that quality is just not a testing problem, and the entire IT organization needs quality focus to improve the defect removal efficiency of the entire IT organization from the current levels. One such strategy that can be adopted to ensure the same is exploratory testing. Of course it would need to be carefully married to the organization’s master test strategy in order for it to yield the planned improvements in quality.
Implementing a combined strategy
Even though most testing professionals focus on scripted test execution, the natural tester’s instinct pushes her/him to explore the system completely in the available time. Structured methods and initiatives can improve the efficiency of exploratory testing when carefully combined with mainstream scripted execution. While attempting to devise a successful strategy one must keep in mind certain ground realities:
- IT organizations continue to struggle with subject matter expert (SME) availability.
- To continue to provide testing resources aligned to an application/domain is not a best practice due to increasing cost of QA. This especially is not worth it when most CXOs are looking to either maintain or reduce cost of testing in comparison to overall project cost.
- Organizations are unable to measure and track “Cost of Quality.”
However, despite these challenges, testers are encouraged to think beyond their test scripts thanks to the focus of IT organizations on quality of applications. The table below outlines a few initiatives which I believe will help in successfully adopting exploratory testing to augment scripted testing:
While I agree with industry gurus on the importance and role of exploratory testing, its role as the primary test strategy, as argued by many, is questionable due to its inability to support large volumes of testing and control the behavior of large test teams. At the end of the day, economics dictate what is feasible and what isn’t, and industry indicators definitely show a move towards the commoditization of the testing function. The concept of introducing quality into the product and pushing for early detection of defects is forcing QA teams to develop test strategies which adopt exploratory testing and integrate it with scripted testing seamlessly. It is also shifting the focus towards validation at the level of business transactions, grooming of SMEs and building business stewards (owners of a business process or end-to-end business transaction) who will have ability to adopt exploratory testing methods effectively. Training and awareness is also a key contributor for the success of exploratory testing methods in this ever-growing tester’s community. QA managers must implement exploratory testing techniques as an additional strategy, combined with scripted testing, to improve the overall quality of applications.