Manage Learn to apply best practices and optimize your operations.

Scripted testing vs. exploratory testing

The merits of scripted testing vs. exploratory testing are often debated among software quality gurus. In this tip, Vasudeva Naidu does a deep dive into both. While automation of scripted tests is certainly becoming more prevalent in the industry, there still is a need for exploratory testing. This tip will tell you how an appropriate balance can be reached in your test strategy.

I recollect using exploratory test techniques to uncover hidden defects for a software product organization, in...

my role as a tester 12 years ago, despite lacking formal training or awareness. It was my urge to uncover hidden defects which drove me to adopt exploratory test methods. Today, even as software testing gurus continue to debate the merits and demerits of exploratory testing, its awareness and practice, with the exception of some software product development organizations, continues to shrink rapidly. When I started thinking about the possible reasons why the majority of the testers today fail to adopt exploratory testing, I realized that the gap really lies in the design of the overall test strategy. Most of the test strategies today heavily focus on scripted test methods with limited or no scope for the usage of exploratory test methods . There is no doubt that exploratory testing does have certain advantages, but in order for us to leverage the same, we need to revisit our application testing strategies and ensure a fine balance between scripted and exploratory test methods and tools. This balance would ensure an increase in the effectiveness of the testing function without compromising testing efficiencies.

What is the Current Role of Scripted Testing Vs. Exploratory Testing?

In the last 15 years, as testing methodologies have undergone several changes, scripted testing has taken center stage. QA teams currently focus on ensuring the application under development complies with the stated requirement specifications, rather than trying to evaluate the behavior of the application under development through:

  • Extensive application and business knowledge.
  • Innovative and creative methods simulating possible user behavior.
  • Ensuring thorough evaluation while managing time.

Some of the possible explanations/situations that led to the industry-wide adoption of scripted testing:

  • Pace of technology advancement (from colossal mainframes to the iPad) has significantly driven change across IT systems.  In turn, this has increased the volume of testing required drastically.
  • Insufficient availability of SMEs has made it difficult to keep up with volume of testing.
  • Rising costs and need to improve efficiency necessitated commoditization of the testing function. This resulted in reduced involvement of SMEs in the testing process.
  • Improved IT project management practices pushed the need for better control on budgets and the outcome of the quality to be quantified. This has significantly pushed scripted testing in comparison to exploratory testing. 
  • Requirements management continues to be a pain point in many IT organizations resulting in a push to strengthen traceability of requirement to test cases.
  • As the size of applications increased and business process flows started to span multiple applications, it became necessary to adopt a more process-oriented approach to test these complex systems. This is precisely what scripted testing offered.

Why we need exploratory testing and its significance in today’s world

While IT organizations across industry verticals continue to improve the quality of their product, according to my experience, the defect removal efficiency of the overall IT organizations has not improved beyond 85%. However, a majority of the testing groups within the IT organizations have transformed themselves and become more efficient, reporting defect removal efficiencies of 95% to 98%. Most IT process gurus are analyzing this situation and trying to devise strategies to address this big quality gap that exists between what testing groups are claiming and what the overall IT application quality really is. It is very clear that quality is just not a testing problem, and the entire IT organization needs quality focus to improve the defect removal efficiency of the entire IT organization from the current levels. One such strategy that can be adopted to ensure the same is exploratory testing. Of course it would need to be carefully married to the organization’s master test strategy in order for it to yield the planned improvements in quality.

Implementing a combined strategy

Even though most testing professionals focus on scripted test execution, the natural tester’s instinct pushes her/him to explore the system completely in the available time. Structured methods and initiatives can improve the efficiency of exploratory testing when carefully combined with mainstream scripted execution. While attempting to devise a successful strategy one must keep in mind certain ground realities:

  • IT organizations continue to struggle with subject matter expert (SME) availability.
  • To continue to provide testing resources aligned to an application/domain is not a best practice due to increasing cost of QA. This especially is not worth it when most CXOs are looking to either maintain or reduce cost of testing in comparison to overall project cost.
  • Organizations are unable to measure and track “Cost of Quality.”

However, despite these challenges, testers are encouraged to think beyond their test scripts thanks to the focus of IT organizations on quality of applications. The table below outlines a few initiatives which I believe will help in successfully adopting exploratory testing to augment scripted testing:

Conclusion

While I agree with industry gurus on the importance and role of exploratory testing,  its role as the primary test strategy, as argued by many, is questionable due to its inability to support large volumes of testing and control the behavior of large test teams. At the end of the day, economics dictate what is feasible and what isn’t, and industry indicators definitely show a move towards the commoditization of the testing function. The concept of introducing quality into the product and pushing for early detection of defects is forcing QA teams to develop test strategies which adopt exploratory testing and integrate it with scripted testing seamlessly. It is also shifting the focus towards validation at the level of business transactions, grooming of SMEs and building business stewards (owners of a business process or end-to-end business transaction) who will have ability to adopt exploratory testing methods effectively. Training and awareness is also a key contributor for the success of exploratory testing methods in this ever-growing tester’s community. QA managers must implement exploratory testing techniques as an additional strategy, combined with scripted testing, to improve the overall quality of  applications.

This was last published in March 2011

Dig Deeper on Exploratory Software Testing

PRO+

Content

Find more PRO+ content and other member only offers, here.

Start the conversation

Send me notifications when other members comment.

By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy

Please create a username to comment.

-ADS BY GOOGLE

SearchMicroservices

TheServerSide.com

SearchCloudApplications

SearchAWS

SearchBusinessAnalytics

SearchFinancialApplications

SearchHealthIT

DevOpsAgenda

Close