Manage Learn to apply best practices and optimize your operations.

Agile testing on large-scale projects

What are key factors to success in agile testing efforts on large-scale projects? Consultant David Johnson suggests they are experience, your agile approach, the size and complexity of your non-agile code base and the long-term ownership of the application under development. Taking each of these into consideration, Johnson describes approaches that will yield the greatest return on your testing investment.

Agile testing on large-scale projects means different things to different test groups. The context is dependent on several factors and the decision on which approaches to testing are most appropriate are dependent on these factors. What are some of the key factors that influence your approach to agile testing on large-scale projects? What testing approaches are more likely to harvest the greatest return on your testing investment -- given these factors?

Key factors

There are several key factors that determine both the success and weight of the overall agile testing effort on large-scale projects. They break down into the overall agile approach being applied, your previous success in applying this approach to agile in the large, the size and complexity of the existing (non-agile) code base, and the long-term ownership of the application being developed. So, key factors, for purposes of this discussion, are:

  • Experience
  • Agile approach
  • Existing non-agile code base
  • Long-term ownership


The more experience your overall project team has in deploying agile in the large, the clearer the understanding of what the escape velocity of defects into production should be. If the overall defect escape velocity into production is deemed to be unacceptable, then appropriate steps need to be taken to mitigate the risk and increase the number of saves. For example:

  • More technical reviews/walk-throughs of the solution
  • More peer reviews of the code
  • An assessment of current deployment velocity -- perhaps the sprints are too big
  • Finally, more testing (automated and manual)

Agile approach

The overall agile approach has a significant impact on the agile testing approach for your large project. If your agile approach includes aspects of self-testing code either through test-driven development (TDD), code instrumentation, peer reviews or some mixture of these techniques and approaches, then the overall weight of post-development testing can be reduced -- the working assumption being that these techniques and approaches are being applied appropriately by an experienced team. If these types of development techniques are not being applied, then the test group needs to consider increasing the testing investment to increase the number of saves (defects not reaching production). For example:

  • Application of traditional outside-in testing frameworks (GUI testing tools)
  • Application of technical reviews early in the development process
  • Application of traditional manual testing techniques (including manual test scripts)
  • Application of exploratory testing techniques

Existing non-agile code base

Regardless of your team’s experience and success with your chosen agile approach, the characteristics of the currently deployed non-agile code base must weigh in heavily on your agile large-scale testing approach. Essentially, for those agile projects that are dealing with a large, traditional code base, the existing application space should be treated as a traditional regression testing challenge. This would normally entail a mixture of manual testing, GUI based automation testing, and some level of exploratory testing. As the traditional code base stabilizes, the emphasis can shift to an automated regression suite.

Long-term ownership

The longer your agile team is going to be responsible for your code, the more the overall team should focus on using testing techniques that automate the testing process. The reason for this is quite simple; the volume of existing code is going to quickly overwhelm your testing group, splitting the focus of your testers and your team (existing vs. sprint). The solution is to integrate testing into the entire agile process always with an eye on how to automate testing. For example:

  • Application of code instrumentation by developers
  • Application of TDD by developers and testers
  • Application of outside-in (GUI based) testing tools

Agile testing approaches on large-scale projects

As we have seen, the agile testing approach is influenced by several key factors. Testing must deal with the functionality introduced by the current sprint, the existing functionality within the application space and the maturity and stability of the existing application space. As there are only a limited number of testing approaches available, the application and investment in any given approach must be weighed against the overall return on testing investment. The following chart weights the overall return on investment of basic testing techniques versus key agile project factors. The testing techniques include technical reviews or walk-throughs (reviews), traditional manual test scripts (manual), exploratory testing (exploratory), and traditional automation (automation). The return on testing investment is rated as high, medium and low in an attempt to illustrate the return in terms of number of saves versus consumption of full time equivalents (tester hours).

Return on Testing Investment

Key Factor




Traditional Automation

Agile Experience in the “large”





Code Instrumentation or TDD





No Code Instrumentation or TDD





Large existing non-Agile Code





Long-term Ownership






What the chart really shows is that getting in front of the testing challenge early in the development process reduces the overall weight of the testing effort. Agile processes that insert quality early in the process reduce testing cost, and those that introduce instrumentation earlier in the testing process significantly reduce testing weight -- given this circumstance lighter-weight testing approaches will yield the same level of quality.


David W. Johnson “DJ” is a senior test architect with over 25 years of experience in information technology across several business verticals and has played key roles in business analysis, software design, software development, testing, disaster recovery and post implementation support. Over the past 20 years, he has developed specific expertise in testing and leading QA/Test team transformations – Delivered Test: Architectures, Strategies, Plans, Management, Functional Automation, Performance Automation, Mentoring Programs, and Organizational Assessments.

Dig Deeper on Software Testing Methodologies

Join the conversation

1 comment

Send me notifications when other members comment.

Please create a username to comment.

crisp and good insight into the topic, thanks, its helpful.