Agile testing on large-scale projects

What are key factors to success in agile testing efforts on large-scale projects? Consultant David Johnson suggests they are experience, your agile approach, the size and complexity of your non-agile code base and the long-term ownership of the application under development. Taking each of these into consideration, Johnson describes approaches that will yield the greatest return on your testing investment.

Agile testing on large-scale projects means different things to different test groups. The context is dependent on several factors and the decision on which approaches to testing are most appropriate are dependent on these factors. What are some of the key factors that influence your approach to agile testing on large-scale projects? What testing approaches are more likely to harvest the greatest return on your testing investment --...

given these factors?

Key factors

There are several key factors that determine both the success and weight of the overall agile testing effort on large-scale projects. They break down into the overall agile approach being applied, your previous success in applying this approach to agile in the large, the size and complexity of the existing (non-agile) code base, and the long-term ownership of the application being developed. So, key factors, for purposes of this discussion, are:

  • Experience
  • Agile approach
  • Existing non-agile code base
  • Long-term ownership

Experience

The more experience your overall project team has in deploying agile in the large, the clearer the understanding of what the escape velocity of defects into production should be. If the overall defect escape velocity into production is deemed to be unacceptable, then appropriate steps need to be taken to mitigate the risk and increase the number of saves. For example:

  • More technical reviews/walk-throughs of the solution
  • More peer reviews of the code
  • An assessment of current deployment velocity -- perhaps the sprints are too big
  • Finally, more testing (automated and manual)

Agile approach

The overall agile approach has a significant impact on the agile testing approach for your large project. If your agile approach includes aspects of self-testing code either through test-driven development (TDD), code instrumentation, peer reviews or some mixture of these techniques and approaches, then the overall weight of post-development testing can be reduced -- the working assumption being that these techniques and approaches are being applied appropriately by an experienced team. If these types of development techniques are not being applied, then the test group needs to consider increasing the testing investment to increase the number of saves (defects not reaching production). For example:

  • Application of traditional outside-in testing frameworks (GUI testing tools)
  • Application of technical reviews early in the development process
  • Application of traditional manual testing techniques (including manual test scripts)
  • Application of exploratory testing techniques

Existing non-agile code base

Regardless of your team’s experience and success with your chosen agile approach, the characteristics of the currently deployed non-agile code base must weigh in heavily on your agile large-scale testing approach. Essentially, for those agile projects that are dealing with a large, traditional code base, the existing application space should be treated as a traditional regression testing challenge. This would normally entail a mixture of manual testing, GUI based automation testing, and some level of exploratory testing. As the traditional code base stabilizes, the emphasis can shift to an automated regression suite.

Long-term ownership

The longer your agile team is going to be responsible for your code, the more the overall team should focus on using testing techniques that automate the testing process. The reason for this is quite simple; the volume of existing code is going to quickly overwhelm your testing group, splitting the focus of your testers and your team (existing vs. sprint). The solution is to integrate testing into the entire agile process always with an eye on how to automate testing. For example:

  • Application of code instrumentation by developers
  • Application of TDD by developers and testers
  • Application of outside-in (GUI based) testing tools

Agile testing approaches on large-scale projects

As we have seen, the agile testing approach is influenced by several key factors. Testing must deal with the functionality introduced by the current sprint, the existing functionality within the application space and the maturity and stability of the existing application space. As there are only a limited number of testing approaches available, the application and investment in any given approach must be weighed against the overall return on testing investment. The following chart weights the overall return on investment of basic testing techniques versus key agile project factors. The testing techniques include technical reviews or walk-throughs (reviews), traditional manual test scripts (manual), exploratory testing (exploratory), and traditional automation (automation). The return on testing investment is rated as high, medium and low in an attempt to illustrate the return in terms of number of saves versus consumption of full time equivalents (tester hours).

Return on Testing Investment

Key Factor

Reviews

Scripts

Exploratory

Traditional Automation

Agile Experience in the “large”

High

Medium

High

Med-High

Code Instrumentation or TDD

High

Low

High

Med-High

No Code Instrumentation or TDD

High

Low

High

High

Large existing non-Agile Code

High

Medium

Med-High

High

Long-term Ownership

High

Medium

Med-High

High

 

What the chart really shows is that getting in front of the testing challenge early in the development process reduces the overall weight of the testing effort. Agile processes that insert quality early in the process reduce testing cost, and those that introduce instrumentation earlier in the testing process significantly reduce testing weight -- given this circumstance lighter-weight testing approaches will yield the same level of quality.

 

David W. Johnson “DJ” is a senior test architect with over 25 years of experience in information technology across several business verticals and has played key roles in business analysis, software design, software development, testing, disaster recovery and post implementation support. Over the past 20 years, he has developed specific expertise in testing and leading QA/Test team transformations – Delivered Test: Architectures, Strategies, Plans, Management, Functional Automation, Performance Automation, Mentoring Programs, and Organizational Assessments.

This was first published in January 2011

Dig deeper on Software Testing Methodologies

Pro+

Features

Enjoy the benefits of Pro+ membership, learn more and join.

1 comment

Oldest 

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to:

-ADS BY GOOGLE

SearchSOA

TheServerSide

SearchCloudApplications

SearchAWS

SearchBusinessAnalytics

SearchFinancialApplications

SearchHealthIT

Close