Tip

Agile testing on large-scale projects

Agile testing on large-scale projects means different things to different test groups. The context is dependent on several factors and the decision on which approaches to testing

    Requires Free Membership to View

are most appropriate are dependent on these factors. What are some of the key factors that influence your approach to agile testing on large-scale projects? What testing approaches are more likely to harvest the greatest return on your testing investment -- given these factors?

Key factors

There are several key factors that determine both the success and weight of the overall agile testing effort on large-scale projects. They break down into the overall agile approach being applied, your previous success in applying this approach to agile in the large, the size and complexity of the existing (non-agile) code base, and the long-term ownership of the application being developed. So, key factors, for purposes of this discussion, are:

  • Experience
  • Agile approach
  • Existing non-agile code base
  • Long-term ownership

Experience

The more experience your overall project team has in deploying agile in the large, the clearer the understanding of what the escape velocity of defects into production should be. If the overall defect escape velocity into production is deemed to be unacceptable, then appropriate steps need to be taken to mitigate the risk and increase the number of saves. For example:

  • More technical reviews/walk-throughs of the solution
  • More peer reviews of the code
  • An assessment of current deployment velocity -- perhaps the sprints are too big
  • Finally, more testing (automated and manual)

Agile approach

The overall agile approach has a significant impact on the agile testing approach for your large project. If your agile approach includes aspects of self-testing code either through test-driven development (TDD), code instrumentation, peer reviews or some mixture of these techniques and approaches, then the overall weight of post-development testing can be reduced -- the working assumption being that these techniques and approaches are being applied appropriately by an experienced team. If these types of development techniques are not being applied, then the test group needs to consider increasing the testing investment to increase the number of saves (defects not reaching production). For example:

  • Application of traditional outside-in testing frameworks (GUI testing tools)
  • Application of technical reviews early in the development process
  • Application of traditional manual testing techniques (including manual test scripts)
  • Application of exploratory testing techniques

Existing non-agile code base

Regardless of your team’s experience and success with your chosen agile approach, the characteristics of the currently deployed non-agile code base must weigh in heavily on your agile large-scale testing approach. Essentially, for those agile projects that are dealing with a large, traditional code base, the existing application space should be treated as a traditional regression testing challenge. This would normally entail a mixture of manual testing, GUI based automation testing, and some level of exploratory testing. As the traditional code base stabilizes, the emphasis can shift to an automated regression suite.

Long-term ownership

The longer your agile team is going to be responsible for your code, the more the overall team should focus on using testing techniques that automate the testing process. The reason for this is quite simple; the volume of existing code is going to quickly overwhelm your testing group, splitting the focus of your testers and your team (existing vs. sprint). The solution is to integrate testing into the entire agile process always with an eye on how to automate testing. For example:

  • Application of code instrumentation by developers
  • Application of TDD by developers and testers
  • Application of outside-in (GUI based) testing tools

Agile testing approaches on large-scale projects

As we have seen, the agile testing approach is influenced by several key factors. Testing must deal with the functionality introduced by the current sprint, the existing functionality within the application space and the maturity and stability of the existing application space. As there are only a limited number of testing approaches available, the application and investment in any given approach must be weighed against the overall return on testing investment. The following chart weights the overall return on investment of basic testing techniques versus key agile project factors. The testing techniques include technical reviews or walk-throughs (reviews), traditional manual test scripts (manual), exploratory testing (exploratory), and traditional automation (automation). The return on testing investment is rated as high, medium and low in an attempt to illustrate the return in terms of number of saves versus consumption of full time equivalents (tester hours).

Return on Testing Investment

Key Factor

Reviews

Scripts

Exploratory

Traditional Automation

Agile Experience in the “large”

High

Medium

High

Med-High

Code Instrumentation or TDD

High

Low

High

Med-High

No Code Instrumentation or TDD

High

Low

High

High

Large existing non-Agile Code

High

Medium

Med-High

High

Long-term Ownership

High

Medium

Med-High

High

 

What the chart really shows is that getting in front of the testing challenge early in the development process reduces the overall weight of the testing effort. Agile processes that insert quality early in the process reduce testing cost, and those that introduce instrumentation earlier in the testing process significantly reduce testing weight -- given this circumstance lighter-weight testing approaches will yield the same level of quality.

 

David W. Johnson “DJ” is a senior test architect with over 25 years of experience in information technology across several business verticals and has played key roles in business analysis, software design, software development, testing, disaster recovery and post implementation support. Over the past 20 years, he has developed specific expertise in testing and leading QA/Test team transformations – Delivered Test: Architectures, Strategies, Plans, Management, Functional Automation, Performance Automation, Mentoring Programs, and Organizational Assessments.

This was first published in January 2011

There are Comments. Add yours.

 
TIP: Want to include a code block in your comment? Use <pre> or <code> tags around the desired text. Ex: <code>insert code</code>

REGISTER or login:

Forgot Password?
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy
Sort by: OldestNewest

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to:

Disclaimer: Our Tips Exchange is a forum for you to share technical advice and expertise with your peers and to learn from other enterprise IT professionals. TechTarget provides the infrastructure to facilitate this sharing of information. However, we cannot guarantee the accuracy or validity of the material submitted. You agree that your use of the Ask The Expert services and your reliance on any questions, answers, information or other materials received through this Web site is at your own risk.