Manage Learn to apply best practices and optimize your operations.

Performance testing in the Agile enterprise

Fitting performance testing into an existing or transitioning Agile enterprise is not easy. In this tip, software performance expert Scott Barber explains the challenges associated with performance testing in Agile environments and describes three approaches that Agile teams take in order to include performance test expertise on the team.

Are you part of an organization that is “going” or has recently “gone” Agile?  Are you involved with performance testing? Are you trying to figure out how to make performance testing work in this new culture? If so, this tip is for you.

Agile teams execute performance test throughout the lifecycle

Most organizations are already rather Agile in the way they conduct performance testing. I say “rather Agile” because of the tendency of organizations to treat performance testing as an “on again, off again” activity conducted by an individual or team that is either entirely isolated from, or loosely attached to, the development team. When that team is developing applications using a non-Agile SDLC, this situation tends to be sub-optimal, but more or less good enough because everyone accepts that although performance testing is being conducted one, or even two, builds behind development, it is happening regularly enough to be dramatically more valuable than conducting either no performance testing or only performance testing release candidate builds. 

However, performance testing itself is inherently Agile due to its continual, iterative feedback loop. Practically speaking, every performance test results in one of three states; no new information, new issue raised, or new risk identified with the next performance activity being entirely determined by which of these states the previous test results in. See Figure 1 for a simple, but useful depiction of the performance testing cycle.

Figure 1: Performance testing cycle

I often describe this figure as representing a repeating cycle of performance testing activities, complicated by unknowns, estimations and approximations.

Integrating performance testing into Agile development is not inherently easy

To understand why integrating existing, Agile performance testing practices into Agile development cycles is not inherently easy, let’s first consider a depiction of a generic Agile software development model. See Figure 2.

Figure 2: Visual depiction of a generic Agile development cycle

I often describe this figure as a repeating cycle of core Agile activities gated by inexact and variable notions of acceptance.

Pretty simple, right? But where things get complicated is when an organization tries to integrate an existing performance testing model into an Agile development cycle that was established without building performance testing in from the beginning. See Figure 3.

Figure 3: Integrating performance testing into an existing Agile development model

If after glancing at Figure 3, you thought, “Wow, that looks complicated!” your assessment is consistent with my experience. The obvious conclusion to draw is that it is far more efficient to integrate performance testing while implementing Agile development. Unfortunately, that message is only valuable for organizations that are planning, but have not yet started transitioning to an Agile approach.

Three approaches to performance testing in Agile

For those organizations that already have some flavor of Agile development, but haven’t fully integrated performance testing, I offer three approaches that I have witnessed work with varying levels of success: on demand, on retainer, and full immersion.

On demand

Also known as a “Center of Excellence,” this model is the functional equivalent of outsourcing to an internal group.  This is not an overly-Agile model, but this is where most organizations that are trying to integrate their performance testing into their Agile development cycles start because performance testing was already an “on demand” model before the Agile transition began.

For the “on demand” model of performance testing to work with an Agile development cycle, there are a few things that need to be handled differently than they probably were for non-Agile-development efforts.  In particular:

  • Make use of “on demand” services periodically, not only for production release candidate builds.
  • Performance targets, goals, objectives, and/or budgets must become a standard part of each user story.
  • Developers must take responsibility for testing unit-level, component-level, integration-level, and single-user performance testing and tuning.
  • A full-time member of the team must take overall responsibility for managing performance related tasks, including those not explicitly mentioned here.

“On demand” is likely to be insufficient in situations where performance is very important, situations where the developers are not doing performance testing and tuning continually at their levels, and/or when no full-time team member takes responsibility for coordinating, championing, and managing performance-related tasks.

On retainer

The “on retainer” model is frequently the model that organizations use as an interim step between “on demand” and “full immersion.” This is typically because the organization does not have sufficient performance testers, performance test environments, and/or performance testing tools to support the “full immersion” model… at least not yet.

In this model, each development project is assigned a specific performance tester (as opposed to whoever is available when an “on demand” request comes in), but that performance tester is assigned to two or more development projects. While this model brings more performance testing expertise to each project and more project-level knowledge to the performance tester, the performance tester falls short of becoming a fully integrated member of the team. The result is that the performance tester tends to work mostly independently, but does provide some advice and guidance periodically. For this model to work and provide incrementally more value than the “on demand” model, the same requirements as the “on-demand” must be true and in addition:

  • The performance tester must be available to the team at times that are critical to the performance evolution of that project

The same situations that jeopardize the “on demand” model apply to the “on retainer” model.  In addition, the “on retainer” model generally requires more performance testers, environments, and tools than “on demand,” but the largest and most common reason for the “on retainer” model to fail to provide increased value, is that times that are critical to the performance evolution of one project have a nasty habit of cropping up at the same time on two or more of the projects a single performance tester is serving.

Full immersion

This should be the goal for every project team in an Agile enterprise, at least if performance is important to the value of the product and/or the reputation of the organization. In the “full immersion” model, there is a full-time member of the team who both has a particular specialty in both delivering and testing performance and has a primary responsibility for coordinating and managing performance related activities throughout the entire development lifecycle, possibly even throughout the entire product lifecycle.

Notice that I said has a primary responsibility, not the sole responsibility. Performance remains a part of everyone’s responsibility, and the performance specialist will most certainly contribute to other aspects of the development process commensurate with their individual skills and the needs of the project.

Achieving “full immersion” is most frequently thwarted by an organization not having enough individuals with the correct skills, not having enough test environments, and not having sufficient tools to assign each development team their own resources that they won’t be asked to share.


Though performance testing’s iterative nature makes it inherently Agile, it can be a challenge to integrate performance fully into an existing Agile team. These three performance testing models provide options in the Agile enterprise that include performance test resources on the team and ensure performance testing is getting the attention required throughout the development lifecycle.




Dig Deeper on Topics Archive