Performance testing in the Agile enterprise
Are you part of an organization that is “going” or has recently “gone” Agile? Are you
involved with performance testing? Are you trying to figure out how to make performance testing
work in this new culture? If so, this tip is for you.
Agile teams execute performance test throughout the lifecycle
Most organizations are already rather Agile in the way they conduct performance testing. I say
“rather Agile” because of the tendency of organizations to treat performance testing as an “on
again, off again” activity conducted by an individual or team that is either entirely isolated
from, or loosely attached to, the development team. When that team is developing applications using
a non-Agile SDLC, this situation tends to be sub-optimal, but more or less good enough because
everyone accepts that although performance testing is being conducted one, or even two, builds
behind development, it is happening regularly enough to be dramatically more valuable than
conducting either no performance testing or only performance testing release candidate
However, performance testing itself is inherently Agile due to its continual, iterative feedback
loop. Practically speaking, every performance test results in one of three states; no new
information, new issue raised, or new risk identified with the next performance activity being
entirely determined by which of these states the previous test results in. See Figure 1 for a
simple, but useful
depiction of the performance testing cycle.
Figure 1: Performance testing cycle
I often describe this figure as representing a repeating cycle of performance testing
activities, complicated by unknowns, estimations and approximations.
Integrating performance testing into Agile development is not inherently easy
To understand why integrating existing, Agile performance testing practices into Agile
development cycles is not inherently easy, let’s first consider a depiction of a generic Agile
software development model. See Figure 2.
Figure 2: Visual depiction of a generic Agile development cycle
I often describe this figure as a repeating cycle of core Agile activities gated by inexact and
variable notions of acceptance.
Pretty simple, right? But where things get complicated is when an organization tries to
integrate an existing performance testing model into an Agile development cycle that was
established without building performance testing in from the beginning. See Figure 3.
Figure 3: Integrating performance testing into an existing Agile development
If after glancing at Figure 3, you thought, “Wow, that looks complicated!” your assessment is
consistent with my experience. The obvious conclusion to draw is that it is far more efficient to
integrate performance testing while implementing Agile development. Unfortunately, that message is
only valuable for organizations that are planning, but have not yet started transitioning to an
Three approaches to performance testing in Agile
For those organizations that already have some flavor of Agile development, but haven’t fully
integrated performance testing, I offer three approaches that I have witnessed work with varying
levels of success: on demand, on retainer, and full immersion.
Also known as a “Center of Excellence,” this model is the functional equivalent of outsourcing
to an internal group. This is not an overly-Agile model, but this is where most organizations
that are trying to integrate their performance testing into their Agile development cycles start
because performance testing was already an “on demand” model before the Agile transition began.
For the “on demand” model of performance testing to work with an Agile development cycle, there
are a few things that need to be handled differently than they probably were for
non-Agile-development efforts. In particular:
- Make use of “on demand” services periodically, not only for production release candidate
- Performance targets, goals, objectives, and/or budgets must become a standard part of each user
- Developers must take responsibility for testing unit-level, component-level, integration-level,
and single-user performance testing and tuning.
- A full-time member of the team must take overall responsibility for managing performance
related tasks, including those not explicitly mentioned here.
“On demand” is likely to be insufficient in situations where performance is very important,
situations where the developers are not doing performance testing and tuning continually at their
levels, and/or when no full-time team member takes responsibility for coordinating, championing,
and managing performance-related tasks.
The “on retainer” model is frequently the model that organizations use as an interim step
between “on demand” and “full immersion.” This is typically because the organization does not have
sufficient performance testers, performance test environments, and/or performance testing tools to
support the “full immersion” model… at least not yet.
In this model, each development project is assigned a specific performance tester (as opposed to
whoever is available when an “on demand” request comes in), but that performance tester is assigned
to two or more development projects. While this model brings more performance testing expertise to
each project and more project-level knowledge to the performance tester, the performance tester
falls short of becoming a fully integrated member of the team. The result is that the performance
tester tends to work mostly independently, but does provide some advice and guidance periodically.
For this model to work and provide incrementally more value than the “on demand” model, the same
requirements as the “on-demand” must be true and in addition:
- The performance tester must be available to the team at times that are critical to the
performance evolution of that project
The same situations that jeopardize the “on demand” model apply to the “on retainer”
model. In addition, the “on retainer” model generally requires more performance testers,
environments, and tools than “on demand,” but the largest and most common reason for the “on
retainer” model to fail to provide increased value, is that times that are critical to the
performance evolution of one project have a nasty habit of cropping up at the same time on two or
more of the projects a single performance tester is serving.
This should be the goal for every project team in an Agile enterprise, at least if performance
is important to the value of the product and/or the reputation of the organization. In the “full
immersion” model, there is a full-time member of the team who both has a particular specialty in
both delivering and testing performance and has a primary responsibility for coordinating and
managing performance related activities throughout the entire development lifecycle, possibly even
throughout the entire product lifecycle.
Notice that I said has a primary responsibility, not the sole responsibility. Performance
remains a part of everyone’s responsibility, and the performance specialist will most certainly
contribute to other aspects of the development process commensurate with their individual skills
and the needs of the project.
Achieving “full immersion” is most frequently thwarted by an organization not having enough
individuals with the correct skills, not having enough test environments, and not having sufficient
tools to assign each development team their own resources that they won’t be asked to share.
Though performance testing’s iterative nature makes it inherently Agile, it can be a challenge
to integrate performance fully into an existing Agile team. These three performance testing models
provide options in the Agile enterprise that include performance test resources on the team and
ensure performance testing is getting the attention required throughout the development
This was first published in October 2011