Manage Learn to apply best practices and optimize your operations.

Building a Performance Assurance Center of Excellence tutorial

Thorough testing prior to deploying a software application decreases glitches, bugs and other issues. In this expert tip, learn how to build a test center focused on performance.


Unfortunately, users usually remember the worst experience with a company better than the best experience. So, performance testing is a crucial component of any deployment because application performance has a direct impact on the end-user experience, which directly impacts your revenue stream. Performance testing can make the difference between a satisfied customer and a customer who switches to a competing provider.

Performance problems often have a significant affect on the user experience because of the extended time and effort required to remediate performance issues. Performance issues don't always get resolved overnight, and performance fixes often require a total interruption in service.

Performance testing mitigates your performance risk by measuring the ability of your application and your architecture to meet the customer's performance expectations before it is exposed to the user community. It's a must, not a nice to have.

Table of contents

Introduction: The role of performance testing
Performance test costs versus application failure risks
Why create a Performance Assurance Center of Excellence?
The Performance Assurance Center of Excellence model
The Performance Assurance Center of Excellence testing lifecycle
Performance testing leadership roles
Performance testing isn't just testing

Performance test costs versus application failure risks

Fixing problems that are found in production due to a lack of performance testing during development often increases costs of the overall project by at least 25 percent and can result in complete project failure, according to a study by Gartner Inc.. Yet, performance testing only costs on average of two-three percent of the overall application development budget. Performance planning and performance engineering costs can extend this cost some, but in a stealthy way; by which I mean that they are usually part of any deployment and often are not factored into return-on-investment. So, it's surprising that companies would rather face a 25 percent loss than make a single-digit percentage investment in performance testing.

A Newport Group study showed, and my own experience confirms, that at least 60 percent of applications that are not performance tested experience a significant performance problem or failure within the first year of their release. One recent example I saw happen was a bug in site management software utilized by companies with a significant percentage of worldwide web traffic. The bug resulted in performance problems for many sites simultaneously and required disabling of the software until the bug was fixed.

So, let's look at the results of the studies cited above in the context of a one million dollar project. So, a $30,000 investment made in pre-production performance testing can prevent $250,000 worth of potential rework. In short, doing pre-production performance testing is at least five times cheaper than not doing performance testing at all.

Effective performance testing mitigates many of the application risks by providing performance metrics against known loads. Performance testing identifies performance constraints, bottlenecks and though-put issues. Performance testing provides the opportunity to optimize the application code and infrastructure/architecture, a much cheaper and less risky proposition than doing so on a post-release live application.

To the top

Why create a Performance Assurance Center of Excellence?

In a previous article, I detailed how to create a Testing Center of Excellence (TCoE). A TCoE does not directly address performance testing. Instead, the TCoE addresses testing the functionality of the applications under test.

Performance testing is really not about testing the functionality of the application, it is about measuring the performance of the overall architecture under load. The amount of resources that should be applied within the performance testing space and the overall capacity of your organization to perform this type of testing are dependent on the answers to the following questions:

  • How often will performance testing occur?
  • How rigorous should the performance esting will/should be?
  • How much risk and/or cost is encored when performance issues occur in production?
  • How many and how often are technical resources assigned to performance testing?

Seriously consider forming a Performance Assurance Center of Excellence (PACoE)
if your answers to these questions show that your application team or company is making serious ongoing efforts to carry out performance testing and if this effort will or does consume several resources then your organization.

The mission of a PACoE is to produce performance "saves" as early in the process as possible; a "save" being defined as the detection of a performance issue before it reaches production.

A PACoE champions performance assurance best practices, defines and implements performance testing processes, rolls these processes out to the operational team and helps set the strategic direction of the organization.

The PACoE's core tasks include:

  • Carrying out performance testing;
  • Supporting capacity planning; and,
  • Supporting performance assurance practices.

To the top

The Performance Assurance Center of Excellence model

The structure of a PACoE resembles that of any center of excellence and mirrors the structure of the Testing Center of Excellence. The difference is that the TCoE is focused on the business functionality of the solution while the PACoE is focused on the capacity of the overall architecture. The disciplines of capacity planning, release management and operations are closely related to the PACoE; indeed, capacity planning can be a component within the PACoE.


The PACoE model supports a group of performance assurance specialists and technical performance testing components leveraged across all projects. The performance testing specialists will educate and supplement the resources that exist within each project team to ensure consistency. This will allow for maximum project penetration with a minimum number of human resources. The PACoE will build a base of shared best management and performance testing practices across projects for a more efficient use of resources. Reusability of performance test deliverables such as test plans, test cases, fault reports, etc. will be promoted though the PACE.  This group will maintain a repository of deliverables as they are created for each project and provide them as a framework and samples, for future efforts.

In general, a PACE can help you achieve the following performance objectives:

  • Increased confidence of success when rolling out new code, new software and new hardware configurations by allowing you to identify capacity constraints, concurrency problems, and other performance bottlenecks before going to productionImprove improve the performance of existing code through application tuning reducing overall resource requirements while decreasing overall response times; and,
  • Gaining a better understanding of the performance profile of your architecture and its scalability characteristics to help in ongoing capacity planning/management.

To the top

The PACE testing lifecycle

Performance testing is only one of the activities involved within the discipline of performance assurance. To get the greatest return from your performance assurance practice, performance assurance activities should be applied from product conception to deployment, including ongoing updates and eventual retirement. The following table illustrates what performance assurance activities and gates should be applied in the context of code maturity.

Code Maturity

Software Exists as

Performance Assurance Gate

(no code)

Requirements &

Preliminary Risk Assessment

  • Application Risks
  • Infrastructure Risks

Construction Complete

Functional Software

Application Risk Assessment
Preliminary Application Performance Test

Functionality Confirmed

Package or Release

Application Performance Test
Application Tuning
Preliminary Infrastructure Tuning

Performance Testing

Package or Release

Infrastructure Performance Test
Infrastructure Tuning

To the top

Performance testing leadership roles

The effective implementation of a PACoE involves the integrated efforts of people acting in several roles. These roles may be assumed by one or more individuals. The size and depth of the performance assurance team is dependent on the overall scope and complexity of ongoing performance testing. If performance testing is an intermittent activity within your organization, then these roles can be assumed by skilled performance assurance testing consultants.

Performance assurance manager or lead

In this leadership role within the PACoE, the manager is responsible for development of a Performance Assurance practices and the implementation of these practices for any given initiative. The manager ensures that the organization follows established performance assurance protocols and grows these protocols as the organization matures.

Performance test engineer

The performance test automation engineer builds, executes, and assesses the performance test. This is a technical role that usually requires a broad scope of experience within the IT space, including software development, database development, operational support/experience and a broad understanding the overall architecture.

Performance analyst

The performance assurance analyst captures the performance requirements for the application space from a business and technical perspective. The analyst then ensures that the tests built and executed by the performance test automation engineer measure the ability of the architecture to meet these performance requirements.


Performance testing isn't just testing

Performance testing is one of the most misunderstood areas within IT. There is a tendency to group performance testing under the general testing umbrella without understanding the intrinsic architectural nature of performance assurance testing. Performance testing is often an after-thought of any given project; this combined with the unfortunate belief that anyone can apply performance testing tools appropriately often leads to systemic performance issues. My experience shows the belief that "anyone can use performance testing tools appropriately" to be demonstrably untrue and the cause of many avoidable post implementation performance issues.

To the top


David W. Johnson, or DJ, is a senior test architect with over 22 years of experience in IT across several industries. He has played key roles in business needs analysis, software design, software development, testing, training, implementation, organizational assessments and support of business solutions. Over the past 12 years, he has worked in the area of implementing test-ware, handling test strategies, test planning, functional and performance test automation and test management solutions.

Dig Deeper on Topics Archive

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.