Get started Bring yourself up to speed with our introductory content.

Choosing, using software, tools for a Testing Center of Excellence tutorial

Lack of an overall test architecture can lead to product choices that may be effective in the short-term but lead to additional long-term costs or even replacement of a previously selected toolset. Learn what testing software and tool vendor your team should select to have the right offerings and qualities.


When first approaching the acquisition of testing software and tools, test organizations often don't consider that defining the overall test architecture plays a key role in successful choices. Lack of an overall test architecture can lead to product choices that may be effective in the short-term but lead to additional long-term costs or even replacement of a previously selected toolset.

If an architectural framework has been defined, then the architectural needs should already be clearly understood and documented. If not, then a general set of architectural guidelines can be applied. The testing software and tool vendor your team selects should have all of the offerings and qualities below:

  • A record of integrating successfully with other testing software vendors
  • A history of operational success in the appropriate environments
  • An established accessible end-user community
  • Ability to support enterprise wide collaboration
  • Ability to support customization
  • Ability to support several (one-to-n) simultaneous test engagements / projects
  • A well-designed, friendly, and intuitive user interface
  • A smooth migration / upgrade path
  • A rich online-help facility and effective training mechanisms

The general architectural requirements for any tool will include more objectives than those listed above but it is important to note that any architectural objective should be applied across the entire toolset.

Figure 3 – Testing Center of Excellence – Test Architecture

Instrumentation and tooling

Once a TCoE reaches a certain size, level of maturity, or workload the requirement to purchase / build testing software or aides becomes apparent. There are several classes of testing tools available today that make the testing process easier, more effective, and more productive.

Choosing the appropriate tool to meet the TCoE long-term and short-term goals can be a challenging and frustrating process. Applying a common-sense approach to software acquisition and implementation will lead to a successful implementation of the appropriate tool and a real return on investment (ROI).

One of the simplest questions to ask when looking at testing software is, "What is ROI?" The simplest answer is "Anything that reduces the resource (people) hours required to accomplish any given task". Tools should be brought into an organization to improve the efficiency of a proven testing process; the value of the actual process has already been established within the organization or within the industry.

There are several classes of tools that are going to be an asset to any TCoE. The following discussion presents an ideal current state, in terms of being ready to move forward with tooling and instrumentation and an overarching set of requirements for each class of tool. It should be noted that these tools have certainly proven their value over the last few years, to the point that the ideal state should no longer be required before considering the acquisition of these tools; but these are only tools not a "silver bullet" that address poor testing processes.

Toolset - Test Asset Management

The TCoE has meticulously tracked Test Requirements, Test Cases, Test Execution, and Defects using spreadsheets, traceability matrix or some other ad-hoc toolset; but is finding this to be a cumbersome process as the test organization grows. It has been shown that this process has reduced the number of defects reaching the field but the cost of maintaining the approach is now impacting its effectiveness. Solution – invest in a test asset management tool or suite of tools.

To support the needs of a TCoE, the Test Asset Management system must enable end-users to author and maintain test requirements, test cases, test results, and test defects over consecutive engagements in a web-enabled, shareable environment - furthermore the test asset management tool shall support test asset management "best practices." Minimum acceptable ROI is 4 hours saved for every hour currently invested.

Toolset - Test Automation

The TCoE has created a suite of manual test cases using a text editor but is finding it difficult to maintain, use, and execute these test cases. The test cases have proven effective in detecting defects before they reach production but the time required to manage and execute these test cases is now impacting the return on investment. Solution – invest in a test automation tool or suite of tools.

To support the needs of the TCoE, the Test Automaton solution shall enable end-users to author, maintain, and execute automated test cases in a web-enabled, shareable environment. Furthermore the test automation tool shall support test case design, test automation, and test execution "best practices. Minimum acceptable ROI is 5 hours saved for every hour currently invested.

Toolset - Test Defect Management

The TCoE has implemented a defect tracking process using e-mail and a relational database but is now finding that defects are being duplicated and mishandled as the volume of defects grows. The solution? Upgrade the current in-house solution or invest in a defect management tool.

To support the needs of the TCoE the Defect Management tool shall enable end-users to author, maintain, and track/ search defects in a web-enabled, e-mail-enabled, shareable environment. Furthermore the defect management tool shall support authoring, reporting, and tracking "best practices." Minimum acceptable ROI is 4 hours saved for every hour currently invested.

Toolset selection and implementation

Here's where the real action takes place, but don't get in a hurry. Take this process one step at a time.

  • Identify a list of potential software candidates by investigating several obvious sources: Generic Web Search, Quality Assurance and Testing On-line forums, QA and Testing e-magazines, and co-workers.

    If you have been very careful and lucky your first encounter with the Vendors Sales force will occur at this time. This can be a frustrating experience if you are purchasing a relatively small number of licenses or an intimidating one if you are going to be placing an order for a large number of licenses. Being vague as to the eventual number of licenses can put you in the comfortable middle ground.

  • Assess the testing software onsite with a full demo version of the software. The assessment process itself will lead to the TCoE gaining skills in the product space. It is always wise to do one final pass of all candidates once the initial assessment is completed. Each software candidate can now be graded against the needs/ requirements and a final selection made. If you are considering more than one vendor remember to include tool integration as part of your assessment.
  • Implement the selected toolset. TCoE will often invest in testing software but not in the wherewithal to successfully use it – tools as the "silver bullet". Investing hundreds of thousands of dollars in software but not investing capital in onsite training and consulting expertise is a recipe for disaster. Investing in the right mix of training, consulting, and vendor expertise the test organization can avoid much of the disruption any change in brings and quickly gain the benefits that software can provide.
The key to success in building a TCoE is buildling it one brick, or problem solved, at a time. Look for small, manageable targets of opportunity before trying to tackle the larger, more critical issues that are challenging your organization. If you build in incremental steps, you'll build the skills and collateral to be successful before tackling high profile engagements.
About the author: David W Johnson (DJ) is senior test architect with over 22 years of experience in information technology across several industries. He's played key roles in business needs analysis, software design, software development, testing, training, implementation, organizational assessments and support of business solutions. Johnson has also developed specific expertise over the past 12 years on implementing "test ware," including test strategies, test planning, test automation -- functional and performance -- and test management solutions.

Dig Deeper on Topics Archive