Buyer's Guide

Browse Sections


This content is part of the Buyer's Guide: Choosing the best software testing tools for your business
Evaluate Weigh the pros and cons of technologies, products and projects you are considering.

What to look for when buying application testing tools

Before purchasing application testing tools, you need to know how to evaluate them. Trial versions, vendor research and assessing your organization's needs may help.

Application testing tools can speed up the test process in major ways. Automated software testing simulates time-intensive tasks, compares those tasks to what is expected and reports a result. Bug tracking software lets teams log, track, search and find the status of issues while also tracking updates and merges. Coverage tools let organizations see what code or requirements are exercised by which tests -- and what needs more work.

Trying to evaluate which tools are appropriate for an organization can be overwhelming. With limited budgets, teams may be tempted to pursue open source alternatives, which may require training, skill or time investments. There may be commercial support for an open source software testing tool, but it often comes at a price. Commercial software might have support or less training time, but it typically has a high price tag, creating budget concerns.

The first step toward performing an evaluation of software testing tools is to recognize where the tools fit into the organization. For any new tool, here are some common problems and solutions to consider and a few key questions to ask before you write the purchase order.

Try before you buy or deploy

Many proprietary commercial tools have trial or community versions that you can download and run for free. Other tools run through a website and may let you create and use an account for 30, 60 or 90 days before requiring payment. Often, these versions have limited features or cannot be saved for repeat use, but they do let the programmer and tester experiment to see if the tool will have the features or the flexibility to meet their organizations' needs.

Give an employee permission to try the application for two weeks. That will help you find out if there are any problems with the install. If the software needs to be bundled with end-user software, such as a crash reporter, get a trial and walk through the entire process with a beta user to see if there are any problems with, say, false identification as a spyware when the software "phones home" with usage statistics. Most open source application testing tools are available for free if you download them directly and install them yourself. When in doubt, demo the software and run it through its paces. Do not commit to spending a large sum of money or investing a large amount of time until you are sure the product will be fit for use.

Which software is deployed on end-user devices?

Some software testing tools will reside on devices employees will use to test the application; some, like crash reports, might even be used by the end customer. For desktop apps, this means client software needs to be installed, and the user needs permission to conduct the install. In most cases, users can download an install file that they can execute themselves; commercial software typically requires a license key, which generally requires a credit card. When buying testing software, consider who is using the test software, where and how they will be installing it, if trials will do for now, and how long approval will take to avoid losing productivity when the trials expire.

Are the tools compatible with your existing OSes and apps?

A typical claim from testing software vendors might be that their tool supports functional, GUI, test management and test automation on all platforms -- but buyer, beware. The claim "all platforms" suggests the tool is trying to be everything for everyone. In practice, when we've tried to use such a tool, we found that one didn't support Java running in the browser -- it could not recognize the objects -- while another ran only on Windows. A third tool required the team to set up a web server. A fourth tool had incredibly good support for the current browser -- about four months after each browser was released.

We had to dig to find out what the tool was really designed to support. For example, many automation software testing tools need to be run as part of a build or continuous integration server. Getting a Windows tool to work if your build server is based on Linux can be a serious challenge, as can getting a Unix tool to work when the build server is Microsoft Team Foundation Server.

When deciding which application testing tools to purchase, see if they line up with your current product; test mix and plot out where your team and product development would like to go over the next year or two. Don't design tests only for the platforms you support today; also design tests for platforms you plan to support in the future.

Can the software readily integrate with your workflow?

When working with different teams, it's not uncommon for two version control systems to be in use. Keeping code and automation in two different version control systems is likely to cause problems. Some tests' tools embed directly into the programming integrated development environment and offer plug-ins to extend their capabilities. Test management applications can integrate with bug tracking tools and with code coverage tools. Research and see which tools are capable of integration and to what level.

What problem do you want to solve?

Each style test tool -- test documentation systems, UI-level automation, automated mobile testing tools, API tooling, and coverage and monitoring tools -- are designed to solve specific problems. Tools that work at the UI level can be useful for mass inspection and repetitive testing that a person would normally perform. API testing tools help test data and workflows before a UI even exists. Coverage and monitoring help your team better understand the testing they are doing now. Select your tool to solve a problem, rather than selecting a tool and figuring out how it might fit your team.

Is training or prior knowledge of special languages needed?

Some software application testing tools have proprietary languages and syntax; most record and playback tools require some amount of training. If the tool can be used by programmers in their native language -- Ruby, Python, .NET and Java are popular for tools -- then it might not require training.

Then again, if the testers or other analysts are to use the tools, they might not know the language. Having to use a proprietary language may limit the ability to integrate with other tools. Using a readily available and well-known language makes it possible to export scripts to be deployed or ported to other tools.

The pros and cons of open source versus commercial tools

By the time you have determined the problem you want to solve, you also will have identified a few tools to help achieve your goal. Some of these application testing tools are open source and require no upfront costs. Some are proprietary and have a cash cost. Be sure to answer three main questions: How effective can the team be with open source software testing tools? Are the open source tools a good fit? And how will the team get support and training?

Open source application testing tools can work exceptionally well, albeit on the platforms and technologies for which they were designed. Change the underlying database, operating system or programming language, though, and the time invested in the tools may end up eating at the budget you saved by going open source in the first place. Some organizations require customizations or changes that go beyond what the open source project can provide. In many cases of open source software, you will have to support yourself when it comes to installing, updating and maintaining these tools.

Commercial tools have similar problems, but vendors have a strong motivation to make things work, as well as to provide support and training. If the product has a list of supported platforms, ask the vendor how they support new browsers and operating systems when they are released.

Next Steps

The future of testing is here. You need to embrace the change and stay the course.

Code refactoring tools are abundant. Which one is right for you?

Going beyond testing tools, there are many testing service providers from which to choose


Dig Deeper on Software testing tools and techniques

Join the conversation


Send me notifications when other members comment.

Please create a username to comment.

What do you consider the most important feature of application testing tools?
I would make sure that the tools will actually work for the purpose that they are intended for (surprisingly often this is not well thought out!), and that the actual users of the tool are on board and believe that the tool will add value to the testing process.
Flexibility and compatibility would be the most important features for me. We have a lot of difficulty finding testing tools that are compatible with our many applications and environments. If they are not compatible out of the box, the tools should be customizable so that the users can at least try to build add-ons to make the tool work for their purposes.
A few months ago I finished a rather long test tool evaluation process for a large organization. I learned that tool evaluation is much more difficult and time consuming than I had anticipated and that the context of the organization are hugely important. I'd like to share my experience and the two key mistakes that I made. Both boiled down to misunderstanding the context of the client:
1. Making wrong assumptions about the business value that the features of the tools were supposed to provide. As a tester, I focused on testing functionality, when in fact traceability and status reporting were more important for the client. This insight radically changed how I evaluated test management software.

2. Making wrong assumptions about who the users of the tools were going to be. As a tester, I assumed that the users would be testers. It later turned out that most employees of the organization were expected to take part in testing to a degree. This had huge implications, since many test tools have a per-user pricing model and some are horribly complex for the casual user.

Thankfully I was able to fix both issues through communication, joint thought experiments and good feedback from the client. The article touched on most of the problems I faced, and it would have been a tremendous help had it been published 4 months earlier :)

It would be great to hear a few more experience stories!
Hi, Thanks for sharing a nice post!! It is an imperative to identify a right tool for the testing needs. Failure in picking a right tool might lead to negative consequences. Let's look at some of the parameters that need to consider when buying the latest application testing tools.
• If the tool/framework suits the requirement.
• If the tool/framework can be used readily.
• Do we have the right resources who are capable of using the tool/framework?
• Can we use existing resources to work with the tool/framework?
• Initial time investment required to implement the tool or framework.
• Training time required for resources on the tool/framework.
• Turnaround time of testing using the tool.
• ROI of the tool.
Let’s learn about a solution/tool which has the power to meet all the testing needs.
What a close topic. I used to write about that extensively :)

On automation requirements:

On evaluation heuristics: TERMS / CRUMBS (Hi, Michael Larsen! :))