Evaluate Weigh the pros and cons of technologies, products and projects you are considering.

7 questions to ask before you select software testing tools

Before you select software testing tools, you need to know how to evaluate them. Explore trial versions, research the vendors and assess your organization's needs and capabilities.

Software testing tools speed up the test process in major ways. Automated software testing simulates time-intensive tasks, compares those tasks to what is expected and reports a result. Bug tracking software enables teams to log, search and find the status of issues, as well as track updates and merges. Coverage tools give organizations insight into what code or requirements are exercised by which tests, and what needs more work.

It might be overwhelming to evaluate software testing tools against your organization's needs. With limited budgets, teams might pursue open source options that are likely to be web-based and require installation, support and periodic upgrades. Some open source software testing tools offer commercial support and might involve less training time, but those typically come with a high price tag. Commercial SaaS is a third option that has a low monthly cost, often in a freemium model.

When you evaluate software testing tools, consider the following common problems and solutions -- and ask a few key questions -- before you write the purchase order.

How will a testing tool help solve your problems?

When people say "We are going to start using Bugzilla," what they really mean is "We have a bug tracking problem, and we've picked a tool to solve it." Often, the problem itself is fuzzy. It's impossible to evaluate software testing tools to see if they will specifically solve vague or insufficiently understood problems.

Most software testing tools don't help you eliminate problems; in reality, tools create different problems. Driving a car from point A to point B is faster than walking, for example, but it also contributes to pollution which is a different problem. The trick, to paraphrase renowned software developer and writer Eric Sink, is to trade the problems you have for the problems you want.

Select your testing tool to solve a specific problem. Don't select a tool and then figure out how it might fit your team.

Where will the software be deployed, and who are the testers?

Some software testing tools reside on devices that employees will use to test an application. Some, like crash reports, might even be used by customers. For desktop apps, this means IT must install client software and grant the user permission to conduct the install. In most cases, users download an install file they can execute themselves; commercial software typically requires a license key and a credit card.

Consider who will use the testing software, where and how it will be installed, and ensure tool selection fits the user base. Some automation tools are designed for programmers and are embedded in the IDE; testers-turned-automators, on the other hand, may find the IDE intimidating. If testers lack required administrative privileges, that will be a problem. If the software under test is web-based, you might want a test tool that runs on the internet.

Is the testing tool compatible with existing OSes and apps?

Testing software vendors typically claim their tools support functional, GUI, test management and test automation on all platforms. Buyer beware. The claim of "all platforms" suggests a tool tries to be everything for everyone.

In practice, we've found one such tool didn't support Java running in the browser -- it could not recognize the objects -- while another ran only on Windows. A third tool required the team to set up a web server. Yet another tool had incredibly good support for the current browser … approximately four months after each major version's release.

Do some digging to find out what the tools are designed to support. For example, many test automation tools must run as part of a build or continuous integration server. Getting a Windows tool to work with a Linux-based build server or a Unix tool to function when the build server is Azure DevOps Server can be problematic.

When evaluating software testing tools, see if they line up with your current app and test how the new process would work. At the same time, design tests not only for the platforms supported today, but for those the team and product development group plan to support over the next year or two.

How well does the tool integrate with your platform and workflow?

As you evaluate a testing tool, consider the platform and tools already in use to increase the likelihood of smoother integration, easier adoption and better test results. A QA team that uses Jira will likely succeed when they use tools built to support Jira. The same is true for Visual Studio users who use tools designed to fit into the Microsoft platform. This is also the case with Java, Oracle and most ERP systems.

When you select tools that align with your overall application platforms, you'll enjoy a more seamless workflow. If you don't, you might end up with yet another silo of information that is essentially duplicated on the way in and forgotten.

Additionally, different teams often rely on separate version control systems. Preserving code and automation across version control systems will likely cause problems. Some test tools embed directly into the IDE and offer plugins to extend their capabilities. Test management applications can integrate with bug tracking tools and code coverage tools. Research and understand the capabilities of all tools before integrating them with your platforms.

This advice also goes back to knowing who the user is. Programmers live in the IDE; the closer you can get to their native environment, the more likely it is they will use the tool. That is a critical step. When failures happen because of changes programmers have made, you want them to approve tests.

Does the tool require training or knowledge of special languages?

Some software testing tools have proprietary languages and syntax. Most record and playback tools require some amount of training. If programmers can use the tool in their native, or preferred, language -- Ruby, Python, .NET and Java are popular for tools -- training might not be required.

Then again, testers or analysts who don't know the language might use the tools. A proprietary language requirement to use a tool might limit its ability to be integrated with other tools. Using a readily available and well-known language makes it possible to export scripts to other tools.

How do open source versus commercial tools compare?

By the time you determine the problem to be solved, you should have identified a few tools to help achieve that goal. Open source software testing tools often require no upfront costs, unlike proprietary tools, but they might not meet the needs of the organization.

Be sure to answer three main questions.

  • How effective can the team be with open source software testing tools?
  • Are open source tools a good fit?
  • How will the team get support and training?

Open source application testing tools can work exceptionally well on the platforms and technologies for which they were designed. Change the underlying database, OS or programming language, though, and the time invested in the tools might eat up the budget you saved by going open source in the first place. Some organizations require customizations that go beyond what the open source project can provide. In many cases, you will have to support yourself when it comes to installation, patches and maintenance for these open source tools.

Commercial tools have similar problems, but vendors have a strong motivation to make things work, as well as to provide support and training. If the product has a list of supported platforms, ask the vendor how they support new browsers and OSes as they are released.

Should you try before you buy?

Many proprietary commercial tools offer trial or community versions to download and run for free. Other web-based tools enable you to create and use an account for free for 30, 60 or 90 days. Often, these trial versions have limited features or cannot be saved for repeat use, but a programmer and tester can experiment to see if the tool's features or flexibility meet their organization's needs.

Give an employee permission to try the application for two weeks to reveal any problems with the install. If the software must be bundled with end-user software, such as a crash reporter, get a trial and walk through the entire process with a beta user to see if any problems pop up. For example, the tool might falsely identify as spyware when the software "phones home" with usage statistics.

Most open source software testing tools are available for free if you download them directly and install them yourself. When in doubt, demo the software and run it through its paces. It is too easy to push through a proof of concept, spend a large sum of money and then find out the "solvable problems" with the proof of concept are either difficult or impossible to fix.

Don't invest a large amount of time until you are sure the product will be fit for use. Will a trial version suffice, at least temporarily? If so, determine how long it will take to get purchase approval to prevent lost productivity when the trial expires.

Next Steps

The future of testing is here. You need to embrace the change and stay the course.

Code refactoring tools are abundant. Which one is right for you?

Going beyond testing tools, there are many testing service providers from which to choose

 

Dig Deeper on Automated and autonomous testing

Join the conversation

6 comments

Send me notifications when other members comment.

Please create a username to comment.

What do you consider the most important feature of application testing tools?
Cancel
I would make sure that the tools will actually work for the purpose that they are intended for (surprisingly often this is not well thought out!), and that the actual users of the tool are on board and believe that the tool will add value to the testing process.
Cancel
Flexibility and compatibility would be the most important features for me. We have a lot of difficulty finding testing tools that are compatible with our many applications and environments. If they are not compatible out of the box, the tools should be customizable so that the users can at least try to build add-ons to make the tool work for their purposes.
Cancel
A few months ago I finished a rather long test tool evaluation process for a large organization. I learned that tool evaluation is much more difficult and time consuming than I had anticipated and that the context of the organization are hugely important. I'd like to share my experience and the two key mistakes that I made. Both boiled down to misunderstanding the context of the client:
1. Making wrong assumptions about the business value that the features of the tools were supposed to provide. As a tester, I focused on testing functionality, when in fact traceability and status reporting were more important for the client. This insight radically changed how I evaluated test management software.

2. Making wrong assumptions about who the users of the tools were going to be. As a tester, I assumed that the users would be testers. It later turned out that most employees of the organization were expected to take part in testing to a degree. This had huge implications, since many test tools have a per-user pricing model and some are horribly complex for the casual user.

Thankfully I was able to fix both issues through communication, joint thought experiments and good feedback from the client. The article touched on most of the problems I faced, and it would have been a tremendous help had it been published 4 months earlier :)

It would be great to hear a few more experience stories!
Cancel
Hi, Thanks for sharing a nice post!! It is an imperative to identify a right tool for the testing needs. Failure in picking a right tool might lead to negative consequences. Let's look at some of the parameters that need to consider when buying the latest application testing tools.
• If the tool/framework suits the requirement.
• If the tool/framework can be used readily.
• Do we have the right resources who are capable of using the tool/framework?
• Can we use existing resources to work with the tool/framework?
• Initial time investment required to implement the tool or framework.
• Training time required for resources on the tool/framework.
• Turnaround time of testing using the tool.
• ROI of the tool.
Let’s learn about a solution/tool which has the power to meet all the testing needs.
Cancel
What a close topic. I used to write about that extensively :)

On automation requirements:
  • http://automation-beyond.com/2014/06/24/automation-requirements-revisited-2014/

On evaluation heuristics: TERMS / CRUMBS (Hi, Michael Larsen! :))
  • http://automation-beyond.com/2012/06/22/coming-to-terms-with-test-automation/
  • http://automation-beyond.com/2012/07/14/follow-the-crumbs/

Cancel

SearchCloudComputing

SearchAppArchitecture

SearchITOperations

TheServerSide.com

SearchAWS

Close