Application testing tools can speed up the test process in major ways. Automated software testing simulates time-intensive tasks, compares those tasks to what is expected and reports a result. Bug tracking software lets teams log, track, search and find the status of issues while also tracking updates and merges. Coverage tools let organizations see what code or requirements are exercised by which tests -- and what needs more work.
Trying to evaluate which tools are appropriate for an organization can be overwhelming. With limited budgets, teams may be tempted to pursue open source alternatives, which may require training, skill or time investments. There may be commercial support for an open source software testing tool, but it often comes at a price. Commercial software might have support or less training time, but it typically has a high price tag, creating budget concerns.
The first step toward performing an evaluation of software testing tools is to recognize where the tools fit into the organization. For any new tool, here are some common problems and solutions to consider and a few key questions to ask before you write the purchase order.
Try before you buy or deploy
Many proprietary commercial tools have trial or community versions that you can download and run for free. Other tools run through a website and may let you create and use an account for 30, 60 or 90 days before requiring payment. Often, these versions have limited features or cannot be saved for repeat use, but they do let the programmer and tester experiment to see if the tool will have the features or the flexibility to meet their organizations' needs.
Give an employee permission to try the application for two weeks. That will help you find out if there are any problems with the install. If the software needs to be bundled with end-user software, such as a crash reporter, get a trial and walk through the entire process with a beta user to see if there are any problems with, say, false identification as a spyware when the software "phones home" with usage statistics. Most open source application testing tools are available for free if you download them directly and install them yourself. When in doubt, demo the software and run it through its paces. Do not commit to spending a large sum of money or investing a large amount of time until you are sure the product will be fit for use.
Which software is deployed on end-user devices?
Some software testing tools will reside on devices employees will use to test the application; some, like crash reports, might even be used by the end customer. For desktop apps, this means client software needs to be installed, and the user needs permission to conduct the install. In most cases, users can download an install file that they can execute themselves; commercial software typically requires a license key, which generally requires a credit card. When buying testing software, consider who is using the test software, where and how they will be installing it, if trials will do for now, and how long approval will take to avoid losing productivity when the trials expire.
Are the tools compatible with your existing OSes and apps?
A typical claim from testing software vendors might be that their tool supports functional, GUI, test management and test automation on all platforms -- but buyer, beware. The claim "all platforms" suggests the tool is trying to be everything for everyone. In practice, when we've tried to use such a tool, we found that one didn't support Java running in the browser -- it could not recognize the objects -- while another ran only on Windows. A third tool required the team to set up a web server. A fourth tool had incredibly good support for the current browser -- about four months after each browser was released.
We had to dig to find out what the tool was really designed to support. For example, many automation software testing tools need to be run as part of a build or continuous integration server. Getting a Windows tool to work if your build server is based on Linux can be a serious challenge, as can getting a Unix tool to work when the build server is Microsoft Team Foundation Server.
When deciding which application testing tools to purchase, see if they line up with your current product; test mix and plot out where your team and product development would like to go over the next year or two. Don't design tests only for the platforms you support today; also design tests for platforms you plan to support in the future.
Can the software readily integrate with your workflow?
When working with different teams, it's not uncommon for two version control systems to be in use. Keeping code and automation in two different version control systems is likely to cause problems. Some tests' tools embed directly into the programming integrated development environment and offer plug-ins to extend their capabilities. Test management applications can integrate with bug tracking tools and with code coverage tools. Research and see which tools are capable of integration and to what level.
What problem do you want to solve?
Each style test tool -- test documentation systems, UI-level automation, automated mobile testing tools, API tooling, and coverage and monitoring tools -- are designed to solve specific problems. Tools that work at the UI level can be useful for mass inspection and repetitive testing that a person would normally perform. API testing tools help test data and workflows before a UI even exists. Coverage and monitoring help your team better understand the testing they are doing now. Select your tool to solve a problem, rather than selecting a tool and figuring out how it might fit your team.
Is training or prior knowledge of special languages needed?
Some software application testing tools have proprietary languages and syntax; most record and playback tools require some amount of training. If the tool can be used by programmers in their native language -- Ruby, Python, .NET and Java are popular for tools -- then it might not require training.
Then again, if the testers or other analysts are to use the tools, they might not know the language. Having to use a proprietary language may limit the ability to integrate with other tools. Using a readily available and well-known language makes it possible to export scripts to be deployed or ported to other tools.
The pros and cons of open source versus commercial tools
By the time you have determined the problem you want to solve, you also will have identified a few tools to help achieve your goal. Some of these application testing tools are open source and require no upfront costs. Some are proprietary and have a cash cost. Be sure to answer three main questions: How effective can the team be with open source software testing tools? Are the open source tools a good fit? And how will the team get support and training?
Open source application testing tools can work exceptionally well, albeit on the platforms and technologies for which they were designed. Change the underlying database, operating system or programming language, though, and the time invested in the tools may end up eating at the budget you saved by going open source in the first place. Some organizations require customizations or changes that go beyond what the open source project can provide. In many cases of open source software, you will have to support yourself when it comes to installing, updating and maintaining these tools.
Commercial tools have similar problems, but vendors have a strong motivation to make things work, as well as to provide support and training. If the product has a list of supported platforms, ask the vendor how they support new browsers and operating systems when they are released.
The future of testing is here. You need to embrace the change and stay the course.
Code refactoring tools are abundant. Which one is right for you?
Going beyond testing tools, there are many testing service providers from which to choose