A particular tool might look great when used on a few tests, but is it workable on thousands of tests? Scalability can be seen as the ability to handle increasing volumes of tests, or to handle continuing maintenance of tests.
You may have the need to test in a variety of environments. Can the tool be ported and used in those test environments?
Depending on the kinds of people expected to use the tool and their level of technical skills, ease of use factors are often an important consideration. It is a big advantage to be able to learn and start using a tool quickly. Steeper learning curves often mean higher frustration levels and sometimes, tool abandonment.
If training is needed, is it available and how much does it cost? Also, are there restrictions by the vendor that prohibit you from conducting your own training on the product?
One of the big considerations in test automation is how to keep the tests current. This has been true since people have used test automation tools, but in agile projects, change is rapid and constant. Therefore, the automated tests should be as easily maintainable as possible.
At some point in your efforts to implement, learn and apply the tool, you will have questions. If you are considering a commercial tool, how well does the vendor support the tool? What are the known bugs? Two things I always do are a) Call the support phone number (or send an e-mail) and see how I am treated, and b) Check the vendor's support forum to see what the known bugs are and how they are being addressed.
Even with Open Source software you must consider support. I use several Open Source tools and find that the community support is faster and more helpful than that provided by commercial tool vendors.
This doesn't necessarily mean free or cheap, but simply understanding your budget resources and constraints. Don't forget to think about the cost of increasing tool licenses for future projects, if needed.
- Technical Fit
Does the tool under consideration work well with the environments and applications you will be testing? Does it integrate with other test tools you are currently using?
Other helpful tips
Use a scorecard
I like to use a scorecard approach with all my tool candidates at the top in separate columns. In the leftmost column, I list my criteria. These criteria can be rather broad as mentioned above, or they may be very specific requirements for the tool – for example, "The tool must support Linux, Windows XP, Vista and Windows 7 environments." Then, I assign a weight from one to five for each of the criteria.
When I evaluate a tool, I give it a score ranging from zero to five, with zero being "no capability" to five meaning, "exceeds requirements". Then I multiple each criteria score by the weighting factor and add all the scores for each tool to get a final score.
This is simply one way to somewhat quantify each tool's fit to my requirements. There are other factors to consider, such as vendor favorability (some companies greatly favor certain vendors and prohibit other vendors), and Open Source policies (some companies will not even consider using free or open source tools).
Evaluation vs. proof-of concept
Regardless of the tool price, you need to perform either an evaluation or proof-of-concept (POC). I prefer the POC because the result is an early deliverable. Evaluations tend to be more superficial and initial views of the tool's features. The POC lets you determine a tool's capabilities and constraints as shown in your applications and environment. It's common for a tool to look good in the evaluation, but upon further use prove to be unworkable when a major constraint is discovered.
Free and cheap tools have less monetary risk, but there is still an investment in time and money to learn and apply the tool. Let's say your company spends $50,000 for licenses of a test automation tool only to learn that it is unworkable with a critical application. If it takes three months of project time to discover the issue, not only is the initial license cost at risk, but three months of time invested in using the tool. Compare this same situation with a free or cheap tool, and you still have the three months of lost time.
Do your homework and at least define your needs before approaching the tool marketplace. Once you try this with a couple of tools, you will be more agile in finding and incorporating tools into your agile projects.