How do you feel about open source solutions? Would like to hear your takes on both "community-driven" (Selenium) and "commercially driven" (soapUI).
I'm responsible for the test automation frameworks and I found myself trying to go the "open source route" as much as possible. Are there any potential risks of doing that, besides the obvious lack of "enterprise guaranteed support"?
I think a few keys contribute to the value and success of tools like Selenium and Watij. First, these tools are often developed by more than one engineer or built for more than one project. As the community uses the tool, it becomes more broadly applicable. Secondly, the people building and using these tools are generally top-notch testers and have a good idea what it takes to succeed. Finally, I find these tools rarely force you into a given test methodology.
Commercial software testing tools definitely have their place. For instance, I'm a happy use of WAPT (Web Application Performance Test tool), which is a commercial Web load generation tool. It's flexible, dependable, and above all it gets the job done quickly. There are some commercial test tools, however, that have a huge, steep learning curve. These don't lend themselves to use in short-term projects or where testers aren't technical enough to use them. Some tools also force organizations to organize and carry out their work in a specific process, which may or may not be applicable to the organization or project.
One thing to keep in mind is that, given the choice between an open source solution and a commercial solution, the decision should be made based primarily on the needs of the project. In some cases, the cost of the tool (a six-figure license agreement, for example) needs to play into the decision, but for the most part I recommend teams make decisions based on how well the tool will meet their needs.
Rocky Nook Publishing has released a book called Software Testing Practice: Test Management (Spillner et al), which has a detailed chapter on evaluating test tools. It's a bit more analytical than I generally like to be, but it does provide a solid overview on how to rate tools based on a number of criteria. I recently reviewed this book on my blog, Thoughts on QA and Engineering, if you'd like more information about it.
One last word about open source solutions. I like them because they often seem to be more agile in terms of implementation. They're often developed and used in continuous integration environments, which appeals to me as a fan of agile.
This was first published in March 2009