Open Source: Not your father's testing tools

There are numerous factors to explain why open source testing tools are gaining popularity, from agile's influence on development to increasing vendor tool costs. Read what analysts and expert's are saying about open source tools and how they will change the industry for good or ill.

A confluence of factors is driving accepted use of open source testing tools, say industry experts and professionals in the trenches. While it's easy to point to a tight economy as a key driver pushing organizations to open source, the increasing influence of an agile style of development and delivery, the rise of cloud computing, and the maturity of the open source tools themselves and the speed with which they keep up with new application...

technology are also significant factors.

The adoption of open source in general "has always been initially grass roots, from developers saying this is easier and cool, and pushing it up toward management," said Rob Fan, co-founder and CTO of Sharethrough, a video distribution network for branded online video content in San Francisco. "But as a lot of open source projects are becoming more mature, and as more companies are adopting them by pushing them up, we're seeing see big companies using these open source projects, and they start adopting them from the top down."

Also, as agile methodology becomes more mainstream, and open source continues to mature and respond quickly to new technology, "the two combined are mutually reinforcing," said John Dunham, CEO of Sauce Labs, a San Francisco start-up building its business around the Selenium open source testing framework. "Agile carries some unique workflow characteristics that traditional tools have left underserved to date; and the open source community has been modeling and crafting in a way to respond to agile development practices."

His colleague Jason Huggins, creator of Selenium and CTO of Sauce Labs, said often the tool selection is "developer driven vs. the CIO being wined and dined" by commercial vendors, which is also a side-effect of agile. "Agile teams have been given that green light to make themselves productive and do it with whatever tools they need," he said.

For medium to larger enterprises, a pilot program with open source dev/test tools is often the pathway to adoption, said Frank Cohen, CEO of Campbell, Calif.-based PushToTest, maker of the open source tool TestMaker. One example is Deutsche Bank, Cohen said, which piloted TestMaker and then made the decision to bring it in. "They were using proprietary dev/test tools, and they started the pilot with a designated champion," Cohen said. "They had a small budget to bring in a tool, define requirements and goals, and have an engineer write a proof of concept. It was enough for them to understand that open source test tools are on par with what they're getting from proprietary [tools]. That tends to be the technique for open source to get introduced to a medium to large organization."

Matt Heusser, a technical staff member at Socialtext in Palo Alto, Calif., and an instructor at Calvin College, goes further, saying that commercial test tools are losing their relevance. "The truth is management doesn't care about tools; they want the work done. We're moving from a very detailed software development oriented practice to a software delivery practice."

Still, there's no denying money is a factor. "The bottom line is, cost is the biggest driver [of open source adoption], said Jay Lyman, an open source analyst at the 451 Group. "The economy helped drive this. Open source software is almost taken as a given that it will deliver cost savings, which can be dangerous for your expectations, but mostly we find it's living up to that." However, it's more than just cost savings, Lyman added. Flexibility of open source software is also a top deciding factor for organizations, he said.

"What I hear is that there's no money in the budget for tools, so let us install these open source tools, and that seems to be working in organizations—except for those that have locked down the desktop because of security best practices," Heusser said. "I'm skeptical of best practices. If you have to get the IT department to approve and they need [the tool to have] support, you've got a problem."

Added Cohen, "The easy choice is open source will save the organization's budget, but the technical team knows proprietary test tools have fallen behind in support for the way software is being developed today." He said this is evident in areas such as Ajax and rich Internet application technology. "The open source test tools are marching forward in support," he said.

Some popular tools Sauce Labs' Huggins said the problem with commercial tools in the past is that they "they want to live in their own silos, and dev and test didn't associate. One thing that is breaking down that barrier is source control systems. Development teams are bringing on what they're using for managing and tracking code changes, so a commercial test management system is counterproductive to agile teams. The best thing is you can store source code and tests in the same place, like Subversion," a popular open source version control system. "We're seeing more collaboration and communication between dev and test, which leads to using a common set of tools."

Testers have numerous open source tools to choose from, Cohen said. Popular tools for functional testing of Web applications are Selenium, FlexMonkey, Sahi and Windmill, he said, and "at the programmer level you have JUnit, and at the framework level you have Watir and FIT. Cohen said all these tools can be orchestrated using TestMaker to produce a continuous testing environment.

In Heusser's opinion, the two big Web acceptance testing tools are Selenium and Watir, a family of Ruby libraries. Heusser also said he uses Bugzilla daily.

At Sharethough, Fan said they use Selenium. They are a PHP shop, so they use Simple Test and PHPUnit for unit testing, BuildBot for continuous integration, and Subversion for source control. He said they are trying to shift to GitHub, a newer software repository "everyone is very excited about. Why it's so great is it automatically has the option to post onto the Web so you can have a centralized external repository of your code."

Asked if integration is an issue given the use of all the disparate open source tools, Sauce Labs' Dunham said it's a "small 'i' integration, because the APIs are so well known. The open source contributors are doing the hard work."

Sharethrough's Fan said there's "definitely integration work to do. But I think it's the same thing with any commercial product—there is an integration cost no matter what the salesperson says."

And of course one down side of open source is there is no one to call for support, Heusser said. However, he added, "I worked in a shop using commercial tools and when we called I didn't get a lot of help anyway. For the database area, there's a strong case for using commercial stuff, but for test tools I haven't had a lot of success [with support]. The commercial tools do have better documentation and prettier UIs, so you do need a stronger technical staff [when using open source]."

There is a growing ecosystem of companies building businesses around open source products, such as Sauce Labs and others, so there are more places to turn for support.

Another downside is the "hidden costs," Fan added. "There may be a bug that you're the first to come across; or if you start scaling you may start discovering new features you need. The key is you have to be willing to dedicate to the [open source] project itself, contributing to the project and embracing the community."

In choosing an open source tool, Heusser suggested it's more a "reputation play." Guys like Patrick Lightbody and Jason Huggins [Selenium creators], if they say they can solve something, you can believe it. So you should be asking about the principles and their reputation in the market. Making a list of features and checked boxes is not what I'd recommend."

Cohen believes that in general open source testing "has established itself, and achieved a certain level of maturity." And Fan believes that corporate IT is starting to understand that. "It's that nagging got-to-go-to-the-gym feeling," Fan said. "You know you should get into it. Big companies also realize that using open source projects is a good way to attract talent and really smart engineers."

Dig deeper on Software Testing Tools and Frameworks



Enjoy the benefits of Pro+ membership, learn more and join.

1 comment


Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to: