Web 2.0 technology and the pressure to shorten application release cycles and speed time to market are moving organizations toward testing automation. Today, with the popularity of open source tools and standard vendor offerings, organizations have many choices. In this article, software testers talk about how they decide which type of automated testing tool, or combination of open source and proprietary, best meets their needs.
"We're moving into new territory, but there's still a lot of traditional mentality over testing and automation." So says Erik Marlin, lead business consultant at Perficient Inc., a multinational IT consulting firm based in St. Louis. With increased use of Web 2.0 technologies, "we're trying to figure out how to best test the crux of the matter; we're trying to move toward automation."
Indeed, with new applications built with Web 2.0 technology, "there's so much more going on from a user perspective, with all the different asynchronous parts of a page that can be different moving pieces, versus the old days when you had a page that could do a couple things dependent on a user clicking button," said Kendall Joseph, QA coordinator at Emma, which provides a Web-based service for self-serve email marketing. "It's more dynamic now, so it's harder testing. There are more things to test, more moving parts."
Joseph said Emma's team did a lot of manual testing in the early days of the company, which was founded in 2002. Now, he said, "The desire is to automate as much as possible to make [testing] more efficient, rather than continue to hire testers."
Both Joseph and Marlin turned to open source testing tools.
For Marlin, who was working on an engagement for FordDirect.com, the application being developed "had parts that needed to be tested that were almost exclusively Flash and Ajax driven with dynamic content. When we looked at the traditional big names we didn't feel we could get the coverage we needed on those pieces as they existed in late 2007."
Perficient chose TestMaker, an open source tool from Campbell, Calif.-based PushToTest for the project; because it was open source as well as offered features for Web 2.0 that he felt the other tools at the time didn't. They used PushToTest in combination with Selenium, an open source test framework, to test against multiple browsers for FordDirect.com, he said.
More recently, Joseph's firm implemented PushToTest because they wanted both open source and affordability. He said the commercial tools like those from IBM and HP are "great tools, but pricey" and they were looking for something less expensive.
Also, Emma has an open source development environment, he said, with all new development being done in Python and existing legacy code in PHP.
"We wanted to find something that worked in our [open source development] environment, and our personal machines are Macs. Finding something Mac compatible was quite a challenge. We started searching about this time last year, and we kept coming back to PushToTest. We liked the idea that you could script once and use those scripts as either functional or load tests," Joseph said. >p>"For Web 1.0 applications, organizations have expensive proprietary tools like [HP] QTP and LoadRunner," said Frank Cohen, PushToTest CEO. "For the rich Internet application space you have limited offerings from [commercial] tools vendors and lots from open source."
According to John Dunham, CEO of Sauce Labs, which offers testing tools and services based on Selenium, "The big contrast with legacy proprietary commercial tools is they were built for earlier era when development and QA were more silo'd. The island of automated testing was not integrated with the development environment."
"The big driver of open source is speed, time to market, time to value, and the ability to reach and grow a developer and user community," said Jay Lyman, analyst for open source at the 451 Group. "The open source players don't have the depth of features but they might be able to hone in on features that are hot, like Ruby on Rails, and they're lighter weight, which is a big part of this."
Jason Huggins, creator of the Selenium open source testing framework and CTO of Sauce Labs, said he developed Selenium to test Web applications that were "Ajax-y."
Brian Bryson, IBM Rational Quality Management Solution Manager, said open source tools have helped the automated testing industry. "They were such simple tools that did a great job of the task at hand, and gave a positive feeling that automation can work. We saw that filter into popular development processes—extreme programming, agile. It's great to provide a proof case that automation can accelerate the process."
However, he said the cost benefit of commercial tools is "easily there. One big difference between open and commercial tools is the breadth of solution. We've been building since the mid-90s. We have a huge platform coverage across browsers, plus Flash, Ajax, traditional technology like ActiveX, etc. That's something you can only get from commercial tools."
In addition, Bryson said a more comprehensive offering like IBM's is suited to the evolution of software delivery, which he said is growing more collaborative with business analysts and architects working with development and test. "You can try to make that happen with a bunch of open source tools pieced together, but we've built a platform [Jazz] for communication and collaboration that's much bigger than that. We don't view the problem as simply quality—it's a better software delivery process."
For heterogeneous environments with a mix of testing needs, Mark Sarbiewski, senior director of products for HP Software & Solutions, said "open source hasn't gotten there yet," and the tools are primarily used by developers. His colleague at HP, Subbu Iyer, senior director of products, ALM, agreed. "Developers use them; they're great to run a quick sanity test before handing it over to test/QA."
PushToTest's Cohen doesn't disagree. "The challenge to open source vendors is that most tools expect you've got a development staff that can build these tests. There's a big gap between testers used to HP to do point and click test development and what open source tests tools are able to deliver."
Companies like Cohen's and Sauce Labs that are building commercial offerings around open source testing tools are addressing some of the limitations. Cohen said PushToTest's TestMaker 5.4 introduces a GUI that is more productive to use, version 5.5 will offer improvements in stability and scalability as well as "an enhancement to the Selenium IDE that lets you record from IE and Firefox." Looking ahead, he said, version 5.6 will offer test management.
Recently, Sauce Labs rolled out Sauce IDE, an integrated development environment for Selenium tests that allows users to do automated cross-browser functional testing without the need to code.
Today, however, pricing and ease of use are attractive to Emma's Joseph. "The HP/Mercury tools could've done what we needed, there are a variety of tools that could have, but we liked PushToTest's ease in scripting, ease in building different types of tests, and the support we get from them is important."
Perficient's Marlin said the learning curve for TestMaker was low because his team was "able to code in a comfortable language. You can use the same coding to code test scripts and run them in an automated fashion. As far as executing scripts and pulling reports, [we learned] that in a couple day seminar with PushToTest." He added, "The pricing is more attractive than the major players."
Use of open source testing tools, particularly for rich Internet applications, is paralleling other technology adoption trends that start from the ground up, Cohen said. "The thing is, most software architects are not defining new applications at the UI level, so a developer looks around, goes to a developer conference, and sees these beautiful UIs. They're all open source so he grabs them."
Open source testing tools are proving themselves, which is leading to broader use, said the 451 Group's Lyman. However, he added, open source has limitations, particularly for large ISVs or large corporate IT development groups. "It may not cover all your different initiatives or team needs," he said.