PushToTest, an open-source publisher of the TestMaker test automation framework, is now offering cloud testing through partnerships with Amazon, RackSpace and GoGrid. The TestMaker automation platform delivers functional testing, load testing and business service monitoring by repurposing the same single test script.
"The neat thing about cloud testing is it has allowed testing companies to get to entirely new levels," said Frank Cohen, CEO and founder of PushToTest. "Just a year ago if a customer said they had to test 100,000 users, we had to beg Sun and IBM to use equipment. Now we can run user levels never before possible—it's easy for us to do tests of 50,000 to 3 million users and to offer that on-demand."
In addition, he said, utilizing the cloud enables testing from multiple locations around the world. "Say there's a difference in performance in East and West Coast services. You can go to Amazon for the East and RackSpace for the West. There's no up-front capital investment in equipment, so you're not buying big leases for testing capability you may not use, which was a problem for many organizations in the past."
According to an October 2008 report from IDC, customer spending on IT cloud services (business applications, application development/deployment, system infrastructure software, storage and servers) will grow almost threefold by 2012, to $42 billion. IDC, in an IDC Exchange note, said the growth "is being driven by the ease and speed with which users can adopt these offerings, as well as the cloud model's economic benefits (for users and suppliers alike)—which will have even greater resonance in the current economic crisis."
With the cloud support in TestMaker 5.3, the company's TestOnDemand service enables organizations to test in their own environment, in the cloud environment, or both, said Cohen. "The organization pays a low annual fee of less than $10,000 to get started, we train their team on how to build their own tests, and they only pay for what they use when they're running the test," he explained.
According to the company, TestMaker 5.3 introduces specific commands to support automatic cloud testing. For example, identify a cloud testing service like Amazon EC2 in a TestScenario. TestMaker creates the TestNodes in EC2 instances, runs the test, retrieves the results, and takes down the EC2 instances.
Testing in the cloud allows organizations to stage a test that more accurately reflects the level of users expected, particularly for an anticipated surge such as a Super Bowl promotion. The cloud also makes the cost of staging such a test much more affordable, Cohen said.
Tribal DDB, a worldwide digital marketing agency, has been outsourcing its performance load testing to PushToTest for about a year, and has utilized the cloud testing, said Alan Rodriguez, IT manager/hosting manager. Tribal DDB manages hosting for all its client sites.
"We manage consumer data for most clients in terms of online registration and opt-ins," Rodriguez said. "We maintain the infrastructure for a number of large companies like Pepsico." For example, he said, "If there's a promotion running ads during Nascar we see thousands of registrants during that period; if we lose any registrants [the client is] upset with us. The clients expect us to perform."
In addition, he said, "We have cost-conscious clients who demand performance and have high thresholds. You couldn't call any average load testing company and pull that off. It became cost-prohibitive to our clients."
Tribal had been doing some load testing before PushToTest, but to run a test for dealing with, say, 60,000 visitors in a 10-minutes period can be expensive, he said. "To generate that kind of load takes a whole lot of machines and efficient software."
PushToTest, Rodriguez said, "is capable of going to very high transaction levels at low [price] rates. We're not covering the capital expenditure of server infrastructure. Unless someone is using a cloud platform, you can't come close to the same transaction level and cost."
The open source community edition of TestMaker is released under a GPL license; PushToTest charges an annual subscription for the enterprise edition which has additional features. Rodriguez said he has no qualms using an open source product for testing. In fact, he said, "I find the open source community as it relates to the innovation rate much higher with open source testing tools. Big companies buying up smaller companies has been a detriment to the industry; they've managed to slow down innovation."
Also, he said, with TestMaker "you can write a functional test and also use it as a load and monitoring test. There's really no tool out there [that allows you to do that]."
Rodriguez also likes that fact that the people who write the code at PushToTest also conduct the tests when Tribal contracts the service. PushToTest has "added functionality to meet our needs in short time frames; we say we need flash remoting and they create [that capability]. They're using the product people to enhance the product as they're on consulting engagements."
TestMaker 5.3 is being released this month. If organizations are using the open source version of TestMaker already have their own EC2 accounts with Amazon, PushToTest will provide instructions on how to utilize the cloud testing, Cohen said.