News Stay informed about the latest enterprise technology news and product updates.

On-demand software testing service pays off for three startups, and Second Rotation turned to uTest to help test their software and say the on-demand service found bugs they otherwise would have had trouble locating.

Exposing your applications to the globe via the Internet can be downright scary, but exposing your applications to a global community of testers on-demand can be downright effective, according to three companies that recently completed pilots with uTest.

uTest gives us the opportunity to be exposed to a variety of people. It's important to have people around world give feedback--it maximizes the ability to find unique bugs.
Sagi Richberg

Founded in 2007, Ashland, Mass.-based uTest offers a hosted infrastructure for QA and usability testing with a pay-per-bug business model. uTest provides a global community of testers, dubbed "uTesters," leveraging a social networking-type model. As of May, uTest had signed up more than 7,000 testers, from a range of countries and with a range of experience levels.

The diversity of uTest's community is a key benefit, according to Anson Kuo, vice president of business development for Scottsdale, Ariz.-based, an international Web portal for
people sharing their interests and favorite things on the Web.

"Our QA is based in Scottsdale, so we only get a one-dimensional perspective," he said. "uTest has testers all over world. The bugs they found were complications from different platforms, different operating systems, different locations, different firewalls. They provide a more thorough testing perspective."

PerfSpot, which launched near the end of 2006, just hit over 17 million members, Kuo said. The uTest pilot tested the live site, primarily the technical and functional requirements of the registration process.

"They found a few technical bugs, and we were very happy they were able to do that. These were issues that stopped the site from working," he said. "To us, that's golden. They were also able to come up with some functional bugs or issues that didn't stop the site but were big enough of a deal where users who were Internet-savvy would see it and know it's not quite right."

Finding financial transaction bugs
For Sagi Richberg, CEO of, based in Ashland, Mass., uTest's ability to leverage social networking and provide a global pool of testers offers a diverse perspective.

"It gives us the opportunity to be exposed to a variety of people. It's important to have people around world give feedback -- it maximizes the ability to find unique bugs," he said. "We were impressed with people finding what we didn't think about."

Founded in 2007, GroupGain enables buyers to collaborate to drive down a product's price through economy of scale. For example, a GroupGain feature called BuyerPower lets the buyer form a group around a product, then sellers can bid for the group's business.

uTest's software testing and QA service
Through uTest, companies can get on-demand QA and software testing services from a global community of testers.
>> Read more about uTest's services

uTest tested the GroupGain platform for usability and webflow, Richberg said. The service tested for "bugs relating to financial transactions, which are most important to us, including credit card transactions, and they were testing a lot of internal things that had to do with intercommunication like chat, email, phone." Richberg said the back-office system that manages sellers, buyers, deal authorization, and the BuyerPower mechanism was also tested.

Previously, GroupGain did QA in-house, "semi-successfully," Richberg said. "We had just one internal guy, so just one way of thinking. I think the diversity [of the uTest community] really did a great job."

In addition to QA, GroupGain had uTest do some usability testing. "We were looking for a diversity of people, from China, etc. We had 20 people give us great remarks worth paying for," Richberg said.

GroupGain hopes to go live by the end of June.

"They found a lot of bugs, and we had to deal with them. Some of them were major. You can't find every variation [on your own.] There is a range of different browsers, different patches to different widgets and DLLs in each operating system. One of the greatest things was finding a major security issue that had to do with SQL injection. That would've been a great security compromise, so by itself that was worth it to us."

A complement to internal QA resources
For Waltham, Mass.-based Second Rotation, uTest helped extend and complement internal resources, according to James McElhiney, chief technology officer. Second Rotation, founded in 2006, buys and sells previously owned consumer electronics and gadgets, and it recycles e-waste if the item has no market value.

The startup is an agile development shop, updating the site every two weeks with new releases, McElhiney explained. Previous to the uTest pilot, Second Rotation had one QA person, then they'd "try to grab anyone from the company to do more testing. That's not really their job and don't know how to be antagonistic, so we didn't get the best results. Everyone is super busy, so they have several things they're working on. It's hard to say 'test this application for couple hours.' People would help, but sometimes we'd release with bugs and find out later through customer support."

McElhiney continued, "That's where the value of uTest is — these people are professional testers, and they know how to be antagonistic. So it really complements the resources we have. And the nice thing about their model is you only have to pay for things they find." Also, he added, the uTest model enables the company to "get tons of people when you need them [to test a release], and when you don't need them you don't have to pay. It's hard to get that infrastructure ourselves."

While all three companies are startups, they said that uTest was something they envisioned that could scale with them.

"I truly see it as a virtual QA team," Richberg said. McElhiney agrees, with this caveat: "We might need more people to manage the [uTest] relationship, but we can be more cost-effective using the uTest model."

All three companies are now also clients. For other organizations considering the uTest Service, Richberg offered this advice: "It's important to define exactly what they want. We specifically said what we wanted them to test. For example, we said we don't want them to tell us about mistakes in spelling."

McElhiney echoed that advice. "Try to make sure you have test plans in place. It is a bit of work to monitor the bugs as they come in. We have an internal bug tracking system, so we have to have a sync-up mechanism. Also, not every bug is an actual bug, so have an upfront list of known defects; otherwise you end up having to pay for those. These might be little things you decided not to fix yet, but make sure the uTest tester doesn't report that."

He added: "It helped us make our release better; it was very cost-effective. It's a relatively low-cost barrier to entry to give it a try. I absolutely think it's worth it."

Kuo agrees. He also praised the customer service. "My advice is to try it out."

Dig Deeper on Software Testing Tools and Frameworks

Join the conversation


Send me notifications when other members comment.

Please create a username to comment.

I do agree that diversity on teams, testing included is a great asset to software development.  I wonder though, why it is companies like needed to use uTest to come to this realization?  So many companies have bad ratios of devs to testers, so I wonder, why is hiring more testers such a hard thing I wonder?
Hi Veretax, I would tend to agree that too many companies are short staffed when it comes to testers. I don't think it's just testers that it's tough for companies to hire, though. I think hiring is just something that a lot of companies are bad at.
Crowdsourcing can be a benefit in that it gives an organization a short window to test assumptions and see if they have asked questions they may not have thought to ask. That's one of the benefits of numbers. My concern with too much uTest (or any other) test coverage is that an organization will lose sight of the value of someone local to challenge and question requirements. truth be told, that is not a function that crowdsourced QA, in any format, will provide.
I have tested on projects for uTest and I can definitely see how it can provide value to a company needing to test their software product. It works especially well for websites and mobile apps. The project testers always hit the application hard, and generate a large number of bugs, ranging in severity from small typos to security issues, in a short amount of time. 

It doesn't replace the need to have full-time QA working with the development team, though. The uTesters' goal is only to find existing bugs, but a QA's role on a dev team encompasses much more than that.
That's a good point, ABuell. Testing services like uTest are great for scaling up and out - especially out - but it's not going to replace your core quality folks.