Every time a major mobile OS is upgraded, there are a few more grey hairs on the heads of the application testers....
It's hardly surprising. The pressure to develop applications faster than ever before -- with new code sometimes being released every night -- has made the need to test mission-critical and the mobile app market take off.
But just because it's important doesn't make it easy. In fact, mobile application testing can be just plain overwhelming, with the large number of devices, networks and operating systems, not to mention the many other variables involved with consumer usage. And when it's this complicated, chances are it isn't being done, or done correctly. According to a CapGemini survey last year, a mere 31% of 1,500 organizations surveyed had a formal process in place for testing.
Organizations wrestling with a solution can turn to mobile testing service providers. Companies like Experitest, Perfecto Mobile, Keynote Mobile Testing and Mobile Labs have been around for several years, but demand for their services is growing fast now as the mobile app market itself heats up. IDC research manager Peter Marston predicts mobile testing is going to be an increasingly large portion of the overall testing market. And using a service provider is far more cost effective for the corporation. "If you want to mow your lawn, you have to buy a lawn mower and other equipment," he said. "Or you can just get a landscaper. That's a better choice for these massive testing efforts."
Experitest: Testing on-site and on the ground
Founded in 2009, Experitest has been testing smart phones from the beginning. The company is seeing a sharp uptick in demand this year after enterprises have spent several years experimenting, says CEO Tal Barmeir.
Experitest offers a software-only solution, SeeTestAutomation, that's on-site using a customer's devices. "That offers full control and security," said Barmeir, who stressed that many companies are uncomfortable with the potential security issues that can come from testing in the cloud. The company's software works with existing ALM environments, and the biggest selling point is that a customer can create a single test and run it on any device or OS with absolutely no changes.
The secret is the use of native Web ID technology, which stays stable no matter what device or network it's on. Barmeir compares it to how your Social Security number stays the same even if you cut your hair or put on glasses. The Web-ID-based image doesn't change from phone to phablet or anything in between, so testing many multiples of devices on different platforms is completely straightforward.
SeeTestAutomation is available through an annual or perpetual software licensing agreement. The company also has a free-of-charge option, SeeTestManual, that lets developers use a computer, mouse and keyboard to manually test devices.
Perfecto Mobile: A flexible cloud-based solution
"We meet you where you are" is how Perfecto Mobile's director of product marketing Carlo Cadet describes his company's mobile testing service. Unlike Experitest, Perfecto's MobileCloud is 100% cloud-based. Using a hybrid-cloud enterprise-grade automation system, Perfecto can offer functional testing, pre-production performance testing and post-production monitoring. The company has four large testing labs -- two in the U.S., two in India -- that customers can access through an on-premises cloud, a private cloud or the company's public cloud.
Cadet says the Perfecto "advantage" is that the company uses a single test environment that's sufficiently elastic that all test requirements can be met while still leveraging a customer's existing tooling, such as HP's UFT, Microsoft's TFS and IBM Rational.
Having access to all those devices and in such a flexible way is very helpful, says a senior manager of a leading financial institution who declined to be identified. A Perfecto customer, the manager (who admits to a whole lot of testing-related grey hair), said, "If you looked behind the curtain we were not very efficient in testing. Now having automation allows us to do everything faster and relieve the tension caused by velocity and quality."
Perfecto offers users a free trial of its services and a variety of pricing and packages from month to month to annual licensing.
Keynote Mobile: A vast choice of devices and locations
Keynote Mobile is also a cloud-based testing company, and it claims to have the largest real-device library located in the company's DeviceAnywhere Cloud. The company has over 7,000 computers and mobile devices sitting in over 245 testing sites around the world. But since having those real devices won't matter if they can't be tested for how users actually handle them, Keynote has created Direct-to-Device testing for more accurate simulation.
To better monitor custom applications in other areas (dashboards, for example), the company offers two APIs to make it easier.
Late last year Keynote introduced "Project Pricing" for its DeviceAnywhere Cloud testing service. Customers can now choose to pay per project if they wish.
Mobile Labs: Portable and on premises
Another cloud-based mobile testing company, Mobile Labs, takes advantage of the cloud, but with a twist. Companies can test their applications in Mobile Labs' private cloud, safely behind the corporate firewall. Mobile's deviceConnect is housed in a small, portable cart containing a server and a rack. It's quick to set up and maintain, and the company says it's the ideal solution for enterprises wanting cost-friendly on-premises testing. And it can be configured for both automated and manual testing.
The bottom line is that application testing is probably going to get more complicated and demanding before it gets easier. But at the end of the day, companies have to come up with a plan or deal with impatient users who can be very vocal on social media if something goes wrong. "You risk your brand reputation if you don't test," IDC's Marston said. "You can't afford to always be in damage-control mode."
Learn how to simplify automated app testing with Perfecto Mobile CQ Lab
Automated image-based testing tool performs tests across platforms