Mixing mobile device users' high expectations for application performance with widespread enterprise adoption of mobile networks increases the risks of high-profile failures. In addition, today's growing demand for rapid, precise mobile application distribution across many devices, operating systems and networks raises the stakes for mobile application development and quality assurance managers. This high-risk scenario requires mastering new and time-tested approaches to pre-deployment performance testing now more than ever.
"You don’t want your end users to be your testers. This is a terrible idea. Before mobile and social media, this might have been OK because the negative feedback would be more muted and not be 'viral' in the sense of infecting your brand before you had a chance to administer the cure," says Dave Berg, senior director of product management for Shunra.
Theresa Lanowitz, founder of voke, inc., agrees that software performance directly links to a company's overall reputation and that it is best to avoid consumer testing. “The software that runs your company is now inextricably linked to your brand. So your brand is reflected through the software you are putting out there; your brand is reflected through the software that your customers are using," she explains.
Research findings: Customers are merciless
Even smaller issues can negatively impact the brand. For example, according to research cited by Aberdeen senior research analyst Jim Rapoza, 7% of users abandon applications after just one second of delay in performance. 11% abandon at two seconds, 18% at three, 25% at four and 50% at five seconds.
New voke research explores the factors that are affecting the testing market, such as cloud and mobility. Lanowitz says, "The testing of software at every stage of the lifecycle with all aspects of the supply chain is a standard and required practice, and continues to grow in importance."
The growing urgency of application performance testing is reflected in recent Aberdeen research as well, which shows that 18% of businesses are currently performing application testing, while 54% plan to in the next year, according to Rapoza.
Testers now must act "in the strategic role of customer advocate and help deliver higher quality software throughout the enterprise by placing a laser focus on assessing the risk associated with every piece of software," says Lanowitz. After all, it's not just about producing quality applications, but also satisfying the expectations of end users and preserving the integrity of the company.
Cost benefit of pre-deployment testing
Conducting pre-deployment performance testing is critical, and it's also much less costly. "It is easier, more cost-effective, and protects your brand to performance test before deployment," Berg says.
Shunra surveys have pointed to the fact that "60% of the total cost in an application's lifecycle come from remediating performance related issues after the app has been deployed," according to Shunra CEO Gary Jackson. "If you cut that number down just a few points, you will see astronomical savings and ROI on the pre-deployment testing you preformed. It is close to 100:1 in cost savings."
The costs can really add up, "because it's not just a matter of having to bring an app down, or bring a newer version down in order to rebuild it and retest it, it's also the lost user opportunity; one bad experience will often drive your users away," Rapoza adds.
Fortunately, the market is responding with various tool and service offerings that address each layer of software performance.
Lanowitz discusses the proliferation of mobile test vendors and their willingness to partner with one other. "You have a whole host of mobile testing companies coming out. What they're doing a really great job of is, they're building this incredibly complex matrix that will allow you to test your mobile application from anywhere in the world into any geography. So you can do your testing from Bangalore, and say, 'OK, I have to make sure this will run perfectly in Chicago, in the United States, on these carriers.'"
She highlights some of the available tools:
- The Hewlett- Packard LoadRunner product has the most market share, and has been offering performance testing for the last decade. HP now offers LoadRunner in the cloud.
- Hyperformix, which is owned by CA, enables architects in the planning phase to explore what happens when using different operating systems and carriers.
- SOASTA offers cloud testing and performance engineering experts to answer questions.
- Shunra ensures your network and application is going to run well, from the pre-deployment environment.
- Keynote DeviceAnywhere and Perfecto Mobile offer solutions in the mobile space.
- Microsoft Visual Studio 2010 Ultimate offers unlimited load testing when you own a license to the product.
"The fact that these companies like Shunra are partnering and participating with companies like HP and like Keynote DeviceAnywhere and SOASTA just goes to show that this whole idea of performance is not a one-size-fits-all type of problem. We have all these different components and different layers now because the complexity has continued to increase, so seeing that these vendors are getting together and partnering is really, really good," she adds.
Rapoza highlights the organization-wide interest in pre-deployment testing: "So now you're seeing a whole awareness of performance that goes across the entire application ecosystem. The importance of testing has gone beyond just developers. Everybody concerned with the application wants to make sure that the application was properly tested beforehand because you don’t want to send up an app and have tens of thousands of people jump on it and they hit a problem that you would have found if you would have done testing."
Companies must now adjust to the fact that software testing decisions are heavily influenced by consumer experience and consumer expectation. "IT was always driven with a purpose and plan from the top down. We are now in a consumer-driven environment that is unprecedented. You have to know how apps will perform on the devices and platforms your employees are using. You no longer have a choice to use one device, one system and have that be good enough," explains Jackson.