You can't manage what you don't measure. It sounds simple enough, but the reality of the mobile applications performance testing market is far from straightforward. Just ask Jim Duffy, software test engineer for Trimble Navigation. An early version of his company's mobile app crashed when only five users tried to log on. That's when they knew they had to get serious about mobile performance testing.
That's a story software project managers everywhere should be able to relate to. They know they need to get aggressive about testing to preserve brand reputation as well as stay ahead of cost overruns and of course reduce risk. But testing is challenging. Add up the number of phones, tablets, phablets, networks and operating systems and then think about the hundreds of millions of users swiping, pressing, typing, shaking and interacting and it's little wonder corporations are struggling to implement a performance testing strategy.
Finally, after several years of dancing around it, experts predict this is the year the mobile performance testing tools market is really going to take off. Perhaps because of some well-publicized disasters (like healthcare.gov), a recent IDC survey shows performance is second only to security testing in importance to corporations for the coming year. And the performance testing wish list is huge: scalable, cost-effective, data-rich, fast, realistic, capable of integration and maybe even predictive.
There are lots of options in the market today, but adoption is not going to be pain free. Software project managers have to think outside the box, move faster than they ever have before and try to find mobile performance testing tools that fit their company's development style. Here are three popular choices.
SOASTA offers great CI integration
First and foremost, corporations have to eliminate the "old school" testing mentality. In the client server days, testing was focused on the 300 or 400 internal employees who used an application. "Today, it's hundreds of thousands of people or millions of people," said Peter Galvin of SOASTA Inc. "This requires a totally different type of testing." No retro-fitting is allowed.
In fact, it's just the opposite. In this fast-paced world, development organizations have completely outstripped QA, Galvin said, which is why, like many other companies, SOASTA has gotten behind continuous integration (CI). CI could bring everyone involved in an application together with scheduled sharing of information, test results and more. To get there, SOASTA has created a CI mobile testing framework with over 100 locations to test from.
The company's TouchTest library integrates with the application automatically and then tests can be run from a CI server. Although the process is automated, it's designed to capture information -- like gestures or events -- that used to be only available through manual testing. That means faster testing and lower costs, Galvin said.
SOASTA offers its service on a subscription basis.
Neotys gets at real device metrics
Neotys USA Inc. is also promising less expensive and faster testing -- 2 to 5 times faster -- and taking a different road to get there. Marketing Vice President Jonathan Bracken says the company's NeoLoad fully integrates network emulation and device simulation within the product. And a brand new partnership with Perfecto Mobile Ltd. allows the company to offer real device metrics under load or what Bracken calls "the complete end-user experience." With access to Perfecto's comprehensive lab, NeoLoad is now able to drill down and look at testing details like heat or CPU usage. NeoLoad has a scriptless, GUI-based design and built-in automation.
Neotys is just trying to keep it real, Bracken said. "If your test is not realistic, your app might 'pass' but when it hits the real world, it won't work."
That's why Trimble's Duffy is now using NeoLoad to improve development of the heavy construction industry company's dashboard-based mobile app. "Are we moving more swiftly on product development?" he said. "Maybe not. But we are moving more confidently because now we know the product and the system are not going to crash."
NeoLoad mobile performance testing tools are available on a subscription basis, but if you just need to test up to 50 virtual users, it's free.
Dynatrace adds the ability to track users
Dynatrace's PurePath technology can be inserted into a mobile device allowing companies to trace each transaction to the method level across the infrastructure down to the server level if necessary. "Using PurePath, we absolutely know how users interact," said David Jones, technology strategist for Dynatrace (formerly Compuware). "We can pull a tremendous amount of information out about the user actions, which are important to understand when you have performance-related issues."
The technology looks at the state of the device, what the user is doing as well as the network and the server and can put all the results in context. You can try Dynatrace's performance testing tools for free for 15 days, and then it's available on a subscription basis.
Mobile performance testing tools are just a piece in the puzzle
No matter the testing provider, though, the key is getting corporations to truly think through the testing process and break down those barriers between developers and testers, said SOASTA's Galvin. "We're all about helping companies come up with best practices. That's what this market needs."
See the answers to some frequently asked questions about mobile app performance
Learn more about the advantages of cloud-based performance testing tools
Get expert advice on building a mobile testing strategy
Learn what real user monitoring can do