Mobile application testing is getting short shrift, with some organizations doing little or no testing at all. Experts say these organizations don't know how to test applications that run outside the firewall. Nor do they understand how mobile applications support key business goals and processes.
In this article, experts explain why organizations are behind the curve on mobile application testing, and what they need to do to assure the quality of these important apps.
The mentality around mobile testing is, 'Most of the app should work. What's the big deal if there are defects?'
software quality analyst, Cloud Perspectives
The root of the problem is that test organizations often don't take mobile applications seriously, said Steve Woodward, owner of software and testing consultancy Cloud Perspectives in Ontario, Canada, and a certified software quality analyst. They look at them as mini smartphone apps with limited importance to the business. "The mentality around mobile testing is, 'Most of the app should work. What's the big deal if there are defects?'" he said. This mindset is dangerous because mobile applications emerging today are designed to align with and carry out the objectives of the business.
Another reason mobile apps don't get enough attention is that organizations don't understand how to evaluate their performance across the wide array of environments in which they will be deployed. As a result, the limited testing that does get done takes place in a lab, "where it is challenging to replicate conditions mobile application users are likely to experience in the field," said Matt Johnston, chief marketing officer for test service provider uTest Inc. in Southborough, Mass.
The World Quality Report 2012-2013, published by Capgemini North America Inc. and Hewlett-Packard Co., supports the experts' views. Only 31% of more than 1,500 global organizations surveyed conduct formal testing for mobile applications, the report found. Those that don't do formal testing said they lack the right resources, tools and methods to address mobile testing challenges. "We were surprised by the relatively low level of proactive structured testing in this increasingly essential area of business connectivity," authors Murat Aksu and Charlie Li wrote in the report's executive summary. "We believe that mobile testing needs to be a fully integrated element of the QA [quality assurance] discipline, so that the mobile strategy of the enterprise takes testing into account right from the start."
What do you expect for $1.99?
A get-rich-quick mentality surrounded the development of the earliest smartphone applications, and testing was barely a consideration, Cloud Perspectives' Woodward said. That mindset remains, even though enterprise mobile applications today bear little resemblance to earlier mini apps designed for individual use. These early mobile apps were all about making money, and developers pushed them out to the marketplace as fast as possible. "Quality was an afterthought at best," he said.
To illustrate his point, Woodward recalled a conference session he sat in on at the Mobile World Congress in 2010. The room was full of young developers comparing notes on how many smartphone users bought their apps. "Some of these guys were looking at 200,000 downloads at $1.99 each, in just a couple of months," he said. He interrupted and asked about application quality. The developers said, "Well, there will be some defects, but we don't worry about them too much. Once the release date is reached, the app goes out." Woodward recalled that one developer said, "What do people expect for $1.99?" He remembers thinking, "OK, thank God these guys aren't writing enterprise mobile applications."
Develop business-based use cases
Unlike those early smartphone apps, enterprise mobile applications today carry out key business processes and transactions. Understanding this fact should drive the mobile application testing strategy, the World Quality Report said, noting that the organizations surveyed did not have business-driven test strategies in place. "Test strategies should consider the objectives of the business owner, how the mobile app is delivered, and the target user for the app, whether that be customers, suppliers or employees," the report said.
Current mobile applications are often used by employees outside the office in "customer-facing" situations, uTest's Johnston said, offering a few examples: Salespeople use mobile applications to take orders and look up customer accounts. Insurance adjusters rely on them to input data and photos from customer sites and start the claims process. Delivery firms depend on mobile applications to signal servers in their back-office systems that a package has been delivered, and often to capture and forward the customer's signature as proof.
Understanding who is doing what with a mobile application and what that means to the business -- that's the foundation of use case testing. The tests should reflect essential business processes, Johnston said. Can the user log in, check inventory, write up a bill of lading, file an insurance claim, write up an order? "If these types of things aren't working, you want to discover that early in the game, not in the ninth inning," he said.
Mobile performance matters
Also crucial to consider are where the mobile application will be deployed and how performance will be affected as the user changes locations, Johnston said. "You need to place testers in the field, not just the lab."
Conditions on the network have an enormous impact on the end user experience for mobile apps, said Todd DeCapua, vice president of channel operations at Shunra Software Ltd., a Philadelphia firm that sells performance and virtualization software. Most companies aren't doing an adequate job of testing for this, he said. "They assume that network conditions in which the app is deployed are fixed and as a result, the application will perform consistently."
But this assumption doesn't apply to mobile applications, which essentially introduce a new set of ways an application can fail. Performance can change constantly depending on the network carrier and the user's location, for example. Poor performance can make or break these mobile applications, many of which interact with the server on a near-constant basis. "Real-time applications such as electronic trading systems poll the server once every 15 seconds," DeCapua said. And mobile users cannot conduct business from their devices if the response time is too slow.
The World Quality Report 2012-2013 said organizations conducting formal mobile QA programs focus primarily on performance (64%) and functionality (48%). But a mere 18% of organizations focus on mobile security testing. "Security is a perennial issue where mobile is concerned," the report said. This doesn't mean that companies are no longer concerned with security, but rather that the perspective on security has changed with the bring-your-own-device culture, it said. When it comes to mobile application testing, "QA has fallen behind the mobile curve."