"Mobile is a vast unknown domain," said Raj Subramanian, Progressive test engineer. For testers, this means mobile is rich with potential, but fraught with challenges -- high expectations, unstable tools and rapid device proliferation, to name a few. These challenges are compounded by the fact that users are fickle and mobile apps are very easy to replace. Carlo Cadet, director of product marketing at Perfecto Mobile, believes this risk is what makes mobile such a high-stakes field. "Mobile is unique in the fact that end users simply abandon apps. They install then uninstall. Enterprises have to recognize that this experience, this platform, this engagement model, is different, and the bar is higher."
However, there is more to lose than simple user engagement in the application; there is the loss of user trust in the organization, or so says Philip Lew, CEO of San Francisco-based XBOSoft, a software quality assurance company. "Mobile usability within the application can influence the way a user thinks or feels -- not only about the application itself, but about the company." Lew, Subramanian and Cadet all have presentations at the 2013 Software Test Professionals Conference (STPCon) that explore this very issue, and they all offer advice on how testers can approach this vast unknown domain.
"I think that probably the biggest challenge is device proliferation and having all the devices covered," Lew said. "You have to be able to anticipate which devices your users are going to use the most and then cover them." Subramanian added that the growing number of testable devices -- phones, tablets, desktops -- imply a growing range of screen sizes, color rendering, operating systems and manufacturers.
[T]his experience, this platform, this engagement model, is different and the bar is higher.
Carlo Cadet, Perfecto Mobile
"The greatest challenge is to ensure the user gets an optimum experience of our website on all of these devices," Subramanian said. "Samsung alone has thirty different screen sizes." An added complexity is that manufacturers such as Samsung add their own skin to the phone's pre-existing operating system. "Samsung has Touchwiz, so what they do is they put their own skin on top of the Android operating system, and it's important to actually test on these devices, too." Indeed, device proliferation seems to have a kind of snowball effect. Variables create variables that create variables.
Luckily, Cadet identified the issue that underlies all the others: immaturity. Relatively speaking, mobile app testing is a very young field. "The biggest challenge is really on the broad scale of immaturity. There are a lot of people who are just starting out and they are asking very early questions on what the best approach could be." He went on to explain that development is getting faster, so these questions need to be resolved in a shorter amount of time. Furthermore, the testing tools themselves are still quite young, which means they are relatively untrustworthy and unstandardized. "When it comes to mobile apps, I have evaluated so many tools and none of them are stable right now," Subramanian said. "They have a lot of disadvantages and they give a lot of false positives."
Advice for testers
To be able to tackle this brave new world of software testing, testers will have to reimagine what quality means in the mobile space. Subramanian believes that, on top of gaining deeper knowledge of Web technologies and operating systems, testers need to think like users. "Testers need to do real-time user experience testing instead of just tapping some buttons on the UI [user interface] and saying, 'It works.'" He provided the everyday example of accessing a mobile webpage in your home, then getting in your car, then arriving at your office, getting in the elevator, etc. This constant switch from Wi-Fi to 4g is one of many contingencies that testers now have to be aware of. "I think the testing world itself is moving more toward the technical side in terms of being able to understand and analyze data, for one thing," Lew said. "For another, it is important to actually do programming that can simulate these different types of user situations."
According to Cadet, the key to tackling all these contingencies efficiently lies in understanding the performance envelope rather than the performance target. "For example, you don't necessarily need to render the entire page in less than three seconds. You just need to render the page that's above the fold. Performance engineers will have to work more closely with the dev team to help optimize and employ tricks like this one." In other words, testers need to strategically simulate test scenarios that take the device, the user and the network conditions into account. "Stress the back end with concern to users and bear in mind that network conditions for real devices are quite different from Web access."
Lew, Subramanian and Cadet all agreed that automation would become more widely adopted and, by extension, automation tools would become more sophisticated. "A big trend in the industry is going to be in developing tools that cover more operating systems, as well as higher levels of functionality and reporting in the results," Lew said. In the meantime, testers will have to get creative.