When it comes to Social Local Mobile (SoLoMo) apps, standard testing measures aren't enough. So how do test managers assure the quality of these mobile applications that are location-aware, on the move and interacting with social networks?
SoLoMo applications are forcing test managers to consider new kinds of failure, give greater consideration to nonfunctional testing and do more beta-testing with users. I hope you find SoLoMo apps interesting and challenging to work with -- because they are making the job of test manager a lot more interesting and challenging.
In this tip, I cover five ways to deal with these new challenges in testing Social Local Mobile applications.
Tip 1: Narrow the near-infinite set of devices.
The first problem test managers face with SoLoMo apps is figuring out what devices to test, given that it's virtually impossible to test every possible device. The Android operating system alone runs on more than 100 devices. Each device can have quirks, and the same device can run a different version of the Android operating system or a use a different browser or browser version.
That's just Android. Test managers also have to consider Apple iOS on iPhones, iPads and iPods, including different browser and operating systems versions -- not to mention Blackberry smartphones as well as future devices. The list is huge.
One place to start is with a breakdown of market share for different devices; that will help you determine where to put your testing effort. If possible, get the breakdown of devices by actual users. Web server logs, for example, can be a good source for this information. They often contain the source operating system and browser from the devices of users actually viewing the site. Failing that, look for general measures of traffic, like the kind available at netmarketshare.com, which provides statistics for Internet technologies.
Tip 2: Present the customer with the points approach for SoLoMo apps.
One approach is to test wide and shallow, with lots of combinations that get a few minutes of attention each. Another is to test deep with only a few devices, or do something in between. A useful strategy is to use market-share data to determine what to test on, then offer the application stakeholder some number of "points" to spread out over various devices. A common choice my clients make is to pick one or two reference devices; for example, one iPhone or one Android device, perhaps a Blackberry, and invest significant effort into those devices with every release.
Tip 3: Pay attention to nonfunctional testing.
How do test managers assure the quality of mobile applications that are location-aware, on the move and interacting with social networks?
A key benefit of the points approach is that the customer is choosing how to invest the test organization's scarce and expensive time. Then, when defects occur on other systems, the conversation will not be about blame, but instead how to fix the problem, and if shifting the test strategy to include other devices makes sense.
Nonfunctional problems -- security, privacy, performance and scalability -- continue to matter with mobile devices. On top of that, mobile devices are moving, which means signal strength constantly changes and can be lost. What happens when the person walks into an elevator? Changes cell towers? When GPS fails?
Another thing to consider is multitasking use cases. Your software can work just fine, but what happens when an incoming call interrupts the application? Does it pause? What happens when the user returns to the application after the call is over? Other interruptions to test for are push notifications and users bringing up the main menu. All of these things should be addressed in your test strategy. Discuss with management how much time is appropriate to allocate to these issues and how you will test for them. It's easy to forget or ignore nonfunctional testing.
Tip 4: For SoLoMo apps, consider crowdsource testing.
Linus's Law states, "Given enough eyeballs, all bugs are shallow." In other words, if you can get hundreds of people to test your software all at the same time, from different places in the world, on different platforms and cell providers, some of those testers will find important bugs -- by sheer math.
Several companies, including uTest, 99Tests and Mob4Hire, offer crowdsource test services to augment and accelerate your existing testing efforts. In many cases, test and operations managers launch the application into production Friday night, get crowdsourced-tested all weekend, then decide Sunday night whether to pull the release or not -- before the business users get in.
Tip 5: Test for integration failures.
Another key testing challenge for SoLoMo apps is systems integration. How do test managers make sure that a status update is propagated to Facebook, or the photo automatically goes to Twitter? All of these services or social media sites are software systems, and those systems can go down. In some cases, the system might be up, but an IT policy might restrict access. For SoLoMo applications, this "other system is down" problem is huge, and it's crucial to test for it. If the mapping software system is down, the device's GPS feature may fail. Perhaps worst of all, the credit card system or e-payment system might reject a sale.
Mobile software touching any of those systems needs to be able to handle these types of integration failures. In some cases, it makes sense to keep those failures invisible. For example, you want to make sure the application keeps running when log messages fail to post to the network drive, but customers probably don't need to know. On the other hand, when a connection to PayPal fails, it should fail publicly. The user needs to know.
Is your organization developing and testing SoLoMo applications? What challenges are you facing? Email SearchSoftwareQuality editors and let us know.