Mobile software introduces its own unique set of failure modes. So many that even the most effective automation tools and emulators can't replicate the conditions of every single point of failure.
The nature of app stores compounds the importance of mobile app testing to find critical errors. Mobile users quickly abandon software that is released in a buggy state, marring it in the stench of bad reviews -- which could keep away future users.
There's also a sizable amount of overhead time when submitting to any app store. Teams might only have one chance to get it right. So, it makes sense to not only run automated checks, but to study all the holes the checks miss and perform exploratory tests for those as well.
This checklist for mobile app testing covers 12 often undertested aspects to consider when developing or updating applications.
1. Single-pixel length links
A directory listing by last name can be frustrating -- if not impossible -- to click accurately on mobile with a finger, even if it works easily on an emulator or by an automation tool. For instance, the capital letter "I" and lowercase "i" can both render as a single pixel in width. The Apple Music application avoids this problem with letters that are bold and two pixels wide. Test how easy it is to scroll through and select individual letters within the application's directories.
2. Dropped wireless service
Imagine a mobile app user who is about to -- or just has -- hit the checkout button but then loses cell service. Consider how frustrating it would be for them to not know if their transaction is complete. Test how the application retains and recalls transaction information when a user regains service. If the application has the functionality to notify users that their action failed, check that feature as well. You can save customers the frustration of accidentally making the same purchase twice or from being so exasperated they give up entirely.
3. External interruption
This entry in the checklist for mobile app testing is similar to the one above, except users don't lose cellular service; instead, their app experience is disrupted because of cell service. Test applications to see how they function when interrupted by a call, a text message, an alarm or a third-party notification, such as one from Google Maps. One method that works for most any team: Try a phone call long enough to force a timeout when the software is being used.
4. User-based interruption
Several mobile applications are used passively when users put their phone to sleep or as they navigate through other apps. For example, many iPhone applications let you play podcasts or music in the background while immersed in the other app. However, there are applications -- like Placebo by Momanda -- that can't play music if the user's phone goes to sleep. Tracks need to be primary in order to play, which won't work if the phone's timeout is set to a low amount of time or if the user puts the phone to sleep as a habit. Verify the developers' chosen approach to app function while not the primary, and see if it works correctly.
5. Battery and cellular data drain
Test how heavily the application uses battery and cellular data. Both forms of consumption directly determine how usable mobile software is in the real world. Take Highlight, a location-sensitive mobile app, which connects users if they have something in common and are nearby. Its core functionality requires a constant back and forth with servers, which leads to extensive battery drain. Once praised at the SXSW Interactive Festival, the app is no longer a serious player in that space. Applications that use up a great deal of bandwidth could suffer a similar fate if they use a disproportionate amount of subscribers' data plans.
6. Heat generation
For processing-intensive applications, such as video players and non-native software, heat is a risk factor. This problem will never show in an emulator for mobile app testing, but it might if an enterprising tester takes a long car trip and leaves a video player on -- not to mention if you constantly charge your phone and play videos' audio through the car speakers as if it were music. In both cases, heat can break down the device, and the application could require a redesign to prevent constant processor use.
7. Memory loss
Run the application concurrently with a variety of others on an older, low-memory mobile device. Look for any performance slowdowns or crashes. They will happen; the challenge is to decide how much performance degradation is acceptable and the oldest devices an app runs reasonably on.
8. Cellular speed problems
Another factor for data-intensive applications is response speed. What might work exceptionally well on a company's internal network could fail on a 3G network just on the edge of cell tower range. Both Apple and Android ecosystems have utilities like network link conditioners that can simulate a slow, delayed or lossy connection during mobile app testing.
9. Deep linking
Take a link to a regular webpage, accessible only after login from a desktop, and email it to a phone. When the user clicks the link, are they directed to log in? If yes, test if users are sent to the specific linked-to page. And if it's relevant, ensure they are redirected to the mobile version of that page. This practice is called deep linking.
10. Portrait mode repaints
No. 10 is a simple but oft-overlooked step on checklists for mobile app testing. Some applications, like Apple's Calculator, display more functionality when the screen turns. All of them repaint the screen, causing a change to the form factor. See if these capabilities can be accessed in practice.
11. Swipes and other motions
Don't overlook how fundamentally different it is to interact with an application via your hands as opposed to a keyboard and mouse. Mobile devices lack a right click, for example, but zoom in and out with a fluid pinch.
If you run an application with a testing tool, or even an emulator, it can feel much different than touching a physical phone. So, test like someone who would actually use the application. You can catch and document issues that directly affect the user experience, hiding in plain sight. For example, ferret out any app functionality that requires a right click, and send it back to developers for a rewrite.
12. Older device dilemmas
Acquire the oldest phones that wireless carriers still support. Expect to see all of the problems listed in the above checklist for mobile app testing manifest earlier in the process. Older devices get hot, crash and run out of battery more quickly than those manufactured today.
Put it all together
The first time you test a new mobile application, you can spend days on the usability and platform problems listed above. As code releases more frequently, you may be tempted to test for only a few of them or perhaps to forget about this sort of extra-functional checklist for mobile app testing altogether.
Don't do that.
Instead, keep a list of potentially consequential risks that you periodically review and mitigate. If you'd like, make a deck of cards with these risks, shuffle them and pull out five to check every deploy. No matter what you do: Stay vigilant.
"A foolish consistency is the hobgoblin of little minds," Ralph Waldo Emerson -- the 19th century philosopher -- said once. If consistent automation means that testers never look outside of what the tools can see, then it might indeed just be a foolish consistency.
Let's compare the mobile, web testing tools Browserstack and Sauce Lab