Problem solve Get help with specific problems with your technologies, process and projects.

Checklist for mobile app testing: 12 gaps to look out for

Emulators and automation tools are useful, but don't rely on them solely. Use this checklist for mobile app testing to ensure that software has no critical flaws.

Mobile software introduces its own unique set of failure modes. So many that even the most effective automation...

tools and emulators can't replicate the conditions of every single point of failure.

The nature of app stores compounds the importance of mobile app testing to find critical errors. Mobile users quickly abandon software that is released in a buggy state, marring it in the stench of bad reviews -- which could keep away future users.

There's also a sizable amount of overhead time when submitting to any app store. Teams might only have one chance to get it right. So, it makes sense to not only run automated checks, but to study all the holes the checks miss and perform exploratory tests for those as well.

This checklist for mobile app testing covers 12 often undertested aspects to consider when developing or updating applications.

1. Single-pixel length links

A directory listing by last name can be frustrating -- if not impossible -- to click accurately on mobile with a finger, even if it works easily on an emulator or by an automation tool. For instance, the capital letter "I" and lowercase "i" can both render as a single pixel in width. The Apple Music application avoids this problem with letters that are bold and two pixels wide. Test how easy it is to scroll through and select individual letters within the application's directories.

2. Dropped wireless service

Imagine a mobile app user who is about to -- or just has -- hit the checkout button but then loses cell service. Consider how frustrating it would be for them to not know if their transaction is complete. Test how the application retains and recalls transaction information when a user regains service. If the application has the functionality to notify users that their action failed, check that feature as well. You can save customers the frustration of accidentally making the same purchase twice or from being so exasperated they give up entirely.

3. External interruption

This entry in the checklist for mobile app testing is similar to the one above, except users don't lose cellular service; instead, their app experience is disrupted because of cell service. Test applications to see how they function when interrupted by a call, a text message, an alarm or a third-party notification, such as one from Google Maps. One method that works for most any team: Try a phone call long enough to force a timeout when the software is being used.

4. User-based interruption

Several mobile applications are used passively when users put their phone to sleep or as they navigate through other apps. For example, many iPhone applications let you play podcasts or music in the background while immersed in the other app. However, there are applications -- like Placebo by Momanda -- that can't play music if the user's phone goes to sleep. Tracks need to be primary in order to play, which won't work if the phone's timeout is set to a low amount of time or if the user puts the phone to sleep as a habit. Verify the developers' chosen approach to app function while not the primary, and see if it works correctly.

5. Battery and cellular data drain

Test how heavily the application uses battery and cellular data. Both forms of consumption directly determine how usable mobile software is in the real world. Take Highlight, a location-sensitive mobile app, which connects users if they have something in common and are nearby. Its core functionality requires a constant back and forth with servers, which leads to extensive battery drain. Once praised at the SXSW Interactive Festival, the app is no longer a serious player in that space. Applications that use up a great deal of bandwidth could suffer a similar fate if they use a disproportionate amount of subscribers' data plans.

6. Heat generation

For processing-intensive applications, such as video players and non-native software, heat is a risk factor. This problem will never show in an emulator for mobile app testing, but it might if an enterprising tester takes a long car trip and leaves a video player on -- not to mention if you constantly charge your phone and play videos' audio through the car speakers as if it were music. In both cases, heat can break down the device, and the application could require a redesign to prevent constant processor use.

7. Memory loss

Run the application concurrently with a variety of others on an older, low-memory mobile device. Look for any performance slowdowns or crashes. They will happen; the challenge is to decide how much performance degradation is acceptable and the oldest devices an app runs reasonably on.

8. Cellular speed problems

Another factor for data-intensive applications is response speed. What might work exceptionally well on a company's internal network could fail on a 3G network just on the edge of cell tower range. Both Apple and Android ecosystems have utilities like network link conditioners that can simulate a slow, delayed or lossy connection during mobile app testing.

9. Deep linking

Take a link to a regular webpage, accessible only after login from a desktop, and email it to a phone. When the user clicks the link, are they directed to log in? If yes, test if users are sent to the specific linked-to page. And if it's relevant, ensure they are redirected to the mobile version of that page. This practice is called deep linking.

10. Portrait mode repaints

No. 10 is a simple but oft-overlooked step on checklists for mobile app testing. Some applications, like Apple's Calculator, display more functionality when the screen turns. All of them repaint the screen, causing a change to the form factor. See if these capabilities can be accessed in practice.

11. Swipes and other motions

Don't overlook how fundamentally different it is to interact with an application via your hands as opposed to a keyboard and mouse. Mobile devices lack a right click, for example, but zoom in and out with a fluid pinch.

If you run an application with a testing tool, or even an emulator, it can feel much different than touching a physical phone. So, test like someone who would actually use the application. You can catch and document issues that directly affect the user experience, hiding in plain sight. For example, ferret out any app functionality that requires a right click, and send it back to developers for a rewrite.

12. Older device dilemmas

Acquire the oldest phones that wireless carriers still support. Expect to see all of the problems listed in the above checklist for mobile app testing manifest earlier in the process. Older devices get hot, crash and run out of battery more quickly than those manufactured today.

Put it all together

The first time you test a new mobile application, you can spend days on the usability and platform problems listed above. As code releases more frequently, you may be tempted to test for only a few of them or perhaps to forget about this sort of extra-functional checklist for mobile app testing altogether.

Don't do that.

Instead, keep a list of potentially consequential risks that you periodically review and mitigate. If you'd like, make a deck of cards with these risks, shuffle them and pull out five to check every deploy. No matter what you do: Stay vigilant.

"A foolish consistency is the hobgoblin of little minds," Ralph Waldo Emerson -- the 19th century philosopher -- said once. If consistent automation means that testers never look outside of what the tools can see, then it might indeed just be a foolish consistency.

This was last published in June 2018

Dig Deeper on Mobile Application Testing Techniques and Tools

Join the conversation

1 comment

Send me notifications when other members comment.

Please create a username to comment.

What mobile application bugs have you only discovered through less conventional testing?