As enterprises move from Web applications to mobile apps, software testers are building mobile software testing skills on top of their existing Web application skills. Unfortunately, testing mobile applications is a lot different than testing Web apps. There are plenty of articles that help beginners take the first steps in mobile testing. This article is intended to give practical advice and actionable steps to testers that are already working with mobile applications.
Multimedia bug reports help gather and present data
More information about mobile testing
This is the first part in a three-part series on mobile testing.
In the second part, you’ll find tips for testing iOS apps.
In the third part, Heusser covers Android app testing.
Most bug reports are plain text; they take time to create, time to understand, and can involve information loss. Using screen captures, or even creating movies to demonstrate buggy behavior, can be faster and clearer than trying to explain it in words. I record movies with a microphone on so I can explain what I am doing; this makes reproducing the problem a snap for the programmers.
With Web-based mobile applications, errors can frequently be recreated on a laptop in Chrome or Safari, which makes it even easier to produce those quick bug clips. Alternatively, a second device can record a movie as bugs are reproduced on the first device.
If an external decision maker -- a product manager of some type -- is the one to decide what to fix, watching a movie that includes a voiceover explaining what's happening will help that person make better decisions about how to treat the bug.
Use developer tools
It is easy enough to simply not support private browsing, but get that in writing.
I've written step-by-step directions for Android developer tools and other Android testing tools, as well as separate instructions for Safari Web Inspector and other iOS debugging tools. Check those out for specifics on how to run through these steps on one platform or the other.
Focus on networking issues
When programmers shift from a desktop world to mobile, they often make assumptions, including that the Internet connection will always be working or accessible. Networking issues on mobile devices span a gamut, from unwittingly leaving coverage areas, to purposefully disabling network connections with airplane mode. These are cases we have to test for.
The classic way to test for changes in signal is to ride on a bus, take a walk, or go into an elevator -- that sort of thing; all of that is very time-intensive. Signal loss could also be simulated by switching to airplane mode in the mobile device settings. This is easier, but has some limitations.
Switching to airplane mode can cause different behavior than losing and gaining an external connection because the operating system knows it should stop getting messages, and the cutover is so fast. Airplane mode testing might not be valid, but then again, it might be.
I suggest trying both and checking for differences in behavior. If the behavior is the same, the at-the-desk shortcut of airplane mode might be appropriate. Be wary, though, of only testing in airplane mode or only testing once. As applications change over time, look for new user actions or functionalities that require long page loads, or AJAX-like requirements that might happen in the background.
Try throttling bandwidth
Testing should also be done under conditions ranging from full to zero connectivity. There are two ways to do this, either by using developer tools to simulate networking issues, or by creating a virtual wireless access point and reproducing adverse conditions with a tool like IPFW. Apple iOS developers will likely choose the former, while Android developers are likely to choose the latter.
Don't forget about private browsing
Some power users do not want their browsing habits stored, so they turn on private browsing, which is similar to "incognito mode." If the application tries to read and write to local storage (to hold data if the wireless connection is lost, for example), then the reads and writes may fail in mysterious ways for users with private browsing on. I recommend a two-pronged attack: Talk to the technical staff about local data storage, but also spend ten minutes using the application with private browsing on.
It is easy enough to simply not support private browsing, but get that in writing. Also, be prepared for that written-in-stone decree to change; private browsing might be used by a surprisingly high number of users. If support calls start coming in with strange errors that can't be reproduced, ask if users have private browsing on.
Read the latest mobile testing strategies