Whether it’s the VP in Finance trying to access a website on his BlackBerry mode RS5.7.34, or that “huge sale” to the company that has its taxi drivers using the Motorola Xoom, suddenly your company is expected to support devices you didn’t know existed -- and certainly can’t pronounce.
Now what? Read on for ideas to help your organization get started with a mobility testing strategy.
Declare a service level capability
If you ask for what devices to support, the answer that you’ll get will likely be “all of them.” If you ask what level of features to support, you will probably get the same answer: “all of them.”
Start by asking what the top two mobile platforms will be. If no one knows, you may find the top existing devices in the system logs.
Once you are armed with the information of the top devices it may be possible, even expedient, to short-circuit that process by declaring what devices you will test, and, later, the issues you find with the application. In many cases, a casual test using some of these tips will provide you with a large pile of defects.
Get up, running and testing
Yes, you can and should go and find simulators for the devices. The simulators for the three most popular categories today are BlackBerry, Android, and iPhone. The installs for most of these devices are straightforward, though the BlackBerry and iPhone installs will take some effort.
To get the BlackBerry running, you will need to first download and install Java then download, install and run an MDS “Server” running in the background to simulate internet. Now you can install the BlackBerry sim. For the iPhone sim, you’ll need to install the entire developer toolkit on a Macintosh, then look in the /Platforms/iPhoneSimulator.platform/Developer/Applications directory.
Now you’ve got the simulators installed, but beware, you can’t always trust the simulators, and you certainly can’t test everything necessary to test with only a simulator. You can’t add g-forces to a simulator, you can’t throw one in a handbag and jostle it around, nor can you contrast the impact of a human touch with the efficiency of a mouse.
My advice is to get your hands on at least one physical device for every category and find a large number of defects by hand. Think about automation later when the bottleneck has shifted back to development.
Define the mobile product
To get around these issues, many teams create a mobile interface that has limited functionality and is optimized for small screens. At Socialtext, our software senses the device type and can re-direct low-resolution devices to a “mobile” Web page directly. We even offer two different mobile experiences based on the capabilities of the browser.
Test and support strategy
A mobile testing project is just that: A project. Testing on a mobile device means a different amount of screen real estate, different processing power, and different common failure modes. So you may have time for one mobile project (perhaps to build that stripped-down interface), but after the project is over, you’ll want to add testing of the mobile devices to your regression-test strategy. That means more work, which in theory should mean the testers get more time or get to add staff.
Sadly, neither of those have been my experience.
How to compress the test window
I look to compress schedules in three ways: By collapsing platforms, by rotating test systems, and by being clear about risks and what “support” means.
When I say “collapsing platforms,” I mean take a careful look at the bugs. If all of the bugs we find on the BlackBerry appear in all versions, we might be able to test just one or two versions of the BlackBerry. The same applies for the iPod, iPhone, and iPad. (This assumes a Web-based application; for a native application, you’ll be able to eliminate platforms that do not belong to the same “family.”)
Another way to collapse platforms is to look at the core systems. For example, www.iphonetester.com on a Macintosh with a Safari browser is not an iPhone... but it’s close, and you can drive it with automation tools.
A rotation strategy looks at the risks slightly differently, doing deep dive stress tests on one or two major platforms per release. You might, for example, rotate testing devices every release, testing the iPhone this time, Android the next, BlackBerry after that. This introduces a little more risk for a few weeks at a time -- you might look at usage statistics on your server to decide if that move is prudent.
Finally, you could separate support from test, defining “support” as “if you find a bug, we’ll address it,” and be clear with management that mobile devices are not the focus of the testing.
Putting it all together
You’ll want to find out a lot more about the target customer for the application and actually use the software in their environment. You’ll want to try a variety of different platforms, both on simulators and on actual devices. You will need time to do the testing work, time for the fixes, and time to verify the fixes. And, yes, all the old rules for attacking software still apply, as well as a couple of new ones.
The bottom line is that supporting mobile devices will take more time, more staff, or more risk. Management gets to choose which, but you can give them information so they make an informed choice.