Over the last few months, I’ve been asked a lot about testing on mobile devices. At my own company and at other companies in the Midwest (which isn’t exactly known as a hotbed of technology). I’ll plead mostly ignorance, most of what I know is from Julian Harty, whose work I follow both in print and at conferences. I think the topic is important enough that in a series of workshops I co-host, we’ve added a full day to the topic.
Yesterday, I read this story on new handsets and the issues that are encountered in the marketplace as vendors work to keep up with the cutting edge and market demand. What I really like about this article, aside from it being an interesting news story, is that I think it provides a nice snapshot of some of the common challenges of testing on mobile devices.
- Meeting launch targets: I’m not sure how unique this is to mobile devices, but it certainly seems like deadlines often coincide with hardware deadlines. For example, if you’re writing software for the iPhone, then to some extent your releases are tied to theirs, if for no other reason than support (see software updates below). If a new “iClone” comes out, there’s a chance you get to port your iPhone app over to that device.
- Handsets are getting more complex: My first Palm had a calendar, took notes, and had a calculator. My first cell phone only made phone calls. They claimed there were other applications on it, but no one was really using the early cell phone calculator. I mean, really? Now, I can read a spreadsheet on a phone, take a picture, stream media, look up directions in real-time, and likely just about anything else I can think of. And I can do that using a keypad, stylus, keyboard, gestures, or the soon to be invented chip implant. I’m sure it’s only two or three years out … really.
- Consumer expectation: Expectations for mobile devices are high. From performance and reliability to recoverability of data, the quality criteria used when testing on mobile devices is subtly different from those of traditional Web and desktop applications. The software I use on my phone, while it’s typically functionally simpler, operations in a much more hostile environment: there’s less processing power, less memory, less disk space, connectivity comes and goes, and my attention span is shorter because I can’t really multitask like I can on my desktop.
- Software updates: While this isn’t a new testing problem, for many programmers this problem has largely been solved. Think of Web browsers: If you want to hit the largest markets, you need to support two to four browsers. For each of those, you’ll make some sort of decision on what versions you’ll update, which service packs, and depending on your software, what hotfixes you’ll need to test with (and on what schedule). In addition, regression test automation for applications that run in a Web browser is largely trivial (compared to even five years ago). On mobile applications, there’s way more than three main platforms, each one offering regular updates (often paired with hardware updates), and your ability to effectively automate much of that testing is largely not trivial given that it’s some mix of emulators and actual physical devices.
- Touch-screen sparks new problems: A subset of the complexity problem above, touch-screens (like graffiti before them) introduce a relatively new method of interaction with the product. This likely means increased usability testing and changes in thinking about design (as the article points out). It’s most likely just the latest step in the evolution of that platform.