In the first article of this series, I examined usability and context. In the second article I discussed functional testing, emphasizing content, business logic, and performance testing. In this article, I will start examining how to test mobile layout.
Mobile Layout Testing
There are three approaches to testing mobile layout: using a layout emulator like iPhoney, using a desktop device emulator such as OpenWave or Windows Mobile emulators, and using an actual device.
Layout Emulators
With the growing popularity of mobile applications, device and layout emulators are growing popular. iPhoney is one of these layout emulators—it is not a device emulator (it will not run software compiled for the device), but it provides a 'canvas on which to test the visual quality of your designs'.
Device emulators are not a final test methodology. I generally use this as a quick-check, proof-of-concept. They are great to use when pair-testing with developers—they give a quick validation of design approach. I don't think I'd ever post a defect found in a device emulator, but I definitely find this to be a useful tool in my testing toolbox.
Device Emulator
The real test for how an application will work is to run it in the actual device. The good news is that many devices have emulation software available which emulate the OS and installed applications of that device. These emulators literally load the device OS into memory, with an isolation layer that emulates the device's intended chipset. There is literally no more valuable tool in device testing than the device emulator. In most cases, emulators are free and available openly. Some emulators are bound by platform (for instance, the iPhone emulator only runs on MacOS inside of the developer SDK), but most are available on Windows.
To test with a device emulator:
1) First, you must find the emulator. Some manufactures like Microsoft make their emulators easy to find and use. To run a Windows Mobile emulator, download a copy of C# Express. Then install the Windows Mobile SDK. You can even download localized editions of Windows Mobile target device platforms.
2) Once the emulator is installed, you need to configure networking for it. This differs by emulator and is also impacted by your networking configuration, so this article won't go into how to configure each emulator. Make sure you can browse both Internet as well as intranet sites.
3) Finally, connect to the site under test, and start looking at your site. Returning to the concepts of usability and context, approach your site in the emulator asking 1) is this easy to use and 2) does it look good on my target device?
Which Emulator?
Some teams wonder which emulator to use. Sometimes this is an easy answer – if the mobile site is an internal line-of-business application, you can target it to the officially-issued device at the company. This might be a Windows Mobile Palm Treo or (if your company is really hip) an iPhone.
Usually the target device isn't so easily factored. In this situation, I find it's best to identify a lowest-common denominator and do all my usability and layout testing against this platform, and then spot check on more functional browsers. For this kind of testing, using the OpenWave emulator, one Symbian emulator, and a Nokia emulator can offer good coverage of the most common smartphone browsers. Run through all of your tests in each of these emulators, making note of layout and interaction differences.
Test Goals
In functional testing, one of the core tests run was standards compliance (XHTML validity) for each page (see the second article in this series). It's important to perform device-based testing, because not all browsers interpret valid XHTML in the same way. In addition, some non-compliant XHTML might slip through testing, and you'll want to understand how devices render your pages even when non-compliant XHTML is presented. This is critical because different device browsers react differently to non-compliant sites. Some browsers will cut off display when a non-compliant command is encountered. Some browsers make their best guess at the appropriate display and carry on. Some browsers just present the command as inline text.
Another reason emulation testing is important, though, is that not all device browsers render even valid markup in the same manner. Some devices get confused on and text wrapping; you'll want to catch this in testing rather than being notified by a customer!
So to conclude, you can accomplish a lot by testing your site in emulators. Some quick testing can be done in layout emulators, but your best testing will be accomplished in a variety of device emulators, which actually run an instance of the device OS. This phase of testing will produce mostly layout-oriented defects, and few functional or server-side issues.
