Coding a mobile application is fun and seeing active usage is even more fun. But going through the release process...
is often a nightmare. Mobile applications not only have several more release gates than Web applications do but quality is more critical because bugs are more apparent and the time to resolution is longer. So, choosing how you test your applications is critical.
The testing options also are numerous. You can do real device testing in a local lab, use emulators locally, try emulators in the cloud, or even physical devices in the cloud -- all of which can be scripted or manual. Here I discuss the benefits of each and suggest that perhaps it is a cocktail of all that could make the ideal mobile application testing environment.
Which testing method you choose is based on several variables, including the size of your team and the nature of your application. Team size will determine how you use both testing methods, as larger teams require testing to be shared and not local. And application type will determine if your device has a unique physical functionality, such as the use of a microphone, which makes real devices a must.
Most shops will be able to find real device testing locally. Usually those labs have a small collection of market-leading devices where the application is loaded and tested. The test is normally run manually and not through a specific process. And developers will also use emulators to do quick sandboxing and testing on their local machine. Both of these make exploratory testing easy and convenient for ad-hoc non-repeatable testing. They're also a good choice when moving small tests very early in the pipeline or even at the feature concept stage.
Sandboxing is not where quality ends
But beyond sandboxing and into full quality assurance (QA), the processes need to be more defined, consistent and measurable. This can only be done with testing environments -- either virtual or physical -- that can be provisioned and de-provisioned every time you run a test suite. The test suite itself should cover regression tests as well as tests of new functionality.
The trick is that in the mobile space, the number of variations of operating systems and browsers is many times larger than you face in Web applications. And that's just at the software layer. On the physical layer you have all the variations of devices and their functionality. For example, for applications that have enabled iOS Touch ID for authentication, that feature will only work and be tested on newer devices and must be omitted from older ones. But you have to test both to make sure the new Touch ID functionality does not break login functionality on older devices.
You can choose to build your own integration environment with a large grid of emulators or choose a cloud service that already has them built for you. These days the cost of building and maintaining your own grid in resources alone can be substantially more than the cost of a cloud service provider. Or you can buy time on or subscribe to a data center that has a massive collection of physical devices.
Development shops that want to be deliberate about their testing will face this decision no matter what. And it can be a battle that is fought among both developer and QA teams. The result, most of the time, in my experience, is that the decision is made as an "or" -- emulators or real device testing. But it seems to me the better answer is "and" unless you are part of a one- to three-person developer shop. Testing mobile applications on both emulators and real devices is the best combination of test depth, breadth and speed.
The benefit of emulators is that spinning up instances in just about any flavor can be done in minutes and at a very low cost. So, if a test fails, which it will, a re-run is just a keystroke away. However, emulators cannot test everything and it is conceivable that device-level variables, especially the cellular network or Wi-Fi capabilities, could have a big impact on the application.
Real device testing is as accurate as you can get. But getting your hands on one takes much longer and usually comes at a much greater cost. The breadth of device coverage is always smaller. The lag between a new device launch and its availability to test on is longer. And if a test fails, a re-run is a lot more effort.
But if you think about it, both offer benefits that are in-line with speed and quality. And if emulators and real devices are combined correctly they could stand to enhance the entire delivery chain.
Here is how I think it is done. First, you maintain local emulators and real device testing labs for quick testing and sandboxing. In your delivery chain you have a cloud-based integration environment that has a large grid of emulators to run your test scripts on. With this environment you do as many releases as you can get away with. Do not be afraid of breaking something. That is the point. Fail fast and find bugs fast.
As a release winds down, your backlog is clear and you are happy to call this version done. Send your application through your entire test suite with the addition of tests for the physical device to a real-device cloud. Think of it as systems testing. You are not only testing the code but also your application's behavior in weird scenarios like turning off Wi-Fi mid-data transfer, the sound quality over a microphone and so on. Your goal here is not to re-run any tests. You hope that all the circles on your testing dashboard turn green and you can submit to the app store(s).
Altogether, it looks something like Figure 1.
The setup is not tricky either. If you use cloud solutions for both -- which, considering the phenomenal task of building your own device grid, it would be silly not to -- what you are setting up is only the process. Wiring up the components takes far less time than choosing how to do so.
I've encountered the real device testing versus emulator debate many times. And, for the most part, it has been a battle of stubbornness. But if you really think about it, both emulators and real devices have their benefits. And you should not be using only one. Use emulators for fast and repetitive testing in a continuous integration fashion. Then use real devices for your end-to-end and systems testing up to delivery.
Looking for mobile emulator tools? Look no further
Mobile testing is not for the faint of heart
Our expert reviews five key mobile testing tools