At last count there were over 500,000 apps in the Apple app store, and over 300,000 in the Android marketplace,
not counting apps on other platforms like RIM Playbook, Microsoft Windows Phone and Windows Mobile. In addition, each of these platforms may be sold with one of their various operating system versions on the various devices. Developing apps requires testing and validating them on these combinations of various models and operating system versions. Automating the testing of mobile apps is the only way release schedules can be maintained. Here we explore challenges in automating the testing of mobile apps. A number of solutions currently available in the market are then identified and brief descriptions are provided.
Here are the challenges in automating testing of mobile apps:
- Need for automated scripting: The need for automated scripting is more acute when an app is released, being used, and subsequent updates need to be rolled out. All existing features need to be tested every time an update is being rolled out just to make sure that no regression errors have found their way back into the code with the updates. Also, with the variety of makers and models, especially with platforms like the Android, automated scripting, and testing becomes inevitable.
- Multiple language and environment scripting: Test scripting may need to be integrated back into the language and test environments usually used in an organization like JUnit, QTP, PERL or Python.
- Distributed testing: Mobile testing is increasingly being outsourced and even off-shored. The developers and the testers may be geographically separated. The test environment may need to deal with test executions from many locations around the world in multiple time zones, and using disparate local telecom service providers. The test environment may need to be available 24/7/365 and Internet/browser accessible.
- Post release automated bug and crash tracking: Users abandon mobile apps after just one or two crashes, and may even delete them. Mobile apps may need to be in beta testing mode internally for a while after the initial release on the app stores. Automated test tools may need to monitor and track bugs and crashes that may be missed during formal testing, at least for a while, even after a formal release.
- Test device registration management: Test device registration, especially for iOS devices, is a chore given that individual phone IDs may need to be registered with Apple. Install kits need to be emailed to testers for installation and testing. Automated test tools that manage the registrations smoothly and automatically make this process highly efficient and effective.
- Multiple phone model availability: Open mobile operating systems like Android have a large number of manufacturers selling mobile devices running various versions of those operating systems. Testing mobile apps in such cases require the availability of a large variety of device manufacturers and models for complete and reliable validation and certification.
- Processor limitations with simulators: Mobile phone simulators, when running on laptops or desktops, may be using other processors than the ones in the mobile devices. For complete and reliable testing, automated testing needs to be done on actual phones, not just simulators alone.
- Remote response testing: Mobile apps could be standalone on a mobile or frequently accessed via backend servers during execution. In the case of the latter, remote testing from multiple geographic locations may need to be part of the automated testing. This is to make sure that response times on the app are reasonable no matter where they are used from.
Here are some popular commercial and open-source automated test environments:
- TestFlight: TestFlight is an open source tool that makes testing on iOS platforms easy by automating the distribution of the app and provisioning files easy. You don’t need to maintain distribution mailing lists once you get the UDID of an iOS device and authorize that device for running a test version of your apps. Beyond that, TestFlight enables setting checkpoints in your app, asking questions of testers and collecting answers at those checkpoints, providing detailed test session logs, keeping track of app crashes, and also keeping track of which parts of the app are popular with users.
- SeeTestMobile: SeeTestMobile by Experitest is an automated scripting tool. You attach a smartphone using a USB cable to a laptop or a desktop running SeeTestMobile. You start the recording, use an app and then stop the recording. This automatically records a test script that can be run again, when a new release is available. The test script is run automatically on both the attached mobile device as well as on the simulator. Results are recorded in the SeeTest environment for analysis. The automated script can be converted and integrated with multiple other existing target test environments like QTP, TestComplete, MSTest, Junit, PERL and Python.
- RealMobile: RealMobile from Conflair builds a wrapper around common test automation tools like QTP and Quality Center and builds a custom environment for mobile testing. Since existing environments are used for mobile testing, integrating it with testing of other applications within the organization is easier. RealMobile provides testing on mobiles as well as simulators. Mobiles can communicate with the laptop or desktop using USB or wireless connections.
- TestCountdown: TestCountdown from BSQUARE is a modular test environment with separate design and execution environments. This enables the geographic separation of the test design and test execution activities. Test execution can also be done from multiple execution instances, and so multiple, distributed testing efforts are facilitated.
- FoneMonkey: FoneMonkey by Gorilla Logic is a free, open source tool that comes in separate versions for iOS and Android. The tool records a test execution as a script and can run it again in an automated way. Scripts from the iOS version can be exported into an OCUnit Test scripts for use with Objective-C, and those from the Android version can be converted into JUnit ones.
- Zap-Fix: Zap-Fix for Mobile generates scripts from recordings of manual tests and can be run again manually or in an automated way. It is integrated with HP QTP and supports WebOS from HP in addition to iOS, Android, Blackberry and Microsoft Windows Mobile.
- DeviceAnywhere: DeviceAnywhere for Mobile from Keynote Systems integrates with HP and IBM test environments. They also provide testing services from multiple locations across the globe for mobile apps that access backend servers for execution. This is to ensure that response times from the app are reasonable no matter where they are used in the world.
Automated testing of mobile apps is becoming essential as more apps enter the app stores and new features are rolled out as updates to existing ones. System, regression and user interface testing need to be rolled into these automated tests to ensure reliable, bug and crash-free execution. As these apps join the ranks of other enterprise applications, testing of mobile apps needs to be integrated with the testing and rollout of other applications. There are a number of commercial and open source tools that satisfy all these requirements and address mobile app testing challenges well.
About the author: Nari Kannan is currently the Chief Executive Officer of appsparq Inc., a Louisville, Kentucky-based mobile applications consulting company. Nari has over 20 years of experience in information technology and started out as a senior software engineer at Digital Equipment Corp. He has since served variously as vice president of engineering or CTO of six Silicon Valley startup companies, working in the areas of business process improvement, IT consulting, automotive claims processing, human resources and logistics applications. He can be reached at email@example.com.