Evaluate Weigh the pros and cons of technologies, products and projects you are considering.

Testing mobile apps: Tips on manual, automated, cloud QA

Testing mobile apps takes a multi-pronged approach. Savvy mobile developers are using cloud tools and automated tests in addition to manual QA.

Mobile application users don’t want to fuss with getting fixes for faulty or slow applications. Instead, they delete and upload a replacement app. Obviously, this common practice in personal use would wreak havoc in enterprise mobile application deployments, which puts software test and quality assurance directors on the spot, according to Eran Kinsbruner, director of product strategy at Perfecto Mobile, a software test ISV.

“The tolerance of the users is way lower than in the desktop era,” said Kinsbruner. “The end-users who adopts mobile applications have high expectations with regards to quality, usability and, most importantly, performance.” 

Eran Kinsbruner

In this Q&A, he discusses trends in mobile application development, such as testing, when manual testing is needed, cloud providers’ mobile test responsibilities and more.

What are the most common mistakes you’ve seen in testing whether an app will perform on real devices in multiple scenarios?

Eran Kinsbruner: Not focusing on poor application performance due to lack of application optimizations, and not taking into account behavior of different devices running on different mobile operating systems (OSes) in different network conditions. [You must be able to] simulate real native apps running on real mobile devices with specific mobile OSes.

Including manual testing scenarios is very important when it comes to mobile application testing strategy.

Eran Kinsbruner, 
Director of product strategy, 
Perfecto Mobile

What situations require manual mobile application performance testing, and why?

Kinsbruner: Including manual testing scenarios and interoperability testing are very important when it comes to mobile application testing strategy. When testing a mobile application, the testing team ought to test the various events which may occur when the application is being executed – Incoming calls, SMSs, low battery, alerts such as emails and roaming. Lately, along with growing technologies, mobile users also take advantage of location based and voice related apps – these rely on much more sophisticated use cases, which can also be tested manually (e.g. Applications like Google maps get input from end users about their locations and use that data for navigation).

What are some guidelines for making the manual-versus-automated test decision?

Kinsbruner: With regards to the differentiation between manual and automated test cases, the rules from the non-mobile space would apply here as well:

  • Automate the most frequently tested cases;
  • Automate cases which have predictable results;
  • For strong return on investment, automate the cases which are easy to automate;
  • Automate the most tedious manual test cases.

To the above, you need to take into account that your automation must run across devices and platforms, so make sure that your scripts are maintainable. Also, in mobile test automation, the solution you choose must support all types of objects.Finally, think about future regression cycles on older devices and mobile OSes as you move forward.

What common performance-related mistakes do organizations make when bringing mobile applications into the enterprise ecosystem?

Kinsbruner: The two big ones are not having rendering performed in the background, and not using application caching.

Are there particular device-related challenges?

Kinsbruner: Note that mobile devices have fundamentally different network performance characteristics. Device-based pitfalls, developers ought to take into account include:

  • CPU and Battery impact mobile performance;
  • Device Memory that repeatedly presents sections of application code will incur heavy memory operations - reducing performance and increasing power consumption, e.g., displaying an image on the screen that is stored on disk;
  • Device form factors and screen sizes can impact application behavior, performance, etc.;
  • JavaScript execution takes longer on mobile devices – Defer parsing until needed by the application;
  • DOM Objects are more complex in mobile; and,
  • Network constraints – Bandwidth on mobile ranges from 3G and slower to 4G and Wi-Fi.

Let’s say a business deploys mobile applications on the cloud or uses a cloud provider’s application? What is the cloud provider's responsibility in reporting on and maintaining mobile application performance?

Kinsbruner: The cloud provider’s responsibility is to enable sufficient coverage for the most critical end user aspects: devices, locations, networks and the top user scenarios. To these aspects, the cloud provider must also provide interoperability capabilities to assure end-user experience is maintained even when calls, SMSs, etc., come in.

Cloud providers give a secure enterprise grade cloud infrastructure and enable performance testing which covers the areas just mentioned. Also, once the customer has the load testing initiated, the cloud provider needs to provide all information and insights such as network traffic, device vitals and more, to allow the customer to perform optimizations and improve application performance. The customer has to define the right key performance indicators for his key transactions and of course the transactions themselves.

Is your team moving into mobile applications? Let us know what questions you have and what challenges you face. We're here to find you answers. 

Dig Deeper on Topics Archive

Join the conversation


Send me notifications when other members comment.

Please create a username to comment.

I'm not convinced automating is any more important for mobile software than it is for non-mobile. The stakes are high because it can take time to get a new version of an app out, but then again, not so long ago products were shipped out on a disk and it took time to get a new version of that out too.

Hi Justin, that's a good point. I don't really understand how, but I think mobile sort of led the way on understanding the importance of automating certain tests. I think we've definitely learned to do the same thing with Web applications and any software that we can potentially deliver continuously. Maybe it popped up in mobile faster because solid automation tools were already available when software innovators started into developing mobile apps and took a new look at testing (just a theory).
The article addresses a lot of mobile QA needs. Definitely, it was a good read. Delivering functionally accurate mobile application into the market and the customer experience of that application are the utmost important aspects.

Automation Testing plays a vital role in reducing effort and time in executing functional regression tests compared to manual approach; there is a need to agree that not all functional test cases can be automated. Automation Testing also helps in testing an application for functional accuracy across devices and platforms with reduced effort and time. Talking about mobile app performance testing, it has a lot of aspects in addition to web application performance testing. Just load testing won’t be sufficient in ensuring customer experience, testing application performance over different network service provider is essential. So, it has to be done across devices and platforms over different network conditions. Security testing is also an important one for any mobile app. Testing a mobile app for functional accuracy, performance, compatibility and security gives the competitive edge that a mobile app should have among the competitors.