As the use of mobile devices continues to grow exponentially each year, testing applications on mobile devices becomes increasingly critical. Applications must be tested for functionality, usability and, most importantly, for performance. After all, if potential customers have to wait too long for the application to load or if it takes too long to navigate between pages, they will quickly find another application. This makes real user monitoring (RUM) imperative. Real user monitoring tracks the length of time between the user's actions and the application's response, and often provides other features as well.
There are many factors that make performance testing incredibly complex. The most important is the sheer number of devices, platforms and browsers -- so many of which have large user bases. All of these devices have differences in processing speed, memory size and communication protocols. The number of network providers adds to the diversity. Given the priority and the complexity of this testing and its importance in application lifecycle management (ALM), the choice of tools becomes a critical decision.
One of the most important factors in choosing a tool is the number and variety of devices and platforms that are supported. The cost of using actual devices versus device emulators can be prohibitively high. This is why an increasing amount of performance testing is being done by software as a service, or SaaS, performance tools. Many enterprises are unwilling or unable to commit dedicated resources to application testing and instead are turning to software in the cloud for testing new and enhanced applications.
Several vendors offer real user monitoring through their services. RUM provides the ability to look at actual traffic from mobile devices to web applications, including response times from those applications.
This approach is becoming increasingly important as mobile applications become more pervasive. Businesses and even government organizations are releasing mobile apps. They have to understand the performance implications of mobile endpoints on their server applications.
Several vendors offer real user monitoring through their services. RUM provides the ability to look at actual traffic from mobile devices to Web applications, including response times from those applications.
In evaluating RUM products and services, it is important to consider the following. First, monitoring should be taking place on real users using real devices. Second, the metrics produced should be easy to understand and be based on measuring what is most important to the business. Third, monitoring scripts should be developed on cross-platform technology. Some products that meet these requirements include SOASTA's mPulse, Compuware's APM and Perfecto Mobile's Mobile Cloud Monitoring. There may be others, but these are three that we have confidence in.
SOASTA's mPulse is a RUM product for monitoring Web and mobile applications. This tool not only supports real user monitoring for Web, mobile Web and native applications but also provides analysis and reporting functionality. It includes aggregation and reporting tools geared to analyze large amounts of data in real time, which makes reviewing performance metrics easy and understandable.
MPulse measures page load times from real users, so results can be used to optimize performance, and presents the data in a very slick graphical user interface. In July 2013, Gartner moved SOASTA to the rank of "innovator" in the Magic Quadrant and recommended the codeless automation environment and analytics of this product.
Real-User monitoring for Mobile and Web Applications from Compuware APM (formerly Gomez) captures complete user transaction interactions for every user. It supports monitoring Web, mobile Web, native mobile and hybrid mobile applications. It also provides a replay functionality that is quite useful in diagnosing the root causes of performance issues. Compuware APM's strength is its ability to track user experience by individual username. This is an exceptionally useful customer service feature.
Perfecto Mobile's Mobile Cloud Monitoring tool is part of its end-to-end mobile testing tool suite. Like SOASTA's and Compuware's products, Mobile Cloud Monitoring measures users' interactions across platforms and networks in diverse geographical areas. This tool also provides analytics and provides comparisons of performance to specific key performance indicators. The advantage of this tool is that monitoring scripts can be easily applied to new devices and platforms. In addition, Perfecto Mobile is one of the strongest tool vendors in the mobile market, and it has partnerships with HP, IBM and Microsoft.
Ongoing real-time user experience monitoring in production is vital to every mobile ALM program. Whether using one of the vendors listed here, an open source alternative or a homegrown solution, the choice of tools should be based on the following goals: First, monitor real user experience on real devices. Second, provide analytics in real time so that corrective action can be taken immediately.