Proving an application meets the bare minimum of its requirements is no longer good enough. Today's software testers also have to make sure that the application in production meets the users' increasing performance needs and expectations.
The question is how. How can testers, who have been charged with pre-production quality and verification responsibilities, make any predictions on how an application will perform in production? The answer is, application performance monitoring. This requires analytics, i.e., the ability to collect data on an application under test and use that data to understand the circumstances under which the application may fail.
This approach to testing is complex. It means that testers not only have to verify compliance with requirements, but also have to predict how the application will work in the production environment, and where it may fail in real use and under load.
Ultimately, the goal is to be able to understand an application's weaknesses and to be able to monitor that application in production to determine when those weaknesses might cause problems.
Ultimately, the goal is to be able to understand an application's weaknesses.
To deliver on this goal, testers today have to collect data on application performance and reliability below the GUI-level. Testers need component-level data on performance and reliability, as well as system-level response curves under user load.
System-level load characteristics tend to be easier to measure during testing. Testers can determine how many users it takes to break the application and how the response time varies based on the number of simultaneous users.
Testers can also dig deeper, using performance monitoring counters or other tools to look at how individual application components behave on a variety of measures, including memory use, CPU use and database access. Used together, these can provide information on which parts of the application are getting stressed as the user load increases.
As for correlating testing results with performance and behavior in production, an increasing number of DevOps teams are employing monitoring from the cloud, using services such as Compuware Gomez, Soasta mPulse or SmartBear AlertSite.
These services typically employ Real User Monitoring (RUM) to get accurate analytics on response times, HTTP and database errors, and other characteristics that can then be compared to synthetic testing results. RUM provides a valuable reality check on test results as well as predictive power on when an application in production may be in trouble. The same characteristics that indicate an application is straining during load testing are important to watch out for during actual use.
With or without collecting analytics, software testers must do some level of load testing, if only to determine whether or not it meets requirements. Collecting analytics allows testers to go beyond that and learn important things about the application that will facilitate decision making in production.
Testers may say that collecting data to make production decisions is outside of their responsibility and expertise. However, when they analyze the load and performance results in test, testers will quickly pick up on production issues that arise. Plus, finding and addressing performance problems is key to retaining customers, which is a software tester's core business value.
Moreover, as more and more organizations adopt Agile methodologies and challenge testers to transform their roles and responsibilities, app performance monitoring in production provides new opportunities. For all testers, it is critical to understand how and where they can add value to the application development and deployment process. By stepping up and showing that they can add this kind of value and develop new expertise if it doesn't already exist, testers can go where they have not gone before and solidify their leadership role in the Agile world.