Testing in production? Really? What is testing in production, and how are traditional testers involved? Testing in production -- sometimes called synthetic testing-- almost always applies to web applications, to ensure correct behavior, performance and availability over the production use of an application. Testers, welcome to the brave new world of synthetic testing!
Synthetic testing is not a lot different than automated testing in general. You record tests by hitting web pages and entering data to simulate common transactions, and then run them periodically. The goal is to make sure that these applications are available when they need to be and perform up to the standards of the provider. These tests also serve to verify that the results of each step are what you expect.
One difference is that these tests are run while the application is live and actively servicing customers. Among the application performance monitoring vendors that provide software and services for testing in production are Dynatrace, Catchpoint Systems and New Relic. You can use such testing services from points of presence around the world, simulating actual customers or other users working from where they might normally be.
You can also use these tests to create load on the application to determine if increasing load affects performance. By generating many test instances of different tests around the world, these tests can simulate user behavior far more precisely than in a lab.
Synthetic testing results are usually displayed on a dashboard in an organization's network operations center or similar monitoring facility. Testers run their tests throughout the world and throughout the day, and watch for signs of unusual behavior.
Testing in production isn't going away
As you might imagine, testing in production is controversial among traditional testers. There is a strong culture that testing is best performed before an application reaches production. Once it has completed preproduction testing, testers have more or less ensured that the application contains no significant defects, and that it will behave as intended in production.
However, there is an inability to capture the production environment in preproduction testing, which causes failures that only manifest themselves once they interact with actual users, in different locations. If deployment is to one or more cloud instances, for example, behavior of the clouds used can affect quality. It occasionally happens that entire cloud data centers have performance slowdowns or go offline entirely, and engineers have to quickly reroute traffic to secondary locations.
Network engineers need alerts to warn them of a wide variety of occurrences, so that they can take action. Alerts may include notifications if a test or individual page load takes significantly longer than usual, or if a test does not complete properly and return expected results. Engineers also get graphs aggregating performance and availability information. A single failure usually isn't an immediate cause for concern, but a pattern of failures or trends in increasingly slow performance is usually the first indication of an application issue.
Further, synthetic tests are likely to also identify sporadic problems with the network, meaning the internet, of course. Organizations can do a number of things to mitigate network availability and performance problems in production, including load balancing, network routing and secondary domain name system services.
Therefore, testing in production is critical to ensure the overall quality of the application. As champions for the customer and guardians of the company's reputation, it is an important part of our role as testers to promote the importance of this type of testing.
The testers who actually perform this testing are usually network engineers with skill sets to understand how server farms and networks work. What they lack is an understanding of the purpose of the application and the steps users would take in performing an action. Also, they may not have a good picture of how much traffic is generated in different parts of the world.
Traditional testers can work from requirements and user stories to devise tests in production that more accurately measure real user actions. They can also develop tests that are designed to test weak points and possibly break the application. Through their intimate understanding of what results are supposed to be generated, testers can be a vital resource in localizing a problem based on the behavior of individual tests.
The bottom line is that testing in production is going to continue and grow as long as applications and the internet become more complex. And for testers, this is both a challenge and an opportunity. We have the opportunity not only to learn new skills if we choose, but also to employ our traditional skills in new ways. Testers who embrace this type of testing will play a significant role in the quality of their organization's applications, and the experience of users, in the future.
A round up of application performance monitoring offerings
Understanding application performance monitoring
How a network engineer will fit in to a DevOps team