This content is part of the Essential Guide: Simplify the production deployment process with these tips
Evaluate Weigh the pros and cons of technologies, products and projects you are considering.

Why testing in production is going to be the next big thing

At a time when manual testing is feeling the squeeze, it's time to think of alternatives. Expert Gerie Owen explains why synthetic testing is going to be important going forward.

Testing in production? Really? What is testing in production, and how are traditional testers involved? Testing in production -- sometimes called synthetic testing-- almost always applies to web applications, to ensure correct behavior, performance and availability over the production use of an application. Testers, welcome to the brave new world of synthetic testing!

Synthetic testing is not a lot different than automated testing in general. You record tests by hitting web pages and entering data to simulate common transactions, and then run them periodically. The goal is to make sure that these applications are available when they need to be and perform up to the standards of the provider. These tests also serve to verify that the results of each step are what you expect.

One difference is that these tests are run while the application is live and actively servicing customers. Among the application performance monitoring vendors that provide software and services for testing in production are Dynatrace, Catchpoint Systems and New Relic. You can use such testing services from points of presence around the world, simulating actual customers or other users working from where they might normally be.

You can also use these tests to create load on the application to determine if increasing load affects performance. By generating many test instances of different tests around the world, these tests can simulate user behavior far more precisely than in a lab.

Synthetic testing results are usually displayed on a dashboard in an organization's network operations center or similar monitoring facility. Testers run their tests throughout the world and throughout the day, and watch for signs of unusual behavior.

Testing in production isn't going away

As you might imagine, testing in production is controversial among traditional testers. There is a strong culture that testing is best performed before an application reaches production. Once it has completed preproduction testing, testers have more or less ensured that the application contains no significant defects, and that it will behave as intended in production.

However, there is an inability to capture the production environment in preproduction testing, which causes failures that only manifest themselves once they interact with actual users, in different locations. If deployment is to one or more cloud instances, for example, behavior of the clouds used can affect quality. It occasionally happens that entire cloud data centers have performance slowdowns or go offline entirely, and engineers have to quickly reroute traffic to secondary locations.

Network engineers need alerts to warn them of a wide variety of occurrences, so that they can take action. Alerts may include notifications if a test or individual page load takes significantly longer than usual, or if a test does not complete properly and return expected results. Engineers also get graphs aggregating performance and availability information. A single failure usually isn't an immediate cause for concern, but a pattern of failures or trends in increasingly slow performance is usually the first indication of an application issue.

The bottom line is that testing in production is going to continue and grow as long as applications and the internet become more complex.

Further, synthetic tests are likely to also identify sporadic problems with the network, meaning the internet, of course. Organizations can do a number of things to mitigate network availability and performance problems in production, including load balancing, network routing and secondary domain name system services.

Therefore, testing in production is critical to ensure the overall quality of the application. As champions for the customer and guardians of the company's reputation, it is an important part of our role as testers to promote the importance of this type of testing.

The testers who actually perform this testing are usually network engineers with skill sets to understand how server farms and networks work. What they lack is an understanding of the purpose of the application and the steps users would take in performing an action. Also, they may not have a good picture of how much traffic is generated in different parts of the world.

Traditional testers can work from requirements and user stories to devise tests in production that more accurately measure real user actions. They can also develop tests that are designed to test weak points and possibly break the application. Through their intimate understanding of what results are supposed to be generated, testers can be a vital resource in localizing a problem based on the behavior of individual tests.

The bottom line is that testing in production is going to continue and grow as long as applications and the internet become more complex. And for testers, this is both a challenge and an opportunity. We have the opportunity not only to learn new skills if we choose, but also to employ our traditional skills in new ways. Testers who embrace this type of testing will play a significant role in the quality of their organization's applications, and the experience of users, in the future.

Next Steps

A round up of application performance monitoring offerings

Understanding application performance monitoring

How a network engineer will fit in to a DevOps team

Dig Deeper on Non-functional testing

Join the conversation


Send me notifications when other members comment.

Please create a username to comment.

What skills would you need to add to your portfolio in order to tackle synthetic testing?
That's a very interesting proposition which I think is being performed currently under the category of Application Performance Monitoring (APM)
One APM method is, exactly as described, to craft a range of typical transactions and then to have them performed by client engines potentially right around the world. These “synthetic” transactions are performed and measured constantly and, depending on the customer may turn into (NOC) alerts that a transaction is not performing that well.
Before that all takes place however the application needs to be good enough to be "released" into production. For most companies this means a thorough set of functional tests must be performed in a test environment. These tests might be manual or automated but either way they are a prerequisite to allowing an application to be released.
The problem, as outlined in the article, is that for a lot of companies that's it: The application is rolled out with little performance and non-functional testing, making customer-users unwitting beta testers.
Performance testing usually implies creating a load (many simultaneous users transactions) and this can be done in the test environment using load generators.
However, non-functional testing is usually environment based, and one of the greatest variables here is the differing network capability/quality (as alluded to in the article) experienced by real world customers. At first it would seem that the only good way to do that kind of testing is in the real world (production environment), but, in fact, companies like iTrinegy produce special kinds of network simulation product (called network Emulators or virtual test network products) that actually allow you to recreate fully operating "real world" like networks in the test environment.
We recommend that after the "pure" functional tests are completed and then the same functional tests are repeated (re-purposed - to save time) in the Emulated real world network environment. This means that any bugs related to real-world networks (as opposed to perfect test networks), that are not performance (load) based can be found before Production. This is known as Operational testing.
Equally Operational testing can be applied in a performance testing environment to make different loads appear to come from different quality networks, and also potentially geographies.
So by conducting Operational testing we make the kind of testing in the production network described an ongoing “monitoring process” on products that are already well tested (Operationally) for the required real world Environments, in particular: Networks.

Performance testing in production - if it is just a search or query is fine ,  but doing an insert or adding new data to a production database, is going to be a problem.

if it is an existing system that has been around for a long time, it may cause misreporting of  your revenue.  unless you have a way to filter all those test data out  so that they don't be used during true production financial reporting or any reporting per se.

also I don't know how it will sit with audit and your investors or stock holders, especially if you are a fortune 100 company, and are traded on the stock exchange, as there is a high risk there, for committing fraud.
Okay, let's talk about risk. Synthetic testing may add a tiny percentage to your application traffic, but if you are transparent on the number of tests being run, any auditors (and I doubt that many companies truly audit the validity of traffic) would be satisfied. There may be some very minor revenue implications based on increased cloud usage.

On the other hand, I think there is a much more significant risk in not knowing that your application is down, or a critical feature is broken, or performance has degraded, until you start getting killed by your users on Twitter. There are companies that have gone out of business because they were perceived as being unreliable.

And remember that it may not even be the fault of the application itself. The cloud data center may be having an outage, or you may be subject to a DDoS attack, or there may be DNS routing issues. You can work around all of these, but you need to know about them first.

I think that most companies that make money (or save money) off of web applications would invest in synthetic testing/monitoring rather than deal with the very real risks that they could lose a lot more money if they don't. And this is fraud? Please.

I agree with this concept as a means to quickly identify issues that will only occur in PROD, for which many organisations cannot fully replicate in a PRE-PROD environment, - the theory here being that it is preferable to catch out defects before your customers do, or at least minimise the time to deploy a fix.

Another consideration I have found when attempting to do any sort of Production Verification Testing is the requirement to have test 'production' accounts set up and approved by the organisation's security and Executive.  Any test transactions generated in PROD will affect financial, taxation, profiling, sales and marketing reporting etc and have subsequent implications for the organisations business areas and possibly government or jurisdictional reporting, - unless the 'production test' accounts and transactions can be specifically identified and accounted for, so as not to muddy the waters inadvertently.

Also from experience, it is not easy to get such production 'test' accounts set up and approved, not the least of which due to the fact it is not commonly understood how to control and manage the risks.  This requires significant effort and championing from senior ICT management.

Good discussion