What to do when the test environment doesn't match production

Digg This!     StumbleUpon     Del.icio.us

RELATED CONTENT Software Testing and Quality Assurance Why do performance testers write new scripts so often? How to create performance testing workload models Fixing Web application performance troubleshooting problems Expert advises on implementation of Selenium IDE for effective software testing When should regression testing occur in an automated test plan? Achieving peak performance in integration testing Getting answers about OpenSTA script problems Defining core software regression tests Breaking in functionality on UI application pages Where to find good methodology guides for software testing Software performance, load and stress testing Performance testing tools - Commercial, less expensive and free Easing software performance testing and usability modeling pressures Software Testing: New software testing technologies bring new challenges Drilling deep into performance testing at STPCon STPCon: Do reality checks on performance test products, panelists advise Ways to approach application performance testing on a tight budget Data warehouse/BI performance testing tool recommendations Software testers facing six big challenges today, StarWest keynoter says Is manually testing a software project for flaws too risky? At the movies: Exploratory, performance, security testing a kiosk Advice from Scott Barber Performance testing SOA Acceptance testing for websites GUI automated testing projects How to set up a test environment Prioritizing software testing on little time Software testing processes and development methodologies Smoke and sanity testing Automating regression test cases Test plan and test strategy User acceptance testing that satisfies users and requirements

RELATED GLOSSARY TERMS Terms from Whatis.com − the technology online dictionary integration testing  (SearchSoftwareQuality.com) performance testing  (SearchSoftwareQuality.com) shotgun debugging  (SearchSoftwareQuality.com) stress testing  (SearchSoftwareQuality.com)

RELATED RESOURCES 2020software.com, trial software downloads for accounting software, ERP software, CRM software and business software systems Search Bitpipe.com for the latest white papers and business webcasts Whatis.com, the online computer dictionary

There is no formula or methodology that is any more accurate than a guess if you are starting from load simulations. In fact, an educated guess is probably the most accurate prediction in this case. For all practical purposes, most organizations have no way to extrapolate performance results from a test environment to predict production performance. Even the best performance testers in the field refer to this kind of extrapolation as "black magic" unless they've been trained by Connie Smith, Ph.D., or Daniel Menasce, Ph.D.

That said, I am a proponent of not doing all of your performance testing in the production environment because it is the most complex environment, obscuring bottlenecks and making them more difficult to pinpoint and resolve. I recommend starting in the most simple, limited, isolated environment possible and adding components only when the performance of the previous configuration is at least understood. This progressive approach saves a fantastic amount of time and effort every time you detect a performance issue.

Now, all this is not to say that I am against testing in the production environment, or an environment that mirrors production. What I am saying, however, is that should account for something like 10% of your total test time, which often means that it can actually be done in production during scheduled maintenance periods, thus saving the money of building a test environment that mirrors production and mitigating the risk of performance testing in production.

The fact is that extrapolation, no matter how scientific or accurate in a controlled, sterile environment, cannot account for someone installing a hard drive with a slower seek speed than planned for in the model, or for a mouse-chewed network cable, or for that one configuration setting that didn't get changed to allow the application to make use of the hyper-threading technology on the new hardware that the model had accounted for. Even Connie Smith and Daniel Menasce caution that their highly scientific and precise techniques are only as accurate as their models and that their predictions need to be validated via simulation before being viewed as anything other than best case scenarios.