Should any testing take place in production?
Requires Free Membership to View
When you register, you'll receive targeted emails designed to keep you informed of the most relevant information on Agile development, application security, testing & QA, software requirements, and more.
Hannah Smalltree, Editorial Director
I love this question. It is one that I find challenges many people's presumptions, not by asking if testing does happen in production, but around the propriety of testing in production. After thinking about this a moment, I kind of wish we were having this chat over a coffee or tea at a conference somewhere. As we are not, I will look at why this happens and see if this applies to your specific context.
Most textbooks and software certification and standards programs will tell you that software should be "completely tested" before it is moved to production. Some manager types will tell people that the software must be "defect free." These two things would, by definition, tell us that the likelihood of finding a defect is something approaching zero. Frankly, I don't know how either of those things can happen in reality.
In some situations it is possible to test every range of potential values in a given operation. However, this is not normally the case. To do so, you'd need to have a complete list of valid and invalid data elements in all parts of a system. For some controlled situations, for example floating point operations on a specific memory address, these values can be calculated and verified. However, carrying this into a full system is impossible. With this alone, we know that "complete" testing is impossible.
In those cases where the direct costs of continuing testing outweigh the benefit of testing in a controlled environment, sometimes software will be released in a limited form. This may be done as a "beta" program for commercial software or as a "pilot" or "Proof of Concept" program for in-house or specialized software. Now, in fairness, most people would not consider that "testing" or "production." However, it strikes me that when that decision is made, the distinction between "beta" or "testing in production" sometimes blurs.
Of course, each organization's practices and tolerance for risk will vary. Systems that are life-critical, for example, medical devices or navigation and traffic control systems, will have far lower tolerance for risk than other systems.
So, in short, Should any testing take place in production? Most people will tell you "No." I suspect it happens far more than some will let on. I agree that it should not happen as a general rule. However, if there is no compelling reason for you not to do so in your situation, your context, and if there is no compelling business reason not to do so, you may find reasons why you should.
This was first published in October 2011