Problem solve Get help with specific problems with your technologies, process and projects.

Release management: Software testing in production

When asked about testing during production, most people might initially recommend against it. However, upon closer examination, expert Pete Walen offers some insights into situations where testing may be helpful in production.

Should any testing take place in production?


I love this question. It is one that I find challenges many people's presumptions, not by asking if testing does happen in production, but around the propriety of testing in production. After thinking about this a moment, I kind of wish we were having this chat over a coffee or tea at a conference somewhere. As we are not, I will look at why this happens and see if this applies to your specific context. 


Most textbooks and software certification and standards programs will tell you that software should be "completely tested" before it is moved to production. Some manager types will tell people that the software must be "defect free." These two things would, by definition, tell us that the likelihood of finding a defect is something approaching zero. Frankly, I don't know how either of those things can happen in reality. 


In some situations it is possible to test every range of potential values in a given operation. However, this is not normally the case. To do so, you'd need to have a complete list of valid and invalid data elements in all parts of a system. For some controlled situations, for example floating point operations on a specific memory address, these values can be calculated and verified. However, carrying this into a full system is impossible. With this alone, we know that "complete" testing is impossible.


In those cases where the direct costs of continuing testing outweigh the benefit of testing in a controlled environment, sometimes software will be released in a limited form. This may be done as a "beta" program for commercial software or as a "pilot" or "Proof of Concept" program for in-house or specialized software. Now, in fairness, most people would not consider that "testing" or "production." However, it strikes me that when that decision is made, the distinction between "beta" or "testing in production" sometimes blurs. 


Of course, each organization's practices and tolerance for risk will vary. Systems that are life-critical, for example, medical devices or navigation and traffic control systems, will have far lower tolerance for risk than other systems. 


So, in short, Should any testing take place in production?  Most people will tell you "No." I suspect it happens far more than some will let on. I agree that it should not happen as a general rule. However, if there is no compelling reason for you not to do so in your situation, your context, and if there is no compelling business reason not to do so, you may find reasons why you should.


Dig Deeper on Topics Archive

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.