Magic formula for successful performance testing

I'm sometimes criticized for taking a more heuristic approach to performance testing as opposed to a more mathematical one. My response has always been that I'm biased toward a commercially driven environment where even heuristic approaches frequently take more time, require more information and need specialized expertise the teams don't have. Adding complex mathematic equations that require more time, fewer unknowns and (in many cases) exceed the mathematic training of all of the performance testers in the team simply isn't likely to make matters better.

Even so, I think I have finally come up with a significantly complex mathematic formula for my critics:

For those of you who have forgotten (or never had a need to learn) integral calculus, the formula roughly says this:

The maximum value for the formula within the function is 100; the minimum is 0. Based on my experience, a good score would be 50, and a typi...

RELATED CONTENT Peak Performance Cut software performance testing costs with built-in measurements Building a Performance Assurance Center of Excellence tutorial Running your first load test with JMeter Ways to approach application performance testing on a tight budget Budget-friendly Web app performance testing, monitoring tips Testing rich Internet applications: 2009's best free tools The controversy surrounding the schools of software testing Testing training: Disturbing behaviors of students Software testers are not helpless Software testers must understand the business side of software quality Software performance, load and stress testing Cut software performance testing costs with built-in measurements Running, debugging and analyzing load tests using JMeter Application performance testing across company networks Testing rich Web services with soapUI Resolving issues in baseline, load and stress testing Testing Web services' performance with soapUI Automation Anywhere tackles automated testing Building a Performance Assurance Center of Excellence tutorial Tips for debugging your JMeter tests Exploratory testing: Addressing misunderstood concepts and common challenges

RELATED GLOSSARY TERMS Terms from Whatis.com − the technology online dictionary integration testing  (SearchSoftwareQuality.com) performance testing  (SearchSoftwareQuality.com) shotgun debugging  (SearchSoftwareQuality.com) stress testing  (SearchSoftwareQuality.com)

RELATED RESOURCES 2020software.com, trial software downloads for accounting software, ERP software, CRM software and business software systems Search Bitpipe.com for the latest white papers and business webcasts Whatis.com, the online computer dictionary

cal score would be 10. "+/- X" acknowledges that whatever the function predicts could be wrong.

Brilliant, huh?!?

OK, I admit it. I'd never use this formula. In fact, the entire notion of a magic formula is intended as a parody. I came up with this while reviewing a paper from an IEEE conference. The paper is now several years old and was related to the performance of software systems. I'm not going to name the specific paper because it's not important, and I don't want anyone to think I'm trying to attack it. I'm not.

Formulas that look (and to me feel) like the one above are all over books and papers that claim to be written for performance testing practitioners. While I know that there are some performance testing practitioners who work in environments where formulas that look like this are not only valuable but also essential to their job, my experience suggests that these are the minority.

That being said, the formula is neither shallow nor is it entirely without merit. If we look closely, it calls out many of the reasons that performance test projects aren't as successful as we would like them to be. Consider the following points that the formula makes:

• The single most effective way to improve your chances of success is with a knowledgeable and experienced team.

• Consistency in the team, tools, processes, etc. over the life cycle of the project (especially if they are consistently high) is crucial.

• Having no application to test (or limited availability of the application, or the application being so broken that most performance testing is either impossible or irresponsible) can destroy efficiency.

• Good tools that the team knows how to use can make a significant positive difference.

• Bad tools that the team is forced to use, but doesn't know how, is worse than having no tools at all.

• Management, processes, business knowledge, environments and real end users can all help or hurt some by themselves, but none of them can make a huge impact either way on their own.

• No matter what we do, there will always be an unexpected "X factor" that crops up during the project that, at least temporarily, can dramatically impact our previously estimated success potential.

Feel free to use my formula if it helps you communicate the relative importance of some aspect of performance testing to your team. If you use it, remember that every input value (with the possible exception of time and the number people on the team) is an estimate that you created, making the formula no more accurate than your input data. I suspect that you'll get more value out of using the formula to demonstrate that even the most complicated looking formulas won't relieve you of the need to think.

----------------------------------------
About the author: Scott Barber is the chief technologist of PerfTestPlus, vice president of operations and executive director of the Association for Software Testing and co-founder of the Workshop on Performance and Reliability.