How to test an application's scalability, performance

Software performance, scalability testing: Why do we test for performance? Use "SCORN" to test the front end of a website for performance What to include in a performance test plan

The traditional definition of scalability testing falls into what I call performance testing. A fantastic resource for performance testing is available on codeplex.com -- Performance Testing Guidance for Web Applications. I like how the authors approach performance testing, although I do simplify test categories a bit more than the authors do in this guide. I generally categorize performance testing into four groups:

1. Response time: This is the measure of time needed for a response to complete (from request to last response) -- at both normal and peak loads for a website. This is generally the user-perceived time, and in a Web application it should generally include the time to load html pages, css, jsp, as well as images.
2. Load: This is the measure of response time as load is increased from 0 to a "very large load." Generally the upper load limit will be a multiplication factor of the highest expected peak load. The key metric in this test is to watch how the response time increases as load increases -- are response time increases linear (as the load level increases, the response time increases proportionally) or logarithmic (as the load level increases, the response time increases more).
3. Mean time to failure (MTTF): This is how engineers generally predict the required maintenance cycle for an application. By estimating the number of transactions per minute, hour, or day for the application, you can extrapolate the number of transactions for a week or a month. The application should then be brought to a high (but stable) load rate. I generally shoot for 85% of my first bottleneck (CPU, RAM, disk I/O, etc.). The site is then allowed to run indefinitely, and the total transactions are recorded. From that number, you can predict how long the site will run given normal conditions. At Microsoft, we generally set a requirement of 30 days of simulated load; at 85% (or 90% to 95% on the server products I worked on), we could generally perform this simulation in about a week.
4. Performance tuning: The final effort is to tune the application. The site is placed under increasing load levels until a bottleneck is reached (site becomes non-responsive due to a downstream dependency). That bottleneck should then be fixed, either by fixing non-performant code (preferable) or by scaling up or out.

Few organizations seem to spend much time on these activities. However, if you've ever seen a site crash due to sustained high demand, you'll understand why load, MTTF, or performance tuning are required. And many organizations I've worked for have preferred to add hardware rather than truly investigate performance bottlenecks. This has always struck me as penny wise, pound foolish. The cost of a developer and tester spending a couple of days identifying non-performant code functions is nominal compared to the king's ransom most enterprise hardware and software vendors charge.

Performance testing really isn't a black art. It's a science, or it can be. It takes some amount of research, a lot of logic, and some experimentation. It's also a critical niche that few testers ever develop expertise in, so spending the time to learn how to do performance testing right can be very valuable from a career perspective. I highly recommend you take time to read through the Codeplex testing guide. Also look for additional titles by Scott Barber, such as his Peak Performance columns, and others who are experts in the field.

Finally, experiment -– even if you need to spend extra hours at work. Learn the ins and outs of your application. Show in raw numbers how your project performs, and develop the expertise to be able to accurately estimate the time to test and fix performance issues. You'll save your company money in the long -- and short -- run.