EXPERT RESPONSE
There are a lot of ways to use virtual environments in your performance testing, so there's no easy answer to this question. I'm assuming that you're referring to hosting the entire application in a virtual environment and running your performance testing against that platform. My answer is that, as always, it depends.
Some research on the topic has found that virtual environments don't scale as well as non-virtual environments. In a study by BlueLock, a company that provides IT infrastructure as a service, they found that "the number of simultaneous users that could be handled by the virtualized server was 14% lower than the number of simultaneous users being handled by the traditional server configuration."
This is consistent with my experience testing financial service applications in virtual environments. Scott Barber, fellow SearchSoftwareQuality.com expert, recently gave a talk on the topic. You can find the PowerPoint slides for the talk here. In those slides he points out some other common challenges with working in a virtual environment.
If you don't have much choice, or if you have a lot of pressure to make it work, I would recommend that you perform a comparison performance test to prove out the new platform. If you can do that successfully, you'll have some confidence that the platform is comparable. But just be aware that over time, as the application changes and the server configurations change (both virtual and the physical servers in production) your comparison will become outdated. It may happen faster than you might think.
As Scott points out in his talk, the problem isn't necessarily virtualization. It's that we don't always pay attention to all the other factors that affect performance. Differences in software and hardware configurations, network devices, geographic location, firewalls and other security measures, and a host of other factors all affect performance. Virtual environments often only make it more complex to track everything since they introduce their own overhead, rely on different network devices, and can reside in different physical locations.
It's not all doom and gloom. You might be able to virtualize some parts of your application quite successfully. For example, at the April Indianapolis Workshop on Software Testing, Ken Ahrens from iTKO shared an experience where he used the iTKO LISA product to enable service-oriented virtualization -- a process where you virtualize services that your application might rely on. In this case, if you were performance testing the core application and not the services it relies on, then that virtualization wouldn't necessarily affect your performance testing at all.
|
|
 |
In that specific case study, before virtualization Ken's customer was unable to run performance tests on a regular basis due to service availability. Testing was an "event" where tens of teams had to get together to make a load test happen, and the high cost meant that this only happened a few times a year. Since they virtualized some of those services, they can run tests daily. Virtualization at the message level also gave them a greater ability to experiment with issues such as "what if this key service slows down significantly?" Or to try different data scenarios, such as "what if the lookup returns 900 records instead of 10 records?"
As you implement virtual environments in your context, pay close attention to the implementation. It may not matter to you if you see a 14% decrease in what you can support in that environment. For some teams, that's a reasonable risk. Before you decide to completely throw out virtualization, ask yourself if there are specific uses for it that make sense, like in Ken's example.
|
