WavebreakmediaMicro - Fotolia

Evaluate Weigh the pros and cons of technologies, products and projects you are considering.

Compare software performance and load testing tools

Performance testing is a key part of the software development process. Learn the essential purchasing criteria for testing tools, and see how the top products measure up.

Recent well-publicized Web-application outages, such as the HealthCare.gov website, have highlighted the need for ongoing application performance and load testing. This applies not only to customer-facing Web applications but also to server-based applications. In addition, development and testing groups have to take into account an increasing amount of smartphone and tablet traffic, which can behave differently from traditional clients.

An effective test strategy requires conducting sophisticated performance and load testing throughout the application development lifecycle using specialized tools. Fortunately, there are a variety of products available, both on-premises and in the cloud.

When looking at performance and load testing tools, IT professionals should consider the types of applications and operating systems a tool supports, the types of performance and load testing available and the scripting languages available.

An effective test strategy requires conducting sophisticated performance and load testing throughout the application development lifecycle using specialized tools.

The more applications the tool supports, the less likely it will be that IT will need to deploy more than one testing tool. The number and types of operating systems a tool supports depends on its clients rather than on the tool itself. The more types of performance and load testing a tool can do, the more comprehensive the testing IT can do on the application in development. If testing is complex, IT will have to use its own scripts. The more scripting languages the tool supports, the better.

A comparison of four of the market-leading tools -- Hewlett-Packard's (HP's) LoadRunner and Performance Center platforms, SmartBear's LoadUI Pro, Appvance's PerformanceCloud and Apache's JMeter -- against these purchasing criteria will help IT professionals determine the performance and load testing tool that is right for their environment.

All these tools are capable of providing a wide variety of comprehensive performance and load tests across multiple platforms and preparing comprehensive reports on the results.

Types of applications supported

Overall, HP's Performance Center works with the widest range of server-based applications, including thick client, n-tier, database and Web applications, although it can take an expert to develop the load test scripts and analyze the results. Performance Center utilizes server virtualization to test both browser and native mobile applications. It's a comprehensive and highly complex product.

The open-source-based Apache JMeter works primarily with Java and Java Server Page applications, but also supports Active Server Pages.NET and database technologies using Java Database Connectivity (JDBC) drivers. JMeter may be used to test performance on both static and dynamic resources. It can simulate a heavy load on a server, group of servers, network or object to test its strength, or to analyze overall performance under different load types. IT can use JMeter for mobile app load testing by opening the product, adding the HTTP Proxy Server component and setting a port to listen in to the component settings. IT also has to set the IP address as well as a proxy hostname.

If a company's environment primarily uses Web-based applications, its IT department may prefer one of the other performance and load testing tools. SmartBear's LoadUI Pro enables more casual load-testing sessions that require less expertise of users. LoadUI Pro lets developers record tests on their devices and use LoadUIWebto load tests to their mobile applications.

Appvance's PerformanceCloud also supports Web applications, and many of the technologies behind them. It has a large and sophisticated list of supported protocols, including HTTP, Simple Object Access Protocol (SOAP), Representational State Transfer (REST), enterprise service bus (ESB), and business process management (BPM). Appvance supports mobile Web apps through the mobile browser interface.

Operating systems supported

Because performance and load testing tools usually operate from a client rather than the server where the testing application resides, it is most important which operating systems the clients support. As long as the technologies used by the application to be tested are supported by the performance and load data collection engines (the clients), testers won't have significant issues collecting the data needed to evaluate performance. Overall, Performance Center, LoadUI Pro and PerformanceCloud support the greatest number of operating systems.

HP Performance Center runs from the Windows OS. It can run tests on any type of server, including Windows, Unix, Linux, Mac OS and even mainframes, since all it is doing is sending virtual users and collecting data on their processing. However, it is complicated to use on a Citrix server and requires installing an additional plug-in.

LoadUI Pro also runs from Windows, and can collect data on any type of server. PerformanceCloud uses a browser on any system for access and generally can run on a cloud or a cloud-type stack. It can work with any type of Web server and supports technologies such as HTTP, SOAP, REST, ESB and BPM.

JMeter can run on either Windows or Linux, and can test against any Web server. It includes support for JDBC databases and other types of Java-based middleware.

Types of performance testing

Performance testing involves looking at the response time of a single transaction. Load testing examines applications while being used by multiple users, usually approximating what is expected in production. Load testing is more complex than simply executing virtual user scripts; testers must consider user sessions, different user execution paths and different session IDs.

Stress testing looks at the number of users it takes to break the application. Soak testing involves running with multiple users for extended periods of time to see if there are long-term issues.

Performance and load testing tools can do a number of different types of testing, which makes them useful while an application is under development, because IT can find and fix excessive processing times or bottlenecks before the application is released.

All these tools will measure the response time of a single transaction, as well as perform load and soak testing. All of them can also perform stress tests. HP Performance Center is the only tool that has the inherent ability to dynamically add users until the application stops responding. With the other tools, developers must set the total number of users at a high-enough level to ensure the application will fail and apply those users gradually.

Scripting languages

It is often possible to record and play back functional test scripts for continued performance and load testing. However, for more complex types of testing, performance testers sometimes have to write unique scripts to drive the process. This might involve scenarios that record and playback don't easily manage. HP Performance Center and Appvance's PerformanceCloud offer the most flexibility regarding scripting languages.

Performance Center uses primarily the C language for scripting. However, depending on the platform under test, it can also use Java, JavaScript or Visual Basic. IT teams can script Appvance's PerformanceCloud using Selenium, Java, Python, Ruby, Groovy, C#, Perl or PHP.

LoadUI Pro generally doesn't use a defined scripting language, but it's possible to build fairly complex load tests using its visual test creator. JMeter can be scripted using shell scripts or Java and Java-compatible languages.


Depending on a company's requirements, its tool selection is likely to reflect its unique performance and load-testing requirements, as well as its IT staff's testing and analysis skills.

HP's LoadRunner and Performance Center are good solutions for large organizations with dedicated performance engineers and many applications, or for consultants who need to specialize while working for many different types of clients. It is less effective for less specialized users who only occasionally have to conduct performance and load tests.

SmartBear's LoadUI is a capable tool even in its free version, although most serious performance testers will want the Pro version to do server monitoring and distributed testing. Its visual-test construction features make it appropriate for more casual or part-time testers, and its statistics and reporting are easy to use and enable both serious and casual testers to understand and display results.

Appvance's PerformanceCloud is a more recently developed testing product that supports a wide range of Web technologies. It's relatively easy to set up and use, and uses the same technology whether installed locally, in a private cloud or in a public cloud. It offers excellent data analytics and reporting capabilities, and would be especially useful for Web applications that use the latest technologies.

Apache JMeter is a bit more technical and difficult to use than some of the other options, but remains popular with more technical testers who like the ability to immediately download and begin using an open source solution. Its data collection and reporting tools are up to par, and it supports most major languages and Web architectures.

Next Steps

Learn about the changing test landscape

Discover how to set up performance load tests using Microsoft Azure


Dig Deeper on Topics Archive

Join the conversation


Send me notifications when other members comment.

Please create a username to comment.

Have you tried any of these performance and load testing tools? Let us know about your experiences, both good and bad.
Thanks for great article. From my point of view LoadUI isn't something that can be used for more or less complex test scenarios due to limited support of parametrization, pre and post processing. In regards to LoadRunner vs JMeter I'd suggest checking out Is JMeter a Viable Open Source Alternative to Loadrunner? post containing extended tools comparison information.
Hi Glinius, Thanks for your input. LoadUI does seem to be the sort of tool that trades a certain amount of power and complexity to make it easier to use. It seems like for enterprise-level testing the well-known options are HP LoadRunner (expensive but less labor intensive) or JMeter (less expensive but potentially labor intensive). Companies like BlazeMeter might help find a middle ground between the two by making it easier to use JMeter without costing as much as LoadRunner. That makes it basically a decision of which size fits the company culture and budget. Any thoughts on newer options like PerformanceCloud? Are they still somewhere on the same budget vs workload spectrum? Are they just a fad or could they offer a new way to think about performance management?
Hi Gerie, Nice article and summary of different performance testing solutions. I’m not sure if you were aware of the very recent addition to the HP’s family of performance testing solutions. StormRunner Load (https://bit.ly/StormRunnerLoad) is our new(in beta) cloud based performance testing service designed to help agile teams test their web and mobile apps. If you get a chance, please check it out.
FYI folks, John Jeremiah is a senior marketing manager at HP, so he's definitely in the know where HP is concerned. Thanks for the update, John.
While your article does an excellent job in explaining the needs for proper load and performance testing (and describes the various types of tests quite well), I think you've done a disservice to your readers by only listing the 4 tools that you have. There are a great many other excellent solutions that one could use to address an organization's testing requirements. And while there's no doubt that LoadRunner is the big player in the space, I suspect most experts in the field would not consider some of the others listed as “market leaders” in the sector. I highly recommend your readers to do their own research and see for themselves.
The solution that my company develops, NeoLoad, is one that should be considered by anyone who is searching for a tool for the load and performance testing of their web and mobile applications. It is far more productive than any of the tools that you’ve listed, allowing users to develop and customize their “user scenarios” in far less time than solutions that require intensive scripting, yet there is absolutely no loss in the ability to create the most complex user scenarios; meanwhile testing applications that use ANY web application technology.
Hi Steve, thanks for weighing in. You make a good point when you say that the four sellers in this article are not the only options and that no buyer should base their buying decision on a single article. We often include a sentence or two to say just that in our articles. In the "Dig Deeper" section, there should be an article called "Cloud services expand mobile performance testing" (https://searchsoftwarequality.techtarget.com/feature/Cloud-services-expand-mobile-performance-testing-tools). That article compares NeoLoad against SOASTA and Dynatrace (which were also not included in this particular article).
You can see all the main load testing solutions alongside real user reviews on IT Central Station: https://www.itcentralstation.com/categories/load-testing-tools

One of our most popular comparison pages in this category is HPE LoadRunner V Neotys Neoload. You can see the comparison page here: https://www.itcentralstation.com/products/comparisons/hpe-loadrunner_vs_neotys-neoload
What about like loadview-testing.com and loader.io?