WavebreakmediaMicro - Fotolia
Recent well-publicized Web-application outages, such as the HealthCare.gov website, have highlighted the need for ongoing application performance and load testing. This applies not only to customer-facing Web applications but also to server-based applications. In addition, development and testing groups have to take into account an increasing amount of smartphone and tablet traffic, which can behave differently from traditional clients.
An effective test strategy requires conducting sophisticated performance and load testing throughout the application development lifecycle using specialized tools. Fortunately, there are a variety of products available, both on-premises and in the cloud.
When looking at performance and load testing tools, IT professionals should consider the types of applications and operating systems a tool supports, the types of performance and load testing available and the scripting languages available.
The more applications the tool supports, the less likely it will be that IT will need to deploy more than one testing tool. The number and types of operating systems a tool supports depends on its clients rather than on the tool itself. The more types of performance and load testing a tool can do, the more comprehensive the testing IT can do on the application in development. If testing is complex, IT will have to use its own scripts. The more scripting languages the tool supports, the better.
A comparison of four of the market-leading tools -- Hewlett-Packard's (HP's) LoadRunner and Performance Center platforms, SmartBear's LoadUI Pro, Appvance's PerformanceCloud and Apache's JMeter -- against these purchasing criteria will help IT professionals determine the performance and load testing tool that is right for their environment.
All these tools are capable of providing a wide variety of comprehensive performance and load tests across multiple platforms and preparing comprehensive reports on the results.
Types of applications supported
Overall, HP's Performance Center works with the widest range of server-based applications, including thick client, n-tier, database and Web applications, although it can take an expert to develop the load test scripts and analyze the results. Performance Center utilizes server virtualization to test both browser and native mobile applications. It's a comprehensive and highly complex product.
The open-source-based Apache JMeter works primarily with Java and Java Server Page applications, but also supports Active Server Pages.NET and database technologies using Java Database Connectivity (JDBC) drivers. JMeter may be used to test performance on both static and dynamic resources. It can simulate a heavy load on a server, group of servers, network or object to test its strength, or to analyze overall performance under different load types. IT can use JMeter for mobile app load testing by opening the product, adding the HTTP Proxy Server component and setting a port to listen in to the component settings. IT also has to set the IP address as well as a proxy hostname.
If a company's environment primarily uses Web-based applications, its IT department may prefer one of the other performance and load testing tools. SmartBear's LoadUI Pro enables more casual load-testing sessions that require less expertise of users. LoadUI Pro lets developers record tests on their devices and use LoadUIWebto load tests to their mobile applications.
Appvance's PerformanceCloud also supports Web applications, and many of the technologies behind them. It has a large and sophisticated list of supported protocols, including HTTP, Simple Object Access Protocol (SOAP), Representational State Transfer (REST), enterprise service bus (ESB), and business process management (BPM). Appvance supports mobile Web apps through the mobile browser interface.
Operating systems supported
Because performance and load testing tools usually operate from a client rather than the server where the testing application resides, it is most important which operating systems the clients support. As long as the technologies used by the application to be tested are supported by the performance and load data collection engines (the clients), testers won't have significant issues collecting the data needed to evaluate performance. Overall, Performance Center, LoadUI Pro and PerformanceCloud support the greatest number of operating systems.
HP Performance Center runs from the Windows OS. It can run tests on any type of server, including Windows, Unix, Linux, Mac OS and even mainframes, since all it is doing is sending virtual users and collecting data on their processing. However, it is complicated to use on a Citrix server and requires installing an additional plug-in.
LoadUI Pro also runs from Windows, and can collect data on any type of server. PerformanceCloud uses a browser on any system for access and generally can run on a cloud or a cloud-type stack. It can work with any type of Web server and supports technologies such as HTTP, SOAP, REST, ESB and BPM.
JMeter can run on either Windows or Linux, and can test against any Web server. It includes support for JDBC databases and other types of Java-based middleware.
Types of performance testing
Performance testing involves looking at the response time of a single transaction. Load testing examines applications while being used by multiple users, usually approximating what is expected in production. Load testing is more complex than simply executing virtual user scripts; testers must consider user sessions, different user execution paths and different session IDs.
Stress testing looks at the number of users it takes to break the application. Soak testing involves running with multiple users for extended periods of time to see if there are long-term issues.
Performance and load testing tools can do a number of different types of testing, which makes them useful while an application is under development, because IT can find and fix excessive processing times or bottlenecks before the application is released.
All these tools will measure the response time of a single transaction, as well as perform load and soak testing. All of them can also perform stress tests. HP Performance Center is the only tool that has the inherent ability to dynamically add users until the application stops responding. With the other tools, developers must set the total number of users at a high-enough level to ensure the application will fail and apply those users gradually.
It is often possible to record and play back functional test scripts for continued performance and load testing. However, for more complex types of testing, performance testers sometimes have to write unique scripts to drive the process. This might involve scenarios that record and playback don't easily manage. HP Performance Center and Appvance's PerformanceCloud offer the most flexibility regarding scripting languages.
LoadUI Pro generally doesn't use a defined scripting language, but it's possible to build fairly complex load tests using its visual test creator. JMeter can be scripted using shell scripts or Java and Java-compatible languages.
Depending on a company's requirements, its tool selection is likely to reflect its unique performance and load-testing requirements, as well as its IT staff's testing and analysis skills.
HP's LoadRunner and Performance Center are good solutions for large organizations with dedicated performance engineers and many applications, or for consultants who need to specialize while working for many different types of clients. It is less effective for less specialized users who only occasionally have to conduct performance and load tests.
SmartBear's LoadUI is a capable tool even in its free version, although most serious performance testers will want the Pro version to do server monitoring and distributed testing. Its visual-test construction features make it appropriate for more casual or part-time testers, and its statistics and reporting are easy to use and enable both serious and casual testers to understand and display results.
Appvance's PerformanceCloud is a more recently developed testing product that supports a wide range of Web technologies. It's relatively easy to set up and use, and uses the same technology whether installed locally, in a private cloud or in a public cloud. It offers excellent data analytics and reporting capabilities, and would be especially useful for Web applications that use the latest technologies.
Apache JMeter is a bit more technical and difficult to use than some of the other options, but remains popular with more technical testers who like the ability to immediately download and begin using an open source solution. Its data collection and reporting tools are up to par, and it supports most major languages and Web architectures.
Learn about the changing test landscape
Discover how to set up performance load tests using Microsoft Azure