Performance testing can involve quantitative tests done in a lab, or occur in the production environment in limited scenarios. Typical parameters include processing speed, data transfer rate, network bandwidth and throughput, workload efficiency and reliability. For example, an organization can measure the response time of a program when a user requests an action or the number of millions of instructions per second (MIPS) at which a mainframe functions.
Why use performance testing?
An organization can also use performance testing as a diagnostic aid to locate computing or communications bottlenecks within a system. Bottlenecks are a single point or component within a system's overall function that hold back overall performance. For example, even the fastest computer will function poorly on today's web if the bandwidth is less than 1 megabit per second (Mbps). Slow data transfer rates might be inherent in hardware, but can also result from software-related problems, such as too many applications running at the same time or a corrupted file in a web browser.
Performance testing can help identify the nature or location of a software-related performance problem by highlighting where an application might fail or lag.
Performance testing can also verify that a system meets the specifications claimed by its manufacturer or vendor. The process can be used to compare two or more devices or programs.
Performance testing metrics
A number of performance metrics, also known as key performance indicators (KPIs), can help an organization evaluate current performance compared to baselines.
Performance metrics commonly include:
- Throughput: how many units of information a system processes over a specified time;
- Memory: the working storage space available to a processor or workload;
- Response time, or latency: the amount of time that elapses between a user-entered request and the start of a system's response to that request;
- Bandwidth: the volume of data per second that can move between workloads, usually across a network;
- CPU interrupts per second: the number of hardware interrupts a process receives per second.
These metrics and others help an organization perform multiple types of performance tests.
Types of performance testing
Load testing helps developers understand the behavior of a system under a specific load value. In the load testing process, an organization simulates the expected number of concurrent users and transactions over a duration of time to verify expected response times and locate bottlenecks. This type of test helps developers determine how many users an application or system can handle before that app or system goes live. Additionally, a developer can load test specific functionalities of an application, such as a checkout cart on a webpage. A team can include load testing as part of a continuous integration (CI) process, in which they immediately test changes to a code base through the use of automation tools, such as Jenkins.
Stress testing, which has two sub-categories called soak and spike testing, places a system under higher-than-expected traffic loads so developers can see how well it works above its expected capacity limits. Stress tests enable the software team to understand a workload's scalability. Stress tests put strain on hardware resources, such as CPUs, memory and hard disks and solid-state drives, to determine the potential breaking point of an application on these finite resources. System strain can also lead to slow data exchanges, memory shortages, data corruption and security issues. Stress tests can also show how long KPIs take to return to normal operational levels after an event. Stress tests can occur before or after a system goes live. A kind of production-environment stress test is called chaos engineering and there are specialized tools for it. An organization might also perform a stress test before a predictable major event, such as Black Friday on an e-commerce application, approximating the expected load using the same tools as load tests.
Soak testing, also called endurance testing, simulates a steady increase of end users over time to test systems' long-term sustainability. During the test, the test engineer monitors KPIs, such as memory usage, and checks for failures, such as memory shortages. Soak tests also analyze throughput and response times after sustained use to show if these metrics are consistent with their status at the beginning of a test.
Spike testing, another subset of stress testing, assesses the performance of a system under a sudden and significant increase of simulated end users. Spike tests help determine if a system can handle an abrupt, drastic workload increase over a short period of time, repeatedly. Similar to stress tests, an IT team typically performs spike tests before a large event in which a system will likely undergo higher than normal traffic volumes.
Performance testing tools
An IT team can use a variety of performance testing tools, depending on its needs and preferences. These tools are examples of the options available.
Jmeter, an Apache performance testing tool, can generate load tests on web and application services. Jmeter plugins provide flexibility in load testing, and cover areas such as graphs, thread groups, timers, functions and logic controllers. Jmeter supports an integrated development environment (IDE) for test recording for browsers or web applications, as well as a command-line mode for load testing Java-based operating systems.
LoadRunner, developed by Micro Focus, tests and measures the performance of applications under load. LoadRunner can simulate thousands of end users, as well as record and analyze load tests. As part of the simulation, the software generates messages between application components and end-user actions, similar to key clicks or mouse movements. LoadRunner also includes versions geared toward cloud use.
NeoLoad, developed by Neotys, provides load and stress tests for web and mobile applications, and is specifically designed to test apps before release for DevOps and continuous delivery pipelines. An IT team can use the program to monitor web, database and application servers. NeoLoad can simulate millions of users, and it performs tests in-house or via the cloud.