Both performance and load/stress tests help determine the capacity of a system. But for the tests to be successful, certain guidelines should be followed. David W. Johnson reviews those guidelines and offers advice for planning tests.
Performance tests and load/stress tests determine the ability of the application to perform while under load. During stress/load testing, the tester attempts to stress or load an aspect of the system to the point of failure. The goal is to determine weak points in the system architecture. The tester identifies peak load conditions at which the program will fail to handle required processing loads within required time spans.
During performance testing, the tester designs test case scenarios to determine if the system meets the stated performance criteria. Example: A login request shall be responded to in 1 second or less under a typical daily load of 1,000 requests per minute. In both cases, the tester is trying to determine the capacity of the system under a known set of conditions.
The same set of tools and testing techniques can be applied for both types of capacity tests -- only the goal of the test changes.
System test -- capacity testing
Scheduling capacity tests
Performance tests and stress/load tests occur during a system test or when enough of an application has been delivered. The earlier capacity testing can be applied, the earlier defects can be detected and mediated. It is critical to detect capacity-related defects early because those types of defects often require architectural changes to the system.
Because these tests usually depend on functional interfaces, it may be wise to delay them until function testing has demonstrated some predefined level of reliability. A balance must be struck between testing early and testing when appropriate.
Designing capacity tests
Lack of capacity testing requirements and goals is on
To continue reading for free, register below or login
To read more you must become a member of SearchSoftwareQuality.com
e of the most common errors made during any capacity testing exercise. Test organizations will often go through the process of capacity testing only to discover the testing results do not present the project with any useful information. The solution is to treat performance and stress/load testing the same as any other testing effort. The test organization needs to perform the following: test planning, partitioning/functional decomposition, requirements definition/verification, test case design, traceability (traceability matrix), test case execution, defect management and coverage analysis.
Implementing capacity tests
Both performance and stress/load testing require a toolset that can put the system under a known load and measure the performance of the application while under the load. Several shops have developed their own solutions for capacity testing. There are also several capacity testing freeware, shareware and commercial products available to meet this need. It is easy to fall into the "over-engineering" trap when implementing a capacity testing toolset, however. To avoid that trap, ensure the solution meets the test organization's goals and that the toolset does not become a "goal" itself.
Implementations
Capacity testing is performed against different aspects of a system. They may exist as standalone applications or as one facet of the system being tested, such as GUI, transactional and batch. The type of application or aspect of the system undergoing capacity testing directly impacts the approach, tooling and skill set required when performing either performance or stress/load testing. While it is possible to test several aspects of the system at once during capacity testing, it's best to focus the testing effort on one aspect of the system at a time -- even if the resulting throughput exercises other aspects of the system.
Graphical user interface (GUI)
When performing capacity tests against a GUI, the focus should be on the ability of the interface to respond to user input. The focus is on the interface, not on any transactional aspect of the system, such as security, middleware, databases and back-end processing
To best isolate the testing to the actual interface, ensure the remainder of the system is under little (if any) load and then perform the appropriate set of performance and stress/load tests. Interfaces are often ignored during performance and stress/load testing, but applying capacity testing techniques against the interface will often discover a host of defects that might not be detected during function testing. These can include memory leaks, memory management issues, object management issues and functional defects. This type of capacity testing can be performed using the same toolset that's used for automated function tests (i.e. record and playback).
Transactional
There are several types of transactional-based systems being deployed today: client/server, telephony, Internet-based, and so on. Each implementation or mix of implementations employs an architectural framework composed of hardware, firmware and software components. The complexity of the architectural solution can often add confusion to the task of capacity testing.
Testing the simplest and most common transactions first will often reveal the most critical defects. For example, the tester could start by implementing a test case that simulates users logging in to the system starting with one user and moving up to the number of users expected to be on the system during peak load. For stress/load testing, simply continue to add users until the system fails.
When testing organizations first attempt transactional-based capacity testing, they're tempted to feed a production mix of transactions to the system. But that often results in so much information being created that the root cause of any failure is difficult to determine. Keeping the transactional mix simple makes the task of defect analysis and remediation easier. Once the system can handle the simpler transactions, a more complex set of transactions can be added into the mix. The tactic is to move from simple to complex in a controlled manner to avoid data overload. This type of capacity testing can be performed using an appropriate load-testing tool or toolset.
Batch
Batch processing can be ad-hoc (user-requested) or scheduled. The circumstances around each type of batch processing are often very different and require different testing approaches.
Scheduled batch processing often occurs during off-peak hours or when no transactions are permitted. In this situation, the tester can focus on the ability of the system to handle large volumes of data with little regard to other aspects of the system (i.e. transactions).
Ad-hoc or user-requested batch processing must take into account other business that can occur on the system during the processing of the job, including other user-requested jobs. Once a set of ad-hoc jobs has been defined, the appropriate level of "background noise" must be created. It is often the combination of ad-hoc jobs and background noise that detects defects. Going from a simple set of batch processes to a more complex situation for batch and background noise will help avoid the issue of data overload.
Planning and staffing
A performance or stress/load testing engagement requires a team that has an in-depth understanding of the systems -- hardware, software, firmware, protocols, transactions and the business -- and load test engineers. Unless capacity testing is an ongoing exercise for the project/system owner, this type of talent does not reside in one organization. The test manager must work with his peers in operations, development, testing and any third-party IT providers to bring a team together that can address all aspects of the system.
Schedule conflicts and resource allocation are the two most common challenges when planning this type of testing effort. The test manager must be both opportunistic (What can be done now?) and risk-sensitive (What must be tested? What should be tested? What can be tested?). Unless capacity testing is a priority to the organization, even the simplest system will not be fully tested -- focus on the tests that will have the greatest chance of detecting defects and actually being executed.
-----------------------------------------
About the author: David W. Johnson is a senior computer systems analyst with over 20 years of experience in IT across several industries. He has played key roles in business needs analysis, software design, software development, testing, training, implementation, organizational assessments and support of business solutions. David has developed specific expertise over the past 10 years on implementing "Testware," including test strategies, test planning, test automation and test management solutions. You may contact David at DavidWJohnson@Eastlink.ca.
