Software testers trying to identify hidden performance problems in Java applications might be using the wrong requirements...
and the wrong tools, argued Gil Tene, CEO of Azul Systems, a real-time Java Virtual Machine vendor. Requirements specified in terms of statistical averages, he said, tend to focus engineers on the wrong problem. This problem is confounded by weaknesses in most performance measurement tools when assessing the worst-case performance of applications.
From a requirements perspective, most organizations tend to specify some statistical level of performance, such as 99.99% availability. Engineers then optimize the Java code to meet these specifications, without taking into account what occurs in the worst cases. As a result, Tene often sees significant decreases in application performance at specific statistical intervals, such as 99%, 99.99% and 99.9999% of average performance.
Tene noted that years of assessing what happens in the worst cases has shown that performance can be hundreds or thousands of times worse than acceptable performance. These problems are confounded when engineers in the rush to improve average performance don't take into account processes like garbage collection, database reindexing and virtual machine overhead. Often, in the worst cases, the result is decreased performance.
Another problem is that common load testing tools, like Apache JMeter, do a poor job of characterizing worst-case performance scenarios. Tene has found that when applications pause, these tools stop identifying the magnitude of performance degradation during the stall. This leads to a mismatch between test log results and real-world performance. As an example, a real-world stall of 26 seconds, as measured by JMeter, mischaracterizes application performance by a factor of 1,000 times.
To address such problems, Tene advocates that organizations always specify performance service level agreements by the worst-case scenarios. Tene also advises testers and developers to pay closer attention to what happens in worst cases by stretching out the display of performance testing data using tools like the open-source HdrHistogram, which he helped develop.
Recalibrating test and real-world environments
At least some of the performance testing challenges identified could be addressed by using virtual and cloud-based testing labs, argued Theresa Lanowitz, senior analyst at voke media, an analyst firm. Lanowitz recently released a report on the productivity and return on investment (ROI) benefits of virtual and cloud-based software testing labs. These labs make it easier for enterprises to set up testing that simulates real-world application performance.
For example, a load testing tool implemented on a single server has difficulty characterizing real-world performance from loads generated by masses of Internet users. Virtual and cloud-based labs make it easier for testers to simulate loads from a distributed cluster of servers.
Organizations see significant boosts in application delivery speed and higher software quality using virtual and cloud-based labs, said Lanowitz. She recommends that organizations consider adopting cloud tools.
Given the availability of virtual or cloud-based labs, Lanowitz said it is surprising that people are still talking about hardware issues. A developer might work on double-core machines for creating single-threaded applications, rather than multi-threaded applications. A virtual lab can simulate the physical hardware in this way. In essence, this virtualizes the hardware environment the application will be compiled and run on as required. She noted, "This eliminates the friction of back and forth communication between developers and testers."
Use app development time efficiently with Java inheritance
Check out these performance monitoring tools for Java apps
Java application development in the cloud