There are few things in the life of an application developer more infuriating than an application that passes extensive...
testing and still breaks down shortly after going into production. At JavaOne 2013, Adam Bien -- who holds both Java Champion and Java Rock Star status -- explained that most testing doesn't happen under real-world conditions. According to Bien, the real testing fun starts after unit and integration tests prove the functions work on their own -- that's where software stress testing begins.
Unit testing gets straight to the heart of a function. Integration testing makes sure component A will play nice with component B. But both of those tests only really look at the code as it performs in the lab, so to speak. To find out how it will act in the wild, application development teams need to perform extensive stress testing on their software.
The problem, according to Java Rock Star Bien, is that unit tests and integration tests are generally run on single threads, but that's never the case out in the real world. "QAs are sometimes happy with 80% 'single thread' code coverage," Bien said, "but your application will never run on a single thread again after deploying it to an application server."
In defense of the quality assurance (QA) folks, that single-threaded coverage is really important. If a function doesn't work right in the lab, it has little chance of surviving in the wild. It's also much easier to figure out where the problems are when you can isolate each part and each integration. However, even after an application proves viable under isolated testing conditions, it still has to prove itself under high stress before it can be deployed as an enterprise application.
Why use stress tests?
Stress tests find flaws in the application that just wouldn't show up in a single instance. "Stress tests increase the probability of finding a critical concurrency bug or memory problem," Bien said. He also mentioned that the majority of problems he finds when he gets brought onto a problem project could have been found with straightforward stress tests. He recommended having fun with software stress testing. "Engage a student with the following mission: 'You have the whole night -- just break our server with [a] massive [work]load.'"
Adam BienJava Champion
If you don't have a student handy to pile on the server requests, the next best thing is automation. Bien pointed out that automation is nothing new and that continuous integration is just the reasonable outcome of IT's evolution. "Actually, the whole of IT is really about automation and streamlining," he said. "It's funny that [we developers] are not able to streamline and automate our own development process."
Continuous integration (CI) is just the logical way to maximize automation and minimize the potential for human errors, in Bien's view. These concepts may also be more common in the enterprise than you would expect. According to Bien, about 90% of his current Java EE projects are employing a CI server, even if they haven't fully automated the deployment process. The big advantage with automation is that stress testing -- as well as other types of software testing that may pop up in the future -- can be bolted onto the existing battery of tests.
Automated testing and continuous deployment may not be strictly necessary, however, especially for organizations that aren't really at the enterprise level. Continuous integration takes a degree of hardware investment that some organizations won't be willing or able to provide. Even at the enterprise level, it can still be difficult to obtain the right hardware. "Without any intervention," Bien said, "You'll get the worst-available machine for stress testing. Management often believes a CI server can run on a virtualized 486 CPU," he kidded.
Stress testing without the full CI package
While Bien noted hardware challenges as the top concern, there are others as well. Continuous integration requires a considerable time investment as well. Software stress testing is not only a QA team issue; developers and operations have to be involved, too. All those automated tests have to be developed and coded, and the hardware needs to be maintained. For organizations that aren't ready to make that leap yet, Bien recommends starting small. "Start with easy things," he said. "Don't try to be perfect; just start stress testing."
Remember that the important thing with stress testing a piece of software is that the application has to struggle to keep up with the load. Give it the biggest task you can think of, and then do it again and again. Don't stop to give the application a chance to recover; you're actually trying to break it. "Performing stress tests is fun," Bien said, "and you learn about the behavior of your application server under heavy load."
And if you still don't want to do any software stress testing, Bien said there is one alternative: "Put your application into production and pray."