Two things top application performance manager J.P. Jackson's wish list: He wants a larger staff to oversee the thousands applications his team of six supports. And he wants developers to take on basic performance testing early in the application lifecycle. "If we required them to do testing [up front], we could avoid problems that come up later in production," said Jackson, a senior project manager for the state of Michigan. "Developers are always surprised when they see how [poorly] their applications perform under load."
Like many application performance managers, Jackson is making do with fewer resources than he needs. To keep ahead, he devised a plan that sets priorities for supporting the thousands of applications running on 4,000 servers that a dozen different development teams working in state offices across Michigan produced.
Priority one is to focus on applications where downtime could endanger lives. Priority two is to pay close attention to the performance of applications that generate much-needed revenue for the state. Third on his list is getting developers formally engaged in application performance testing to prevent performance problems from happening in the first place.
Getting developers to conduct scalability tests early in the application lifecycle is an important best practice, said Michael Azoff, principal analyst for UK research firm Ovum. "Using APM tools during development and testing enables teams to iron out problems at an earlier stage." He said software organizations with mature practices are doing this today. "But, in general, early performance testing by developers has not yet been widely adopted."
In this article, application performance manager Jackson explains his priorities and offers advice to other managers dealing with small staffs and limited resources. And APM experts weigh in on the value of early performance testing.
Focus on applications where downtime could threaten lives
Businesses use the term mission critical to refer applications that keep a company operating. But mission critical takes on a new meaning when lives, not just revenue, are at stake. The state of Michigan has identified 18 critical applications where downtime could threaten lives, including prisoner management, disease surveillance, newborn screening and offender management systems, Jackson said, offering a few examples. Monitoring these applications -- and investigating alerts that could indicate performance problems, or failure -- is top priority for his team.
Focus on apps that make money
Paying close attention to applications that generate revenue sounds more like a priority for big business than for state government, Jackson said. But focusing on revenue is vital to keeping the state of Michigan running efficiently. Two apps that Jackson's team keeps a close eye on: an online reporting system that lets businesses owners pay taxes, and a vehicle registration system that allows drivers to renew their car registrations without putting in an appearance at the department of motor vehicles. "If we don't have to see you face-to-face, it's a huge savings for us," Jackson said.
Narrow down troubleshooting time
Everyone tests for functionality, but no one tests for scalability until the last minute.
When problems occur, critical and revenue-producing apps take top priority, and Jackson's team works quickly to prevent downtime and restore performance. Like all the state's applications, these apps are monitored using Compuware APM. The software sends an email notification when it finds a glitch, specifying the condition that triggered the alert, Jackson said.
The team has relied on the Compuware offering for six years and over time Jackson has fine-tuned the APM software to make more effective use of staff members' time. Initially, he set up the software to issue an alert every time it encountered a glitch. But not all glitches turn out to be real problems. And that meant Jackson's team was spending time on low-priority issues that would have resolved themselves anyway. "You have to narrow down your troubleshooting time," he said. "You don't want [your APM software] to alert you too often."
A better approach is to specify more precise conditions that will trigger an alert. Jackson offered an example. "If the same error occurs in two different locations, ten minutes apart, you know you have a problem."
Help developers get smart about performance
By keeping performance problems at bay, Jackson can turn his attention to his ultimate goal: getting software developers to understand the value of performance testing early in the application lifecycle. That can be challenging to do, said John Van Siclen, general manager for Compuware APM business unit. "Everyone tests for functionality, but no one tests for scalability until the last minute."
Jackson takes a long-term view. He hopes performance testing by developers will become a standard, mandated practice throughout his organization. But in the meantime, he makes the most of his interactions with developers.
Right now, Jackson's team interacts with them on an ad hoc basis, typically when a production application runs into trouble. "We respond to requests from application owners who need help," he said. "They call us in, say [for example], remote users in Saginaw are experiencing long delays. We do triage activity with them, point them at a solution and then we do production monitoring."
Instead of just fixing the problem and moving on, he engages developers and gets them to see the big picture of how their applications interface with other applications.
Draw a map of the app
To get his point across, Jackson asks developers to sketch a diagram, mapping out the points at which each database and legacy system connects with their application. The process is illuminating, because more often than not, developers leave out a key component or two. "They discover that the application is actually accessing a system they didn't know about," Jackson said.
"There are no simple apps. They all interface with legacy systems, carrying out diverse activities to complete a transaction." That drawing activity helps developers understand the complexity of the network on which their applications are deployed, and Jackson hopes that will create awareness of potential performance pitfalls. "We have such a large network. Many apps are used remotely. Without load testing and simulation it is hard to get an idea of how they perform in the field."
If all goes as planned, developers will engage Jackson's team before their applications run into trouble in production. "Preproduction, we can prevent performance problems from happening in the first place," he said.
Do your developers do performance testing early in the application lifecycle? Tell us what you think.
This was first published in February 2013