If there's one performance bottleneck that's harder to fix than all others, it's this: the one that originates in an application component owned by a third party. The IT organization didn't design, code or test the component and yet is responsible for its continual availability and performance. Completing a performance bottleneck analysis can help project managers identify what performance issues reside with third-party code and prove that third-party developers are responsible for fixing them.
It's tough to motivate outsiders responsible for various pieces of an application to take performance bottlenecks seriously. It's tough to get people who report to others to take action. Getting those pieces of the problem fixed is a challenge leadership experts call managing without authority. It's a challenge many project managers face.
In this edition of Quality Time, I share application performance experts' advice on analyzing and fixing performance bottlenecks that originate in third party components.
SLAs can't solve every performance bottleneck
The easy answer on how to improve partners' performance is to hold them to a service level agreement (SLA). But too often, these agreements aren't specific enough to produce effective results. They typically focus on desired response times and availability rates. Enforcing them requires precise data which pinpoints the source of the performance bottleneck. Otherwise, there's a whole lot of finger pointing going on. Who can really say where the performance bottleneck originates? Can't we just throw more hardware at the problem? These are common questions that partners raise when confronted with poor performance.
Show me the evidence
To end the finger pointing, present compelling evidence that proves a performance problem stems from the partner's piece of the application. "You need visibility into calls the application makes to third party systems," said David Jones, director of field technical evangelism for Detroit-based Compuware, which sells application performance software. "You can't just say: 'you guys are providing slow service.'"
Instead of winging it with anecdotal evidence, Jones recommends saying something like this: "We are monitoring this process, and seeing these response times, poor availability at these times, and the following exceptions are being thrown." Gathering that specific evidence requires some performance bottleneck analysis on the part of the IT organization.
He also recommended constructing an SLA around that data, and stipulating the penalties applied when performance lags.
Preventing performance bottlenecks
Educating partners about the importance of addressing performance early in the application lifecycle helps prevent these bottlenecks from forming.
"The reality is that lots of organizations don't value performance," said Theresa Lanowitz, who heads Minden, NV-based research firm Voke. "You have to plan for it, put in the time up-front, establish the proper architecture, and specify performance as a non-functional requirement." Too often, software teams don't do that. Lanowitz was quick to point out that performance shouldn't be something the development team has to figure out as they go along.
Lanowitz also recommends that organizations and their partners approach application performance as a continuous activity. This is especially important given the rapid changes occurring in mobile devices and operating systems, she said. "On the preproduction side, you have to take into account the entire infrastructure." Enterprise mobile applications are likely to be deployed on a variety of platforms, Lanowitz said, "You have to think about the iPhone, Android phones and the Blackberry, understanding how each mobile operating system impacts application performance."
Also important is teaching partners to fix poorly-constructed code that may be the root cause of a performance bottleneck, said Jones. "It's not uncommon for developers to take a short cut to meet tight deadlines. The code works, but it has performance issues."
He offered an example: A developer writes a piece of code that looks into a database to request information, and re-uses that same piece of code in different parts of the application. If they aren't careful, developers may find that the piece of code actually makes the request 100, or even 1,000, times, when one would suffice. "Obviously, that has a huge impact on performance," Jones said. Performance bottleneck analysis frequently points out this type of problem.
The performance analysis party
Another way to approach performance bottleneck analysis is to host an end-user testing event. This event can include partners whose components are part of the larger application. This approach helps everyone gain an understanding of how a complex, distributed application behaves when it's deployed.
"Organizations do extensive load testing. They test the application tier, the data tier, and everything gets green lights," he said. "But you don't [fully understand] the performance impact until it's rolled into production, until you put all these different components together."
An end-user test event is easy to launch. "Get employees to stay late, order pizza, test the app, and stress it out," Jones said.
Getting partners to pitch in to improve application performance bottlenecks is more difficult. It takes precise data to make your case, not to mention strong persuasion, influencing and negotiating skills. Even with the necessary leadership skills, most program managers will not be able to convince external developers without performing some performance bottleneck analysis to gather hard data.
Still, software quality engineers are good at managing without authority. They have to convince developers to fix code that doesn't work, and challenge business stakeholders to develop better requirements. Convincing partners to get their code to perform better is just part of the job.
Let us know how you persuade partners to improve performance and you might be reading about yourself in a future column.