An organization has decided to put an end to app crashes and disappointed customers for good and purchase an application performance management tool. Here comes the hard part: picking the right package. From platform compatibility, physical or virtual machines on to the cloud, data collection, and reporting, there are plenty of fundamental issues to look at including hard-to-measure items like ease to learn and support options. APM customers have to balance the features with team skills and price. It's unlikely that they'll find a single tool that does everything an enterprise needs, aligned with the skills the team currently has, at the right price. Instead, this article covers different aspects of software used to monitor application performance and how it affects one's decision on purchasing an APM tool.
Some products that monitor application performance specialize in just a layer or two in the stack, while others reach deeper. Selecting the right depth can help solve issues much faster. Some APM tools are also designed to run on specific technologies, such as Linux/Java or Microsoft/TFS. Tools that specialize in one or two technologies may have access to more debugging information more easily. It's important to pick a tool that matches the stack. Organizations with several different systems, and different stacks, might want a more general tool -- but those tend to have weaker integration.
Variety and dashboards
Complex data centers don't want to monitor one stack; they want to monitor all stacks. That might mean physical servers in one place, platform as a service (PaaS) offerings in another, some cloud applications and some pure-database systems in yet another. One common pattern is to have plug-ins that run and monitor on each system, then report back to a central server. From the central server, operations can create dashboards at different levels -- from red (crash), yellow (slow) and green (healthy) at the transaction level -- to detailed reports on each system. In some cases, it's possible to connect dashboards, so when a technical leader sees yellow, they can click in to find out that the delay is caused by one specific system that isn't meeting its service level commitment.
The right APM software will ideally cover most, if not all, of the platforms that are important to an enterprise and also integrate them into a holistic view of system performance.
Scheduling and data mining
Most hand-created monitoring systems were polling systems that ran on a schedule. Every five minutes, a hand-created script might wake up and ping every server in the network; if any response was slow, it would email an administrator. If a response timed out, the system would send a text message. This is known as taking the heartbeat of the system. Most APM tools continuously take the heartbeat of the system, creating a huge amount of data, which can be incredibly powerful.
Consider, for example, a system that is used during the business day. The "average" response time of a 24-hour day will include all the quick responses overnight when system traffic is low; it can also be thrown off by a few very slow responses at peak traffic times. What managers that are looking at this data likely want is the median response time of that eight-hour day only.
On the other hand, those few slow responses could be saying something. There might be a few peaks in usage during the day -- in the morning when everyone logs in, for example -- that could lead to lost sales and renewals or, for internal software, complaints and lack of productivity. Worse, the peak time could be the start of a trend that will only grow if the software is popular. If those are issues in your organization, dig into what features the APM software has for data mining, for looking at, say, the average of 10% of the worst responses. The ideal APM tool will proactively point out problems as they inch toward a services commitment breach, not after.
Collecting data is a big step to improving application performance; knowing what to do with it is another. There are lots of different ways to report and notify, but the first place company's employing APM software should start is deciding what they want to know about their product. To start, many people are interested in downtime, or periods of time where a product is not accessible, and extreme resource usage like high memory and CPU usage. These few things are the blood pressure and temperature of a product, vitals that show when something is right on or way off.
Software that can monitor application performance may provide many answers, but different APM tools answer different questions. It's important for companies to first come up with a list of questions they want answers to -- such as what is the average transaction time for login, where their users are and when are they active, which database processes are the slowest, what is the average response time for their API -- and find which APM products answer those questions.
Once an IT or DevOps department knows if the APM software can capture the data they want, they should consider if visualization, dashboards, alerts or warnings, along with email lists and responsibility by system are core needs. Some APM software provides the ability to visualize system performance on a monitor as it is happening. If developers or testers want people to see performance on public screens, then find out what particular information certain individuals in certain roles on certain teams need and see if the APM tools can provide reports that drill down to the level the organization requires.
Anyone that has dealt with software has probably come across security tools. APM software is no exception. There are administrative functions; data that might be sensitive, especially for companies in regulated industries like healthcare; and reports that people should not be privy to. In healthcare systems, data is carefully guarded. Someone seeing data that they shouldn't have access to often means large fines. If an enterprise is one that is concerned about who is accessing data, then security is a must have.
Nearly all APM software can be configured for security and access levels; the trick is to figure out if it will integrate seamlessly with the directory system you have without requiring an administrator to add and remove permissions for users. Consider, too, if the data is sensitive and if it could somehow emerge in reports. For example, if Web page links include transaction IDs and those appear in reports of slow pages, it might be a problem.
Platform compatibility and integration, reporting ability, and security are all aspects of APM software that can help guide your purchase. An enterprise should start by figuring out what it needs from the tool and how it will support its business goals, and then move from there.
As IoT gains traction, APM software focus on UX should too.
For continuous improvement, use data collected by APM software.
Check out this cloud APM software vendor assessment.