Large enterprises demand a lot from their software vendors. They want 100% up time and no hiccups. So when Houston-based BMC Software Inc. decided to take an acquired product line "upscale" to target the enterprise market, reliability and quality were paramount. The company turned to automated source code analysis
"At the highest level, the real business problem was the reliability of the system," said BMC lead developer Samuel Dillon, referring to the Action Request product line, the Remedy assets BMC acquired from Peregrine Systems. The platform for help desk, incident management and other business service management applications was first designed more than 10 years ago and targeted midsize companies, Dillon explained.
But with large enterprises as the new target for the platform, "the system needs to be reliable," he said. "You can't say you've got memory leaks and every few days restart the system to clear out. They don't go for that."
Dillon said they were using IBM Rational Purify, a runtime analysis solution that provides memory leak and memory corruption detection, "but a runtime analysis only covers cases you test, whereas a static analysis tool looks over all the possible code paths, so you don't need to come up with a test for every error condition. When you're working with a platform there are so many things; executing all paths through the code would take forever."
It would also tax the quality assurance area, Dillon said.
"We have a QA staff that writes custom tests. It's an uphill battle for them because you can't test everything," he said. "And while they're testing, we're adding more features, so you never catch up."
BMC wanted a tool that could "improve the coverage of the examination of code" that it couldn't get from Purify. That led the company to look at source code analysis tools.
"Developers constructing unit tests will test what the function should do, not every exception," said Gwyn Fisher, CTO of Klocwork Inc., a source code analysis vendor in Burlington, Mass. "And QA's temptation is to test what it's supposed to do and maybe some of what it's not supposed to do. In a dynamic environment, constructing enough cases to test all the run-time adaptations is almost impossible, but the [static analysis] tool doesn't care. It will step through every feasible code path, and if you go down the exception path what happens. All things are considered equally important to the tool, so developers are not on the hook for constructing an infinite number of unit tests."
Integration with IDEs key
Initially, BMC brought in a source code analysis solution from San Francisco-based Coverity Inc., which the company still uses. But Dillon said they continued to look at tools in this category, and when they came upon the Klocwork K7 Enterprise Development Suite, its integration with Visual Studio and Eclipse integrated development environments (IDEs) was a key selling point.
"You just pound out new code and punch a key. It was a really big selling point for us," Dillon said.
Fisher said a priority for BMC was how the product works with the existing developer workflow.
"With the click of a button on the Visual Studio toolbar, we show up with a lot more context and value," he said. "It fits with the existing way of doing what they do. There's no culture shock; it's just another tool."
Fisher added, "The big deal in this industry is how much latency you build into the process to find problems. With a traditional tool, you check in broken code to analyze, and it could be hours, days or weeks to find out."
Architectural analysis important to BMC
Another feature that sold BMC was K7's architectural analysis capabilities.
"You can submit some or all of the code, and it will extract the architectural information and allow you to do what-ifs, like what if you pull this out and make it a separate library," Dillon explained. "If you like the way you did the what-if on the architecture, you can punch a key and print a list of what I do to make that happen. It's a pretty cool feature. I've also started to use it in the patch process. I do a before and after architectural analysis, look at the differences, and make sure someone didn't make some change that has some far-reaching consequence that they didn't notice and test for."
According to Dillon, BMC built Klocwork right into the company's process.
"We had a few engineers use it for while and figure out best practices, then we did some tutorials with the engineers," he said. "We have written policies and procedures of how to go about development, so it's kind of written into that. The process now includes running Klocwork over any change you make through the IDE. You're supposed to do that before you check in any code."
Although BMC was most focused on reliability and quality, the fact that Klocwork also scans for security vulnerabilities was a plus, Dillon said. "From a platform point of view I think it was relatively secure, but it's a good feature and we use it," he said.
Quality, productivity improved
BMC has used Klocwork for about two years, and Dillon said it has made an impact on both quality and productivity. For example, he said the character of the problems being reported to the support organization has changed.
"We don't end up finding some odd case a customer runs through the code path that was never completely run out and is leaking memory and the server crashes -- we used to get those fairly often," Dillon said. "Now with the support calls a lot more are customer errors. And the problems that do make it all the way back to engineering tend to be not basic coding errors but more logical flaws, so the feature doesn't work the way the customer thought it should, but it doesn't crash."
In terms of productivity, he said, "We spend less time chasing dumb things. There's more time to focus on fixing logical problems or implementing features."
Dillon said BMC continues to use the Coverity as well because the overlap is not complete.
"Coverity finds a few things that Klocwork never finds and vice versa," he said. "Probably 15 to 20% of the defects each product finds in coding errors are things the other doesn't. It's enough that it makes it worthwhile to keep both."