Although black box security testing is extremely important to do, researchers at Fortify Software have found it isn't enough to help developers find and repair code flaws.
In a report released Monday about black box security testing -- sometimes referred to as penetration testing -- Fortify researchers found that such tests failed to provide adequate results in three areas:
- Security test coverage -- Black box tests don't tell you what percentage of the code was hit, said Barmak Meftah, vice president of products and services at Fortify. "Without that parameter, the gauge of security isn't clear," he said.
- The inability to pinpoint the location of a vulnerability -- A black box test can tell you only what page the vulnerability is on, Meftah said. It can't give such details as where on the page or in the application the vulnerability is.
- Not all input sources are tested -- Because black box tests address just the Web interface, they don't give you all the problems, Meftah said. An application has a lot more sources of input, he said.
"While black box security testing is an important tool for analyzing the security of deployed applications, its scope is limited by the fact that it resides outside of the application," Meftah said.
To remedy that, Fortify has created a product to complement black box testing and give developers and testers greater details about test results. Fortify Tracer, whose announcement coincides with the release of this report, sits inside an application and provides "more measurable and actionable output," Meftah added.
For example, Fortify Tracer injects monitors in all of the attack surfaces and around all the functions of the application. Then when a black box test finds issues with an application, Fortify Tracer tells how much of the code was hit and where specifically the problem is.
"Once the issue is found, we can give more information about the cause of the problem because it sits inside the application," Meftah said.
Charles Kolodgy, research director of secure content and threat management products at IDC, reiterated the importance of making applications as secure as possible. The key to application security tools, however, isn't just what they can find but how accurate the tools are at finding real vulnerabilities while minimizing false positives. Following that, it's important to be able to remediate the discovered vulnerabilities, he added.
"This is exactly what Tracer does best in cooperation with the 'black box' testing," Kolodgy said. "It can isolate the exact location of the vulnerability identified by the application scanner in the source code. This should make it easier to be fixed and should also allow people to determine if it is an actual vulnerability."
Fortify Tracer currently works on any J2EE executable (.war/.ear) files. Dashboards communicated key metrics and allow users to compare runs, inspect issues and find flaws. In addition, it generates detailed reports showing vulnerabilities according to their categories, such as cross-site scripting (XSS) and SQL injection.
Meftah said Fortify Tracer will be integrated with Watchfire's AppScan, but the product will also work with any black box security tester, he said.
Available immediately, Fortify Tracer costs $24,000 per named end user.