Doing things right the first time is the "motif" at Betfair that permeates down to the developer culture. If you've got a tool that will help you do that, why not make it second nature and bring it into your development process?
That's exactly what the leading online betting exchange and Europe's largest e-commerce site is doing with the deployment of Fortify Software's source code analysis tool, explained Matt Young, distributed development director at Betfair. By automating the mundane parts of code review, the young and fast-growing company frees up developers to concentrate on the more creative and business-differentiating parts of their jobs, better manage their outsourced projects, and most important, create an institutional memory around quality best practices.
Founded in August 1999, Betfair processes 5 million transactions a day and more than 300 bets a second. The company has 100-plus in-house developers, as well as a joint outsourcing effort in Romania and additional contracted outsourcers. In addition to the exchange, Betfair has a games portfolio that includes Betfair Poker, Betfair Casino and a number of exchange-enabled games.
Young likens the business to financial organizations, with similar speed, scalability and reliability demands. Quality and security are paramount.
"We have more transactions in a day than all the European stock markets online," he said. "Unlike the stock markets, we run 24/7 -- there's no time of day when there's not someone logged on from somewhere to place a bet. In addition to all the usual security requirements any company has, reliability is extremely important because the nature of what we're selling is time-sensitive."
With its product portfolio and code base growing, as well as the need to manage outsourced projects, Betfair sought an automated solution for part of its code review, what Young terms the "low level."
Young explained, "At the low level you're looking for slipup bugs, like you forgot to release this resource or forgot to validate some input. It has to be done, but it's quite mechanistic. We thought, 'Is there a way that the repetitive part of the code review can be automated?' So we started looking at products."
Two key requirements were breadth of language support, as Betfair has code written in Java, .NET, C++ and more, and the ability to write custom rules, Young said. After looking at various open source products and then having a bake-off with some commercial offerings, Fortify SCA 5.0 from Fortify Software in Palo Alto, Calif., made the cut.
"What we didn't want to do was buy a separate tool for each language," Young said. "We also wanted something that could prove it could find useful bugs. One thing that's misleading when you look at some of these products is they'll spin through the code and find vulnerabilities, but the numbers are not very meaningful. You look at the details, and half the things they found are like a semicolon with space before it. We already have tools for doing stylistic things. It feels like some of the things in there are to bump up the numbers."
Young also knew that the tool would have to be customizable and trained.
"For a business growing like ours, we're trying to do so much, so we can't really afford to be making the same mistakes over again. We have our own custom libraries, and as we've grown up, we have our own idioms for doing things," he said. "So how to do it at Betfair won't come out of the box with any tool. We want to tell the tool, 'Here's the wrong way to do it, find it in the current code,' and then have tool check code every night to make sure no one else made the same mistake. The great thing is it provides memory to your organization that you can't do through process alone."
Quality flaws as important as security flaws
While Fortify has positioned itself squarely in the application security space, and Betfair was not looking for an application security tool per se, Young said he views code quality and security as part of the same spectrum.
"I'm quite conscious that in the marketplace there is a dichotomy between security in code and quality in code," he said. "On one end of the spectrum there's security in terms of your customer's private details, and on the other end there's quality like a mistake on the home page. In the middle it's a gray area."
Whether it's a security flaw or quality flaw that brings a site down, "the net result is the same -- it's losing money and I want to fix it," Young said.
Young said Betfair is deploying Fortify SCA gradually, starting with outsourced projects.
"They're projects with a self-contained code base, so it's straightforward to run through Fortify, sort through the false positives, then raise it back to the outsourcer plus what we found manually," he said. "With the other [in-house] teams, we'll work with them to find a suitable point in the development cycle to introduce the new tool."
Even before refining and training the tool, Young said they can capture the nightly output to find trends.
"The raw number of vulnerabilities a tool claims to find is not of in and of itself a meaningful number, but it's interesting to watch the trend over time," he said. "You may find that a sudden peak or trough often corresponds to something odd that happens, like someone who checks in a bunch of new code without reviewing it because they're new."
The tool also introduces a healthy sense of competition among the developers, Young said, because they can get comparative numbers from the nightly run and the teams can see how they're doing.
Young warned, however, that how a tool such as this is introduced can impact acceptance.
"You have to make sure when presenting it to developers that you're explaining that the intent behind it is to make their life easier," he said. "If it's presented in a thoughtless way, it can seem like another hurdle you have to jump through. That's not the intent; the reason is to take away the drudgery from the code review process."