In the first of this two-part series on code coverage, we briefly looked at how code coverage works with a specific example. We found that while code coverage tools may measure code that has been executed with unit tests, it doesn’t necessarily validate the usefulness of those tests. In this second tip, we look more deeply at the different types of code coverage metrics and end with recommendations on how to best increase code coverage...
and where to go for additional information.
Different types of coverage
Most code coverage tools provide different levels of coverage. There are three broad categories of coverage: statement (or line) coverage, branch coverage and path coverage. We saw an example of statement coverage in the example above, but let's look at branch coverage and path coverage in a bit more detail.
Looking at the example code in Listing 2 below, suppose you had a unit test and passed in the value 1 for both the row and column parameters and as a pre-condition set that square on the board to be null.
def isTargetBlank?(row, column)
if @board[row][column].getSpaceValue() == nil then
Listing 2: Example method for evaluating an empty space on the board in Gobblet.
In that case, with statement coverage you'd exercise the if statement line, return true line, and the end lines. Most statement coverage tools would likely give you five lines out of seven. I say most tools, because some tools can calculate coverage a bit differently.
However, with branch coverage we're more concerned with the evaluation of control structures (like the if statement). So in our example above, we'd likely get 50% branch coverage for that method. That's because we've tested one of the two possible evaluations of the state of the space (it being null or not null).
With path coverage we're looking to evaluate if every possible route through the code has been executed. For this example, our branch coverage and path coverage will match. Because we have an if statement (and only one if statement), so we also have 50% path coverage. There are only two paths through the code, the if condition or the else condition.
If we change the example to something like what's shown in Listing 3, it gets a bit more complicated (but still not too complicated).
def removePiece(row, column)
if (row < 0) or (row > 3) then
setMessage("Row not valid: " + row.to_s)
if (column < 0) or (column > 3) then
setMessage("Column not valid: " + column.to_s)
Listing 3: Example method for removing a game piece from the board in Gobblet.
In this example, if you wanted 100% branch coverage, you'd need at least five tests. You'll need one for each side of the or condition in each of the if statements. That's four tests. And you'll need one that successfully "passes" each if statement to execute the return true statement. For 100% path coverage however, you'll only need three tests - one for each of the two error checks (returning false) and one that successfully removes a piece. Once again, those numbers may depend on which tool you're using and how they've implemented their coverage algorithm.
It's been my experience that most teams measuring coverage focus on getting high statement coverage with reasonably high branch coverage when possible. I've not worked with a team that's given much attention to path coverage. As near as I can tell, for code of any complexity, it's likely more trouble than it's worth.
Tips for increasing code coverage
When you start testing for a particular piece of code, don't think about coverage at all. Just write the tests you feel are necessary to implement the functionality or behavior that you're looking to get from the code. Focus on good testing, not an arbitrary coverage metric. As you write your tests, you want to be thinking of clean interface design, test data, and mocks that support your testing.
Once you feel you have the tests you need, then run the coverage numbers. Use that information to help figure out what you might have missed with your earlier testing. Is there a condition you missed that you feel you should have a test for? Go ahead and implement it.
Keep in mind that at this point if you feel your tests cover all the functionality, code that's not exercised might indicate code that's no longer needed. Double check. One way to increase coverage is to reduce the footprint of the code base by removing code that's no longer needed and not exercised.
When you're trying to increase path coverage, many times you might have to refactor the code into more testable methods. High cyclomatic complexity translates into a lot of paths to test. If you can lower that complexity, often times you'll find it easier to increase coverage.
There are a number of coverage tools listed out on the Wikipedia page for code coverage. If you're looking for a tool, that's a good place to start. More important than the focus on tools, I’d recommend some articles on code coverage that might help make sure you’re focusing on the right aspects:
- In addition to what I've presented here, Steve Cornett’s article Code Coverage Analysis is another good primer on code coverage.
- For a short look at some problems with code coverage, with some examples of some of the issues, take a look at TotT: Understanding your coverage data from the Google testing blog.
- I also really like Andrew Glover’s article In pursuit of code quality: Don’t be fooled by the coverage report. It has some excellent examples.
- Finally, no list on code coverage articles would be complete without Brian Marick’s paper, How to Misuse Code Coverage.
For more on measuring quality, see Quality metrics: A guide to measuring software quality.