Code Coverage Caveats

A relatively common criticism of statement-level code coverage tests is that, even though all statements may have been executed (i.e. you have 100% statement coverage), you may not hit all possible execution paths. Understanding that in the abstract is one thing, but yesterday I came across a case where this was made very clear to me in a practical setting.

I had a function that looked something like the following:

You’ll note that there are six possible paths through this function that can be taken depending on the value of a (1, 2, or neither) and b (2, or not 2). If we were to run a statement-level coverage test here, it would be very clear which cases we’re hitting and not hitting.

So here I am, using my coverage tool to help make sure I’m not missing anything with my test cases. Suppose I’m partway through adding tests and they look like this:

I run the tests and my handy dandy code coverage tool reminds that I’m still missing two statements. Thanks, code coverage tool!

But! Look at those inner branches! Aren’t they tantalizingly similar to each other?! Why don’t we factor those out into a function, like this:

Ah, how lovely. We’re repeating ourselves less. But wait! Let’s check out our tests:

Oh no! My statement coverage went up to 100% without adding any new tests! Yikes! It’s a good thing I already have a pretty comprehensive list of tests to add, because if I was just relying on my code coverage tool, I’d be in trouble!

Branch coverage tests can help in this kind of case, but unfortunately they’re very time consuming to run, so it’s not something I’ll be running after every test I add.

The moral of the story is that even a simple change in code structure can artificially boost your coverage numbers, and so you should take those numbers with a grain of salt — especially when writing new tests and refactoring.

Full source code for the example: