Monday, 18 February 2013

Testing Strategy Part 1 - Is 100% Coverage Enough?

When discussing testing, a question that sometimes comes up is "How can you demonstrate that your tests are correct?". One particular answer that I've encountered for this question is the use of tools.

I've been on a bit of a kick recently, and I feel that the rationale behind code coverage is very amenable to bayesian reasoning. Without going into too much detail, one major thing that bayesian reasoning provides is an ability to gauge the strength of evidence. Particularly, it demonstrates (with a solid mathematical foundation) that strong evidence for some event is only weak evidence for the opposite. In context, that means that if we have a metric that is strong evidence for poor test coverage, then that same metric is only weak evidence for good test coverage.

How does this relate to code coverage? For starters, code coverage doesn't change whether your tests are correct or not. All that a code coverage tool can give you is a percentage value. Now, if you take the extremes of code coverage then you can say with pretty much a certainty that if you run your tests and get a 0% code coverage result then your tests are bad. Your tests have not evaluated a single line of your actual code. This is extremely strong evidence in favour of poor tests.

So how about the other extreme, 100% coverage? Well, in this situation, you know that all of your code has been evaluated. Surely this is then strong evidence that your tests are correct? Not really, as it gives no indication as to whether the code is executing correctly. It merely tells you that it is all being tested. Your tests are able to be incorrect in the situation of 100% coverage, so this is only very weak evidence towards your desire to know if your tests are correct.

So where does this leave you? Well, if you are tracking code coverage and have a very low percentage,  then you are highly likely to want to increase your tests. If you are tracking code coverage and have a very high percentage, then you are going to want more evidence before you can make the call that your tests are highly likely to be correct.

Further reading: If bayesian reasoning is a fairly new concept to you, I can recommend An Intuitive Explanation of Bayes' Theorem

1 comment:

  1. Hi,
    In some situations, 100% code coverage is just the minimum acceptable case, because as you say, it only indicates that all your code is 'inspected', but that doesn't guarantee that all cases are contemplated... or well contemplated.