Tuesday, 26 February 2013

Testing Strategy part 2 - Unit tests

Following on from looking at code coverage and the central question of "How can you demonstrate that your tests are correct?" I'd like to consider another aspect of bayesian evidence - the tests themselves.

Taking the viewpoint of strong evidence and weak evidence again, I'd like to examine what level of evidence various tests provide. There are various types of test that are typically used in software development, each type provides a different level of granularity and a different balance of focus (developer-centric, performance-centric, business-centric, etc).

At the bottom of this stack is the most prevalent test type - unit tests. A unit test is a developer-centric test with a very fine granularity. A unit test is intended to be the most granular test, as each test should have an indivisible unit of functionality/behaviour as the subject.

Taking a bayesian perspective, I would characterise a single passing unit test as weak evidence of system correctness. Conversely, a single failing unit test is strong evidence of the system being incorrect. To understand this viewpoint, consider a typical case of hundreds of unit tests. Each test was written by a developer, usually with little or no direct input from other sources, with the aim of demonstrating that a particular unit of code works as expected.

These tests do not show that the system as a whole functions together, and they do not show that the system works as the customer expects it to. Instead, what is demonstrated by a full suite of passing unit tests is that each individual unit of behaviour works as the developer(s) expected, which I take as weak evidence of overall system correctness. A single failure in this suite of tests shows that some functionality isn't working as the developer(s) expected, a very strong piece of evidence that the system is not correct.

Where does this leave you? Taken in addition to code coverage, we now have two pieces of weak evidence for system correctness, but have yet to address the issue of test correctness. I'll attempt to move closer to this issue in future posts where I'll consider other types of test.

Further reading: If bayesian reasoning is a fairly new concept to you, I can recommend An Intuitive Explanation of Bayes' Theorem

No comments:

Post a Comment