in defense of code coverage

"How much do I care about code coverage?"

"100%!" (Kidding.)

SQLite's article on testing is helpful in this discussion. SQLite famously has 100% code coverage, but developers still find and fix bugs in SQLite. Why? Because code coverage is just one part of a testing strategy.

When I find a software project that publishes code coverage metrics, it tells me some things:

• The developers know what a unit test is, and they care about writing tests. This increases my confidence in the stability of the project and my respect for the developers.
• If I contribute to this project, I will need to learn enough to contribute a test for my feature as well.

Yes, it's helpful to know the percentage of code coverage for a project. "100%" doesn't mean the project is bug-free. It means that the developers care about shipping quality software, and they have an (imperfect) metric to help achieve that outcome.

Who are my expected users?

This question informs how much effort I put into code coverage.

A) I'm writing a library or API that will be used in many different ways by different teams that do not communicate with me or each other: code coverage tools are very important.

B) I'm writing a specialized tool that will only be used in one well-understood case by my immediate team of developers: coverage is nice to have, but I don't prioritize this as much.

C) I'm writing a one-off script that I will only be the sole user: This depends on a lot of factors of course, but normally I'm not going to put much effort into writing tests.

Of course the answer to "who are my expected users on this project?" can change.  One time I wrote a project for myself that ended up drawing a lot of unexpected users. It's important to periodically reevaluate the nature of a project's user base. Example: "Given the last 12 months, have my users' expectations for stability and quality changed?"

How are my users changing?

On one project I wrote, my first user base was very small. It consisted of developers and hackers who were very involved with feedback, design, and testing. Those original users moved on to other responsibilities, and new users have replaced them who are unfamiliar with the code. They have very different expectations and want your project to "just work".


This is an entirely different scenario. Documentation and regression testing are critical to sustaining the growth of the project. The new user base does not share the initial users' tolerance for breaking changes.

Bug reports will continue to come in. On one recent bug report, once I identified the root-cause and the exact function that is buggy, the next question I ask is "Do the unit tests cover this method?" Code coverage tools can quickly answer this question. This makes it easier for me to confidently modify the method because I know that I'm not introducing regressions.

Summary

Unit tests and code coverage are important, and the question is "how important?".

The best way to evaluate the importance of unit tests and code coverage is asking the question "What is today's cost of introducing and fixing a regression"? On a personal project, it's very low. On a popular core library with many users, the cost is very high. To answer that question accurately, you must understand your users.