However, it drinks the code coverage cool-aid that started like 30 years ago when code coverage tools emerged.
Management types said "high test code coverage == high quality"; lets bean count that!!
A great way to achieve high code coverage is to have less than robust code that does not check for crazy error cases that are really hard to reproduce in test cases.
Code coverage is a tool to help engineers write good tests. One takes the time to look at the results and improve the test. It is a poor investment to be obsessed with code cover on paths where the cost to test them greatly exceeds the value.
10% coverage and 100% are both alarm bells. Don't assume naive, easy to produce metrics are the same as quality code.
Python is the language with one of the highest 100%-coverage-to-effort ratios. The included unittest.mock framework is making it quite easy to trigger obscure errors and ensure they are handled properly.
Combined with thoughtful use of `# pragma: no cover` a 98% code coverage nowadays is an immediate warning that something was rushed. With this and type checking I feel RuntimeErrors much easier to avoid these days.
And typing, not even a mention?! :) But otherwise a great article, thank you!
The problem with using mocks extensively for testing is that you then end up mostly testing the mocks. You'll know you've hit that point when you have 100% coverage, but things still break routinely due to interaction (sometimes very indirect) between components.
From the point of view of a programmer setting up a new project, type checking tools are yet another optional linting operation that can be integrated in pre-commit hooks or other automation.
100% coverage as a side effect of careful testing isnt a red flag.
Coverage is a decent (among other things) measure unless it becomes a target. Once it becomes a target you get shitty rushed tests that act mostly as cement surrounding current behavior - bugs and all.
He's not saying it's difficult to get 100% coverage. He's saying that 100% is a bit suss because you probably wouldn't achieve 100% coverage unless that is your goal (even if Python does make it easy) and that's the wrong goal.
It's a real goal for dynamic languages. You have no other option to be sure your code is not broken. Another option is to use 100%-non-any type hints. It's a way more harder.
I can see it's a non goal then you have access to deployed code and Sentry. But as library author or author of customer apps there is no other way around.
However, it drinks the code coverage cool-aid that started like 30 years ago when code coverage tools emerged.
Management types said "high test code coverage == high quality"; lets bean count that!!
A great way to achieve high code coverage is to have less than robust code that does not check for crazy error cases that are really hard to reproduce in test cases.
Code coverage is a tool to help engineers write good tests. One takes the time to look at the results and improve the test. It is a poor investment to be obsessed with code cover on paths where the cost to test them greatly exceeds the value.
10% coverage and 100% are both alarm bells. Don't assume naive, easy to produce metrics are the same as quality code.
Otherwise, and excellent article.