This idea came to us from Amir's good friend Eran Pe'er, when he was visiting Net Objectives from his home in Israel.
I'd like you to imagine something, then I'm going to ask you a question. Once I ask the question you'll see a horizontal line of dashes. Stop reading at that point and really try to answer the question. Once you have your answer, see if you can work out why you answered the way you did. Consider putting a comment below with your thoughts.
Here is the thing to imagine:
In front of you are two magic buttons, A and B. You can only push one of them.
- If you push A, you will instantly achieve 80 percent code coverage, only 20 percent of your code will be left without test coverage
- If you push B, you will only get 70 percent code coverage, leaving 30 percent of your code uncovered
But there is one more difference. If you push A you will have no idea where in your code the lack of coverage is, you just know that some 20% of your code is not covered by tests. If you push B, you will know precisely where your covered and uncovered code is, though of course there will be more uncovered code.
Which would you press, and why?
--------------------------------------Stop and Answer------------------------------------
For me, the answer is unequivocally B. A big part of the value of TDD is reducing uncertainty and communicating with precision. Coverage is all about adding a degree of safety when making changes whether they be via refactoring or when making enhancements to existing systems. If I know where the uncovered code is, I know where I have to be more careful, and conversely if I know where the covered code is I know where I can be aggressive.
Also, if I push A, then I know that for every line of code there is a 1-in-5 chance that it is uncovered.
But I think this is only part of the answer. We'd like to hear from you...
The difference between A and B is too small so I immediately chose B. Make it 50% coverage for B and I'd a least think about it before choosing B.
ReplyDeleteIt's all about risk, and determining which risk is less costly - the risk of not knowing which part of your system is regression protected vs. the risk of not having regression protection.
DeleteI would stipulate that under certain circumstances I would take button B giving me 10% coverage, and I would still choose it. Why? Because the area that *is* covered is the area that is being worked on, and changed. If your design is adequate, and you have some compartmentalization of your code, the area that is not covered by tests may not be of importance because it is not affected by the changes in the area that is.
Feel free to answer the question if B were 50%, if that's more interesting.
DeleteB.
ReplyDeleteNot knowing what is NOT covered is the risk here for me. I'd rather know what is NOT covered and then have to do the work to cover what is needed to be covered, than having a false sense of security based on the size of the coverage.
In Lean Startup parlance, button A gives me a "vanity metric" that can fool me (and my sponsors) into feeling awesome -- we've got 80% coverage!-- but it's totally useless to take action on (assuming 100% coverage is wasteful).
Inversely, button B gives me an "actionable metric": I know exactly what doesn't have coverage, allowing me to be Lean by making calculated cost-benefit trade-offs and only create coverage where needed. I also retain the knowledge of where the remaining risks lie, and should the benefit of creating additional coverage change and outweigh the cost, I can respond appropriately.
B
ReplyDeleteA few years back, I had some excellent conversations with some Reliability Engineering experts at Bell Labs. Those conversations convinced me that the value gradient in testing some lines of code over others is very steep. I would gladly go much lower to 50% or possibly less to know what is covered and what is not. For the uncovered code, I would apply the principles of Reliability Engineering to know which areas of the code to focus additional tests on.
How about none of the above? Much value comes from the thought process that produces the tests. Magic buttons don't generate thought. I would prefer to start at 0% and work my way up rather than work with a system that I don't really understand. That said, when I join a project already in progress I'm in exactly this position (that is, Button B).
ReplyDeleteFair enough, but the question has a purpose, not to be taken literally (which is why I used the term "magic"). It has to do with the value of certainty of coverage vs. degree of coverage.
DeleteI'd take B with even less coverage if I got to decide what was covered, rather than it being just some arbitrary code.
ReplyDeleteI'd take B, it gives a me confidence of what has been tested and allow me to plan for the rest
ReplyDeleteI liked your article and I hope you will have many entries or more.
ReplyDeleteonline internship
online internships
watch internship online
online internship for students
the internship online
online internship with certificate
online internship certificate
python online internship
data science online internship