Magic Buttons and Code Coverage

This will be a quickie.  But sometimes good things come in small packages.

This idea came to us from Amir's good friend Eran Pe'er, when he was visiting Net Objectives from his home in Israel.

I'd like you to imagine something, then I'm going to ask you a question.  Once I ask the question you'll see a horizontal line of dashes.  Stop reading at that point and really try to answer the question.  Once you have your answer, see if you can work out why you answered the way you did.  Consider putting a comment below with your thoughts.

Here is the thing to imagine:

In front of you are two magic buttons, A and B.  You can only push one of them.

• If you push A, you will instantly achieve 80 percent code coverage, only 20 percent of your code will be left without test coverage
• If you push B, you will only get 70 percent code coverage, leaving 30 percent of your code uncovered

But there is one more difference.  If you push A you will have no idea where in your code the lack of coverage is, you just know that some 20% of your code is not covered by tests. If you push B, you will know precisely where your covered and uncovered code is, though of course there will be more uncovered code.

Which would you press, and why?

--------------------------------------Stop and Answer------------------------------------

For me, the answer is unequivocally B.  A big part of the value of TDD is reducing uncertainty and communicating with precision.  Coverage is all about adding a degree of safety when making changes whether they be via refactoring or when making enhancements to existing systems.  If I know where the uncovered code is, I know where I have to be more careful, and conversely if I know where the covered code is I know where I can be aggressive.

Also, if I push A, then I know that for every line of code there is a 1-in-5 chance that it is uncovered.

But I think this is only part of the answer.  We'd like to hear from you...

TDD and Design: Frameworks

Increasingly, in modern software development, we create software using components that are provided as part of a language, framework, or other element of an overall development ecosystem.  In test-driven development this can potentially cause difficulties because our code becomes dependent on components that we did not create, that may not be amenable to our testing approaches, and that the test cannot inherently control.

That last part is particularly critical.  In TDD, we want to create unique, isolated tests that fail for one specific reason and thus specify one narrowly-defined behavior of the system.  Tests that involve multiple behaviors are hard to read (as a specification) and will have multiple reasons to fail.  When a test fails, we want to know unequivocally why it failed so that we can efficiently address the issue.

We write tests in TDD to specify the proper behavior of the system, allowing us to confidently create the right things.  But they also, later, serve as tests to ensure that defects have not been introduced.  In this second mode, what does the test actually test?

A test will always test everything that is in scope which it does not control.  If a test is to be narrowly focused on just one thing, then everything else that is in scope must be brought under the control of the test, otherwise it is testing those things as well.

Let's roll the dice and look at an example:

Most people are familiar with the game Yahtzee.  Briefly, you roll five dice and try to make the best pattern you can.  Examples are three of a kind, or numbers in a sequence (a "straight") and so forth, all the way up to a "yahzee" which means all five dice are the same.  "Chance" means you have no pattern at all, just five unrelated numbers.

public enum Result { CHANCE, ACES, TWOS, THREES, FOURS, PAIR,
                     THREEOFAKIND, FOUROFAKIND, FULLHOUSE,
                     SMALLSTRAIGHT, LARGESTRAIGHT, YAHTZEE }

public class Yahtzee
{
    public Result RollDice() {
        char[] dice = new char[5];
        Result myResult = Result.CHANCE;
        Random rand = new Random();

        dice[1] = (char)rand.Next(6);
        dice[2] = (char)rand.Next(6);
        dice[3] = (char)rand.Next(6);
        dice[4] = (char)rand.Next(6);
        dice[5] = (char)rand.Next(6);

        // Logic to determine the best result and set myResult

        return myResult;
    }
}

The idea here is to roll five dice and have the game tell you what the best pattern is that you can make from the five random results that you got.  The default is "Chance" unless something better can be made from the die rolls you got.

What we would want to specify here is that the logic (which is commented out for brevity) would correctly identify various patterns of die rolls.  If we rolled 4 5's, for example, it would identify it as Result.FOUROFAKIND even though it is also true that we have three of a kind.

The problem is that Random is in scope... we are using it.  But unless we bring it under the control of the test we are also testing Random, which is not what we want.  Also, we cannot predict what Random will do.  We could seed the Random class with a known value, but even so we are testing more than we wish to.  How can we truly bring random under control?

This same issue would exist in code that is dependent upon any component: a GPS module, the system clock, a network socket, etc...

A Design Principle

In seeking to ensure high-quality designs, we need standards or rubrics to apply to any proposed design.  We want to check out thinking, to make sure we're not fooling ourselves or missing anything.  One such rubric is this:

When examining an entity (class, method, whatever) in our system we ask: is this entity aware of the framework, or aware of the application logic?  If the answer is "both", then we seek some way to separate the two aspects of the entity from each other.

If you examine the code above you'll see that this game entity is aware of the framework (how you create a use the Random class) and also the application logic (the rules of this particular game).  This is a clue that we should reconsider the structure of our code.  Note that this concern also impacts the testability of the code because, as we've already noted, the test does not want to be vulnerable to the Random component.  We want the test to be solely concerned with the game logic.

A Testing Adapter

One way to bring the framework component under the control of the test is to wrap it in an adapter, and then mock the wrapper:

class DieRoller
{
    Random rand;
    public DieRoller() {
        rand = new Random();
    }

    public virtual char RollDie() {
        return (char)rand.Next(6);
    }
}

..and then change the product code to use this class instead of using the framework element directly.  For the test, we could mock [1] this class and inject the mock instead of the adapter, bringing the die rolled in each case under the control of the test.

One example:

class MockDieRoller : DieRoller {
    private char roll;

    public void setRoll(char aRoll) {
        roll = aRoll;
    }

    public override char RollDie() {
        return roll;
    }
}

An Endo Test

Creating an adapter class is a viable option when it comes to framework components, but it may seem like overkill in some cases.  When the issue in question is very simple, as in our example, you could also control the dependency through a simple technique called "endo-testing."

public class Yahtzee
{
    public Result RollDice()
    {
        char[] dice = new char[5];
        Result myResult = Result.CHANCE;

        dice[1] = rollDie();
        dice[2] = rollDie();
        dice[3] = rollDie();
        dice[4] = rollDie();
        dice[5] = rollDie();

        // Logic to determine the best result and set myResult

        return myResult;
    }

    protected virtual char rollDie() {
        return (char)(new Random().Next(6));
    }
}

All we have done is extracted the use of the Random class into a local, protected virtual method.  This is a very simple and quick refactor; virtually any decent IDE will do this for you.  The test will look like this:

[TestClass]
public class YahtzeeTest
{
    [TestMethod]
    public void TestGameResults(){
           
        TestableYahtzee myGame = new TestableYahtzee();
        // conduct the test against controllable results
    }

    private class TestableYahtzee : Yahtzee {
        private char roll;
        public void setRoll(char aRoll) {
            roll = aRoll;
        }

        protected override char RollDie() {
            return roll;
        }
    }
}

Now the test can control what dice are rolled and conduct all the various scenarios to ensure that the rules of the game are adhered to.  Also, we've satisfied our design principles by separating game logic and framework knowledge into two methods, rather than two classes.




[1] See our blogs and podcast on mocking for more details:
http://www.sustainabletdd.com/2012/05/mock-objects-part-1.html

TDD and the "6 Do's and 8 Skills" of Software Development: Pt. 1

This post is not about TDD per se, but rather a context in which TDD can demonstrate its place in and contribution to the value stream.  This context has to do with the 6 things that we must accomplish (do) and the 8 skills that the team must have in order to accomplish them.  We'll describe each "do", noting where and if TDD has an impact, and then do the same thing with the skills.

6 Dos:

• Do the right thing
• Do the thing right
• Do it efficiently
• Do it safely
• Do it predictably
• Do it sustainably

8 Skills:

• Programming
• Designing
• Analysis
• Refactoring
• Testing
• Dev ops
• Estimation
• Process Improvement

 

Do the right thing

Everything the team does must be traceable back to business value.  This means “the right thing” is the thing that has been chosen by the business to be the next most important thing, in terms of business value, that we should work on.  TDD has no contribution to make to this.  Our assumption is that this decision has been made, and made correctly before we begin our work.  How the business makes this decision is out of scope for us, and if they make the wrong one we will certainly build the wrong thing.  This is an issue of product portfolio management and business prioritization, and we do not mean to minimize its importance; it is crucial.  But it’s not a TDD activity.  It is the responsibility of project/product management.

An analogy:

As a restaurant owner, the boss has determined that the next thing that should be added to the menu is strawberry cheesecake.  He made this decision based on customer surveys, or the success of his competitors at selling this particular dessert, or some other form of market research that tells him this added item will sell well and increase customer satisfaction ratings.  It will have great business value and, in his determination, is the most valuable thing to have the culinary staff work on.

Do the thing right

One major source of mistakes is misunderstanding.  Communication is an extremely tricky thing, and there can be extremely subtle differences in meaning with even the simplest of words.  “Clip” means to attach (clip one thing to another) and to remove (clipping coupons). 

A joke we like: My wife sent me to the store and said “please get a gallon of milk -- if they have eggs get six.”  So I came back with 6 gallons of milk.  When she asked why I did that, I replied “they had eggs.” 

The best way we know to ferret out the hidden assumptions, different uses of terms, different understanding, missing information, and the all-important “why” of a requirement (which is so often simply missing) is by engaging in a richly communicative collaboration involving developers, testers, and businesspeople.  The process of writing acceptance tests provides an excellent framework for this collaboration, and is the responsibility of everyone in the organization.

The analogy, continued:

You work as a chef in the restaurant, and the owner has told you to add strawberry cheesecake to the menu.  You prepare a graham-cracker crust, and a standard cheesecake base to which you add strawberry syrup as a flavoring.  You finish the dish and invite your boss to try it.  He says “I did not ask for strawberry flavored cheesecake, I asked for a strawberry cheesecake.  Cheesecake with strawberry.”

So you try again, this time making a plain cheesecake base and adding chopped up strawberries, stirring them in.  The boss stops by to sample the product and says “no, no, not strawberries in the cake, I meant on the cake.”

So you try another version where the plain cheesecake is topped by sliced strawberries.  Again the boss in unhappy with the result.  “Not strawberries, strawberry.  As in a strawberry topping.”

What he wanted was a cheesecake topped with strawberry preserves, which he has always thought of as “strawberry cheesecake.”  All this waste and delay could have been avoided if the requirements had been communicated with more detail and accuracy.

Do it efficiently

For most organizations the primary costs of developing software are the time spent by developers and testers doing their work, and the effect of any delays caused by errors in the development process.  Anything that wastes time or delays value must be rooted out and corrected.

TDD has a major role to play here. 

• When tests are written as the specification that guides development, they keep the team focused on what is actually needed. 
• The tests themselves require precision in our understanding of a requirement and thus lead to code that satisfies the exact need and nothing more.  Traditionally developers have worked in an environment of considerable uncertainty, and thus have spent time writing code that ends up being unnecessary, which wastes their time. 
• Without TDD, defects in the code will largely be dealt with after development is over, requiring much re-investigation of the system after the fact.  TDD drives the issue to one of bug prevention (much more time-efficient) rather than bug detection.

 

Do it safely

Software must be able to change if it is to remain valuable, because its value comes from its ability to meet a need of an organization or individual.  Since these needs change, software must change. 

Changing software means doing new work, and this is usually done in the context of existing work that was already completed.  One of the concerns that arises when this is done is: will the new work damage the existing system?  When adding a new feature, for example, we need to guard against introducing bugs in the code that existed before we started our work.

TDD has a significant role here, because all of our work proceeds from tests and thus we have test coverage protecting of our code from accidental changes.  Furthermore, this test coverage is known to be meaningful because of how it was achieved.

Test coverage that is added after a system is created is only guaranteed to execute the production code, but not to guarantee anything about the behavior that results from the execution.  In TDD the coverage is created by writing tests that drive the creation of the behavior, so if they continue to pass we can be assured that the behavior remains the same.

Do it predictably

A big part of success in business is planning effectively, and this includes the notion of predictability.  Every development initiative is either about creating something new, or changing something that already exists (and, in fact, you could say that creating something new is just a form of change: from nothing to something).

One question we seek to answer when planning and prioritizing work is: how long will it take and how many resources will be required?  Although we know we can never perfectly predict these things, we want to reduce the degree of error in our predictions.

TDD has a role to play here:

• TDD increases design and code quality.  There are many reasons for this, but the shorthand explanation is that bad designs and poor code are very hard to test.  If we start from the testing perspective, we tend to create more quality.  Higher quality creates clarity, and the more clarity you have the better your predictions will be.
• TDD points out gaps in analysis earlier than traditional methodologies.  These gaps, when discovered late, create unexpected/unplanned for work, and this derails our predictions.
• TDD provides meaningful code coverage.  This reduces the creation of unexpected errors, and fewer unexpected anything increases predictability.
• TDD helps us to retain knowledge, and the more you understand a thing the more accurate your predictions will be about changing it.
•  

Do it Sustainably

The team must work in a way that can be sustained over the long haul.  Part of this is avoiding overwork and rework, and making sure the pace of work is humane.  Part of this is allowing time for the team to get appropriate training, and thus to "sharpen the saw" between major development efforts.  Issues like these are the responsibility of management whether the team is practicing TDD or not.

However, this work is called "Sustainable Test-Driven Development" for a reason.  TDD itself can create sustainability problems if the maintaining the test suite presents an increasingly-significant burden for the team.  Much of our focus overall has been and will continue to be avoiding this problem.

In other words, TDD will not create sustainability unless you learn how to do it right.

Next up, How TDD impacts the 8 skills of software development