Agile - TDD, Simple Design, Refactoring, Evolutionary Design

Test Driven Development, Simple design, Evolutionary design and Re-factoring are interrelated topics. I am going to cover these topics, one by one.

TDD: 

TDD stands for Test Driven Development. I was fascinated with this term during my initial days in eXtreme Programming (XP). I was curious. I wanted to understand the philosophy behind it. I joined XP team from Waterfall background. Best Practices of XP were new to me. However, the terminologies were interesting!

As the name suggests, Test Driven Development is when development work is driven by testing. That means you create your unit tests first, then you progress through coding. How can it happen?

Ø        First you write automated unit test.

Ø        Run the unit test.

Ø        Test will fail since there is no supporting business code yet.

Ø        Next thing is you write the business code.

Ø        Run the same unit test.

Ø        Test will pass if your code is correct.

You test first, and then you develop the code. This practice is often termed as TFD, Test First Development.

Creating a unit test helps a developer to really think what needs to be done. Requirements are understood and nailed down firmly by unit tests. There can be no misunderstanding a specification written in the form of executable code.

Let me explain the process:

Ø        You are a programmer. A user story is given to you.

Ø        You break down the story into tasks.

Ø        Again you break down a task in your mind into pieces of requirements.

Ø        You create one unit test to define some small aspect of the problem at hand.

Ø        Then you create the simplest code that will make that test pass.

Ø        You take next small piece of requirement from the user story.

Ø        Create a second test

Ø        Implement code to satisfy the second test.

Ø        The above process continues. Unit test suit builds up. By the time you code the whole user story, it will be 100% unit tested.

So now you realize, writing of code is driven by unit tests. TFD is turned into TDD.

Benefits:

1.        Developers know what exactly the requirement. Requirements are understood and nailed down. You are writing clean code that works.

2.        Less chance of scope creep.

3.        No untested code makes it to the system. TDD is a tool to ensure full code coverage.

4.        There is also a benefit to system design. It is often very difficult to unit test some software systems. These systems are typically built code first and testing second, often by a different team entirely. By creating tests first, your design will be influenced by a desire to test everything of value to your customer. Your design will reflect this by being easier to test.

Mike Cohn recommends to do TDD for its testing benefits; any potential design improvements it brings as a bonus.

5.      You can get suggestions from your pair programmer while doing TDD to ensure better quality of code. Please check pair programming session at: http://chandrimachoudhury.blogspot.in/2013/11/agile-pair-programming-what-why-when.html

Kent Beck popularized TDD in case of XP. Beck's concept of test-driven development centers on two basic rules:

1.        Never write a single line of code unless you have a failing automated test.

2.        Eliminate duplication.

There is well-known objection about TDD. They say “Test First Development (TFD) takes longer; we don’t have time to waste.” I searched Mike Cohn. And he says: “There is evidence that doing TDD takes about 15% longer than not doing TDD (George and Williams 2003). But there is also evidence that TDD leads to fewer defects.”  TDD may take longer initially, but this is a tool you can use to make your code-to-the-point, assure higher percentage of code coverage. Those white box testers out there, do you have any different opinion?

My personal take is, if your code is testable 100% by unit tests, there is reduced number of defects during black box testing. As a result, reduced bug fixing and maintenance time. One more aspect is, developer nails down the requirement while writing unit tests first, the code has to satisfy the unit test. In this process, developer tends to understand the requirement in detail. This gives a chance to clarify the requirement gap found in stories (if any), leading to reduced number of defects found in the later stages of testing.

Simple Design: 

Sounds so simple! Yes, it is.

Simple design is something you have to practice while coding in agile. Agile (SCRUM and XP both) preaches simple design. Generally, in waterfall or V model, when a new functionality is conceptualized,

Ø        Requirement documents (SRS) are written.

Ø        HLD(s) are derived from requirement documents. There are rounds of reviews with client, subject matter experts, and application development teams. Sometimes requirements are progressively elaborated in iterative manner and HLDs keep getting complicated.

                  HLDs cover the design/architecture of new implementation that is part of whole system/application’s architecture. It contains database design, interface details, interface dependency details.

Ø        HLDs are broken down into LLDs (Lower Level Design). Low Level Design (LLD) is like detailing the HLD. It defines the actual logic for each and every component and each transactions of the system/application. UMLs (Unified Modeling Language) are normally part of LLDs. LLDs may contain class diagrams or object diagrams with all the methods and relationships between classes/objects. Developers interpret the diagrams and understand the relationship that may be called as Instance level relationship or Class level relationships.

Ø        What I want to tell here is developers/lead developers concentrate on whole design of the new code implementation and how the new design fits into the existing design/architecture.

Ø        They start coding in big-bang style when HLD/LLD is baselined.

Ø        When coding is complete, the code-build is deployed to test machines for testers to test it. (Functional/black box for example).

People coming from waterfall background, will find the “simple design” practice uncommon. It is a paradigm shift from how we proceed to code in conventional waterfall model to how we are supposed to code in agile. It is technical practice change.

Philosophy of agile is to have trust on the team. Team will deliver high quality, potentially shippable code at the end of each sprint/iteration. And when team has to deliver a working product after each 15 days (normally the length of each iteration/sprint), it is unlikely team will invest much time on thinking about big up-front design/architecture, review it, baseline it.

Am I talking design is not at all required in Agile? No, not at all.

While a robust architecture for application is important, the implementation strategy for individual user stories should be simple. The idea is to not over engineer the implementation of user stories, only plan and implement exactly what’s necessary. If you over engineer there is a high likelihood that the user story will change, and the additional work put in will amount to waste.

To sum up:                                                                                                                                     

1.        When you start working on a story, your first intention would be to implement the story, with simple solution in mind. You pair program, code the story, integrate the code with existing code by continuous integration tools, let the code and story pass unit tests and acceptance tests and let the whole integrated code pass if automated regression testing is applicable. You have shippable working product now. With this, you are in a position to give demo to the client.

2.        Now, if you feel an urge to improve the code design, if you feel to eliminate some duplicate code, or if you feel like moving a piece of code from one class to another (in case of object oriented programming) to make the module independent, maintainable, understandable – you do it. This is normally called re-factoring. I am going to elaborate this later in this session. Generally SCRUM/XP team keeps last 2-3 days for re-factoring work to be done. One or two persons in the team can identify areas of improvement and they re-factor the code. If, for example, an experienced team member is already aware of an impending story from the next iteration/sprint, and if he knows the current re-factoring of the code will help to implement that future story in a plug and play manner, he can go ahead with re-arranging/re-factoring the code in current sprint. Remember to follow activities like re-run unit test, re-run acceptance test, continuous integration, regression test after re-factoring is over. At the end of the day, customer wants a working product from you.

3.        Keep your design simple so that whole code is testable by Test Driven Development (if you follow), to make sure no untested code makes it into the system. On the other hand TDD helps to improve the design.

4.        Keep your design simple to make sure you can reach up to a certain code coverage percentage set by your client/product owner/internal quality policy guidelines, without spending significant amount of time.

Any objection? Yes, objections are bound to happen. Architecture experts often come up saying “I must decide architectural design first because I am working on a complex system”.

Yes, on a complex or large system, you probably will have a vision of architecture. However, the idea that being agile is about finding a right balance between anticipation and adaptation. How much architectural thinking to do up-front is of course your decision. Having said this, do not over-think design when it is not actually necessary.

In DonaldKnuth's paper "StructuredProgrammingWithGoToStatements", he wrote: "Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.”

Evolutionary design: 

Evolutionary design is popularized by XP. Unlike waterfall model, there is an understanding in Agile that user story elements will change with time, so a Scrum team should expect to update the implementation strategy for those user stories with every new iteration. Evolution of design is an expected change in Agile. It is a viable design strategy of Agile. The idea is that the implementation strategy can change as new information is determined, so the Scrum team makes the updates accordingly, and quickly seeks feedback from the Product Owner.

Experts in Agile say programmers are expected to get used to life without big design. Why to invest big time to think on permanent design of the system, when requirement change is inevitable in Agile, hence, re-work is inevitable?

Refactoring: 

The definition says it is the process of clarifying and simplifying the design of existing code, without changing its behavior. Refactoring changes the structure of code, not its behavior. In some organizations, refactoring is termed as “technical debt”. That means they indicate team will come back and will improve the quality of code without changing its behavior.

Let me give you an example. Suppose a programmer has two methods (A and B) that each contains five identical or repetitive statements. In this case you refactor in this way:

Ø        Identify five identical statements.

Ø        Write a separate common method C that contains only those five identical statements.

Ø        Call C from two parent methods – A and B.

What I achieved by this? Well, I can list them down:

1.        Better readability

2.        Better maintainability since I reduced duplication. The duplicated code is now in a single location, method C.

3.        Method C can be reused in other places when required.

When refactoring, take care of the following:

1. If the new method C (above example) is a procedure (does not return any value) and just executes the statements, refactoring is easier.

2. If method C (above example) is supposed to do initialization operation or is supposed to return value to its calling method A and B or it is supposed to have certain signature (input parameters), assure you are taking care of these factors.

Try this code snippet:

Before Refactoring:
 
/* populate combobox1 with names */
private void PopulateCombo1()
{
String[] names = { "Vick", "Dave", "Miranda", "Felix", "Joseph" };
       comboBox1.Items.Clear();
       comboBox1.Items.Add(names[0]);
       comboBox1.Items.Add(names[1]);
       comboBox1.Items.Add(names[2]);
       comboBox1.Items.Add(names[3]);
       comboBox1.Items.Add(names[4]);
}

/* populate combobox2 with departments */
private void PopulateCombo2()
{
String[] departments = { "Sales", "Marketing", "Engineering", "Human Resource", "Administration" };
       comboBox2.Items.Clear();
       comboBox2.Items.Add(departments[0]);
       comboBox2.Items.Add(departments[1]);
       comboBox2.Items.Add(departments[2]);
       comboBox2.Items.Add(departments[3]);
       comboBox2.Items.Add(departments[4]);
}

After Refactoring:

/* populate combobox1 with names */
private void PopulateCombo1()
{
String[] names = { "Vick", "Dave", "Miranda", "Felix", "Joseph" };
       PopulateCombo(comboBox1, names);
}

/* populate combobox2 with departments */
private void PopulateCombo2()
{
String[] departments = { "Sales", "Marketing", "Engineering", "Human Resource", "Administration" };
       PopulateCombo(comboBox2, departments);
}

/* populate a given combobox control with a given String array */
private void PopulateCombo(ComboBox control, String[] sArr)
{
control.Items.Clear();
       foreach (String item in sArr)
           control.Items.Add(item);
}
 
 In the above example, methods A and B turn to be PopulateCombo1() and
PopulateCombo2(). Method C is PopulateCombo(). The "duplicate statements" in PopulateCombo() are replaced by foreach loop to give the code professional look.

Hope this helps!

Programmers with C# skill may wish to go through following:

http://www.jetbrains.com/resharper/features/code_refactoring.html

I used ReSharper tool that could be integrated with Visual Studio. You may wish to analyze the code manually and try to refactor the code. If you aim to refactor in large scale or if you want to refactor multiple functions, you may take help of a refactoring tool to avoid issues leading delay.

I cannot suppress my temptation to mention a few lines from Mike Cohn:

“Refactoring is not only crucial to the success of TDD, but it also helps prevent code rot. Code rot is a typical syndrome in which a product is released, its code is allowed to decay for a few years, and then entire re-write is needed. By constantly refactoring and fixing small problems before they become big problems, we can keep our applications rot free.” Robert C. Martin calls this the Boy Scout Rule.

The Boy Scouts of America have a simple rule that we can apply to our profession: Leave the campground cleaner than you found it. If we all checked-in our code a little cleaner than when we checked it out, the code simply could not rot.

Refactoring should be made a regular practice. If the whole team needs couple of days to refactor in every iteration/sprint, then probably it is a sign of different problem. If the team insists the product owner not to bring any change in current sprint because they are refactoring the current code, then again probably it is a sign of different problem. In these cases refactoring probably can be part of product backlog itself.

Reference:

1. http://www.extremeprogramming.org/rules/testfirst.html