Opinion: Test-Driven Development Is Not About Testing

Discussions

News: Opinion: Test-Driven Development Is Not About Testing

  1. Dan North, of ThoughtWorks, has written a piece on Test-Driven Development. If you intereview with ThoughtWorks in the future, watch out for the question: "What is the point of test-driven development?". The answer isn't to do with testing at all according to Dan. Instead say something like "It's about seeing how little you actually need to do and how cleanly you can do it!".

    To read more of Dan's thoughts on Test Driven Development, read the article Test-Driven Development Is Not About Testing.

    How many of you can put your hand on your hearts and say that you are doing TDD right now? :)

    Threaded Messages (36)

  2. So what are the impediments?[ Go to top ]

    I've done TDD and yet still find myself knuckling under, reverting to "blowing code" when under time pressure. I can't imagine why anyone wouldn't want to work test-driven, yet so many resist it or falter in their initial attempts. What are the roadblocks? What's stopping everyone from doing TDD? It seems so common-sense....
  3. So what are the impediments?[ Go to top ]

    What stops test-driven development in most cases is client demand or the pressure of a very short timeline. If you look at common project approaches, testing is usually one of the final steps of a timeline, phase or iteration. It is also the part of the project that is most encroached upon by every other facet of the project and the least likely to receive adjustments if encroached upon. Even in methodologies where testing is a part of a phase or iteration, a client will suggest cutbacks in the testing portion. This is extremely foolish and reckless and is the number one cause of timelines slipping.

    Convincing a client to adopt a test-first or test-driven approach is quite a challenge and one worth fighting for. However, economic demands force our hands far more often than we'd like. Hopefully test-driven development will become as common-place in our projects as patterns have become in design and implementation.

    Fight the good fight. Just don't be surprised if the client wins.
  4. So what are the impediments?[ Go to top ]

    "Convincing a client to adopt a test-first or test-driven approach is quite a challenge and one worth fighting for. However, economic demands force our hands far more often than we'd like."

    Your statement here says one of two possible things:
    * Test-driven development isnt the most economic way to develop the solution in acceptable time, budget, quality and maintainability. If this is the case, test-driven development doesnt and shouldnt have a place in the development process. Its as simple as that, if theres a more effective way to the same end-result, use that instead!

    * "Us nurds" arent good enough at explaining and rationalizing the quality and economic benefits of a test-driven approach to management, hence it is not used although it should be according to the previously stated "rules".

    I dont believe in absolute truths, but I tend to lean towards the latter example in most cases. But this still is food for thought: why do people find it hard to buy into the benefits of test-driven development? Is there a good reason to their hesitance?
  5. Re: So what are the impediments?[ Go to top ]

    "Us nurds" arent good enough at explaining and rationalizing the quality and economic benefits of a test-driven approach to management, hence it is not used although it should be according to the previously stated "rules".

    Well... it's not that simple. The reason why it is so hard to convince a client that TDD is a good thing is :
    - he knows that what it will cost (more time spend)
    - he doesn't know how much it will gain.

    Usually, what a client want is figures showing the ROI. With TDD, estimating it is really hard.

    BTW, my client is quite caricatural. We don't even find ways to convice him to buy modelling tools... and we work on a web site with a budget of $4.000.000 a year!
  6. Re: So what are the impediments?[ Go to top ]

    I can see where you're comming from with:

    >> "Usually, what a client want is figures showing the ROI. With TDD, estimating it is really
    >> hard."

    It would be good to have these figures. I agree

    In the absense of specific costs, I've seen text about the ROI benefits of testing as early as possible (which TDD is endorsing) that youself, and other people, may find useful. Off hand, the following are a useful source: "rapid development", McConnel,1996,Microsoft Press International, ISBN: 1556159005 (well worth buying in MHO);, and "The Econonic Impacts of Inadequate Infrastructure for sofware testing" (http://www.mel.nist.gov/msid/sima/sw_testing_rpt.pdf), USA's National INstitute of Standards and Technology (NIST) . Anybody know any others? . These text should give you enough general source information for justifying TDD on a ROI basis. Basics of the texts are: the longer you leave a bug the more its costs (cost rises exponentially with length of time). Infact, if my memory serves me right, Beck's original motivation for test first (in extreme programming and then TDD) was to flatten this exponential bug cost curve. You could check there for more justification as well. Good luck. Also, if anybody has found a justifaction that worked with their boss/client then please post..... I would find it useful & i'm certain that many others would find that information useful as well

    Personally, I've seen a project or three where more money was spent fixing bugs (which the customer didn't pay for) than spent on development!! You can get into the nasty situation where the customer is annoyed because they can't do their business (because the software doesn't work) and you're company is losing money fixing the bugs (because you have to pay for them). Do you think that the customer came back? and worse.. you know what they say about unhappy customers talking to other customers about you.... If only the developers had convinced the PM to release a little less code/extending timelines & spent some time focusing on quality issues like unit testing instead: everybody would have been happier...

    To me, its the age old problem of communication (something that we 'nurds' are traditionally bad at - myself included): this is what Wille said earlier. I agree with him. I believe that we need to educate our peers about the benefit of the TDD/unit testing approach (to reiterate myself). Its a tough job & I still can't get the idea through to some of my peers; I need to try a lot harder with these people. IF your boss/customer doesn't understand the consequences of not using TDD then they won't want to pay for it. As many a wise person has said, you should start with yourself when blaming somebody (for not having time to do TDD!)..

    p.s Anybody have any horror stores of spirling costs where TDD may have worked? These cases might help use justify TDD. Any costs involved would be very useful...

    Cheers

    Andy
  7. Development Model[ Go to top ]

    My 2 Cents.
    TDD needs to be forced on developers who have no discipline in coding. A lot of developers try to write about a few 1000 lines of code before even trying to compile it and run it. By forcing them to write unit tests you can try to get a measure of discipline in the coding process.
    A general misconception is that Unit Testing is about testing and can replace the QA process. Unit tests only validate what the developer *thinks* is the right behaviour. So in all likelyhood if S/he has bugs in the code S/he will have the same bugs in the Unit test. Of course, a good developer will catch a few bugs when he starts to write unit test code. Most developers simply copy paste the function to the unit test and make the necessary changes to ensure that the test passes.
    A good number of Unit tests are not a good metric to measure the quality of the code. In fact they shouldn't even be part of the QA process. They are useful only as a development tool while refactoring but then again that depends on the quality of the unit tests as well as your developers.
  8. So what are the impediments?[ Go to top ]

    What's stopping everyone from doing TDD? It seems so common-sense....


    In my experience there are three major factors:

    1. developers knuckling under.
    2. developers blowing code.
    3. developers under time pressure.

    Taylor
  9. Faster dev[ Go to top ]

    I dunno about TDD, but I found that in serious projects we spent more time in the test/debug cycle than in the initial development (of course this was for shrink-wrap, not custom programming for a client). So anything that reduces the test/debug cycle reduces the time needed to ship, even if dev time increases.

    The dramatic results I got weren't w/ TDD, but w ASSERTs that are NOT turned off in production and that log errors to a file (this was not performance critical). I reduced the test/debug cycle to nothing because way before my code could even get to the tester I had to pass all my numerous ASSERTs. When it got to QA, she found 1 bug. Took me less than 5 minutes to fix because it was in the log. Never heard about any bugs in that code after that (and it was pretty complex too)!
  10. I believe the phrase Dan North is looking for is Design By Contract.
  11. Contract is certainly a part of it[ Go to top ]

    The main benefit, however, comes from using the test case to drive out the expected interactions and behaviour of an object, which goes further than pre/post conditions and invariants.
  12. If read OOPSLA proceedings posted on this site, you can find that TDD might be downright risky.
    You can end up with really short-sighted design without a overall theme arising from a lack of will to understand the big picture by being lost in the details.

    I'm not saying TDD is not good, but I'm suspicious of it being promoted as a cureall.

    Every now and then a fad comes along.
    Speaking cool acronymns may be a marketing help.
  13. I agree tdd can be risky, though useful at times, the code may require more maintenence down the road. The correctness of the tdd viewpoint may depend on your role within a project. A architect's role is to favor a design that solves as many potential problems as possible while keeping in mind the specific issues at hand. A programmer's role is to only solve the specifics using minimal effort.
  14. TDD is a good practice at the developer level but it can't scale when you have teams over a certain size. In such a case, you usually have a QA department about the same size which will do a much better job at writing tests than developers, if only because by then, the testing framework has become fairly convoluted and unless special precautions have been taken, it will be a much bigger challenge for a developer to make a test part of the framework than for a QA person.

    Ideally, you will still see developers provide embryos of test to their QA counterpart, but even this is surprisingly hard to achieve past a certain size.

    And of course, TDD just goes out of the window when you are approaching a deadline anyway.
  15. I agree that most projects need a separate QA team to fully test an application.

    However, I'd like to point out that Beck and other advocates of TDD never claimed that TDD would cover ALL aspects of testing on a project. In fact, in his book on TDD, Beck clearly notes that TDD is for unit testing. He also notes problems in trying to apply test first principles to application level testing. He refers to this as Application Test-Driven Development (ATDD).
  16. However, I'd like to point out that Beck and other advocates of TDD never claimed that TDD would cover ALL aspects of testing on a project. In fact, in his book on TDD, Beck clearly notes that TDD is for unit testing. He also notes problems in trying to apply test first principles to application level testing. He refers to this as Application Test-Driven Development (ATDD).


    Seems like splitting hairs a bit doesn't it. ATDD vs TDD. Testing in general is about feeding in a set of values and comparing output to a corresponding set of expectation values whether on a unit level or application or service level.

    The problem with TDD is where the responsibilty lies for defining and managing test cases. When specifications are a moving target, which is a given in XP development environments and drive the test cases, the TDD tasks can overwhelm the actual development tasks of providing a product. Also, the test environments invariably become interwined in the architecture. It is better to have a separate QA test group as the responsible party. In this way, there is the separation of concerns mentioned in previous posts as well as a "double blind" testing methodology that in the long run will provide better metrics on the application.

    - Frank Bolander
  17. Which part of "is not about testing"?[ Go to top ]

    TDD is a tactical development practice. You still need a high-level strategy (let's call it "enough design up front") so that you know roughly where you are going. Test-driven development drives out the detail, and prevents you going down blind alleys and writing code that you don't need.

    Testing requires testers. Good testing doubly so. There is a whole science around decent software testing that as a programmer I have huge amounts of respect for.

    TDD doesn't preclude or replace user and system testing. It just pulls the rug out from under the people who think you can know everything about your solution up front, leaving the "trivial" task of churning out the actual code.
  18. From the article:

    We start by writing some client code as though the code we want to develop already existed and had been written purely to make our life as easy as it could possibly be. This is a tremendously liberating thing to do: by writing a model client for our code, in the form of a test, we can define programmatically the most suitable API for our needs. In addition, we assert the behavior we want.


    These test clients should reflect the use cases of the required system - no more or no less as you put it. So I'm writing a test client but it's not testing, I'm just trying to get it to compile. OK. No harm up to this point.

    How do you know the test client is meeting the requirements of the system? Just because it compiled. You have to run use case data through it to make sure the output you're getting is consistent with the expectations of the business. This is definitely testing.

    Now, not only do you have an iterative lifecycle for your code development, but also your testing development which are highly coupled.

    > TDD is a tactical development practice. You still need a high-level strategy (let's call it "enough design up front") so that you know roughly where you are going. Test-driven development drives out the detail, and prevents you going down blind alleys and writing code that you don't need.
    >

    I my experience(last two out of three projects), TDD leads to top-down functional decomposition of complex systems that have just as many blind alleys as other techniques.

    How is(from the article):

    This then becomes your development rhythm: write a test, write some code, refactor.

    Any different than removing blind alleys and removing code that wasn't needed. Distillation takes a lot of energy and resources($$$ and time) no matter how you phrase it.

    > Testing requires testers. Good testing doubly so. There is a whole science around decent software testing that as a programmer I have huge amounts of respect for.

    Me too. That's why it should be orthogonal to the rest of the application development process.
  19. TDD is a good practice at the developer level but it can't scale when you have teams over a certain size. In such a case, you usually have a QA department about the same size which will do a much better job at writing tests than developers, if only because by then, the testing framework has become fairly convoluted and unless special precautions have been taken, it will be a much bigger challenge for a developer to make a test part of the framework than for a QA person.

    >
    > Ideally, you will still see developers provide embryos of test to their QA counterpart, but even this is surprisingly hard to achieve past a certain size.
    >

    I wouldn't agree.
    My personal belief is that the bigger the project the more needed is TDD. Usualy BOTH developers and QA engineers are involved in the testing. Developers
    by developing unit tests and QA persons by all the other means of testing.
    Therefore the developers take part in the testing without having the knowledge of an experienced QA engineer. Developing unit tests is not the same as developing test cases for example.

    > And of course, TDD just goes out of the window when you are approaching a deadline anyway.

    Yes, most likely it will happen but not because TDD added extra time but rather the estimatie was not quite accurate. I think using TDD will not add more man/hours to the overall estimate if applied properly. It will add time during development but will ease you at the end of the project and during the maintenance. I think it is a good practice writing tests first but it takes time to realize it and get used to it.

    Kalin
  20. <raj>
    You can end up with really short-sighted design without a overall theme arising from a lack of will to understand the big picture by being lost in the details.
    </raj>

    Quite contrary, the most well-designed frameworks I've used were TDD-ed. The reason is very simple: with TDD you produce more modular/pluggable/flexible code, and the design is minimal.

    Ara.
  21. sorry just a test
  22. Quite contrary, the most well-designed frameworks I've used were TDD-ed. The reason is very simple: with TDD you produce more modular/pluggable/flexible code, and the design is minimal.


    Yes, frameworks and libraries are very much 'unit testable', but the applications that are using them are much less 'unit testable' and much less suited for TDD. For business applications you can write mich less usefull unit tests then for frameworks and you need a separate QA team for any serious app.

    Mileta
  23. business applications are MOST certainly unit testable. often, business applications are designed by non-developers which can make developing the application frustrating, but that's a different issue i suppose. its still very possible to test your businesss logic and do it in a tdd type of methodology. ejb's are unit testable, web pages are unit testable, i suppose there's unit tests for gui interfaces too but that's not my area these days.

    business applications will also always have a qa team (i hope). normally they only validate the external interfaces of an application and do integration testing. does the ordering system get orders to the delivery system propertly? that's a different level of testing than tdd (black .vs. white box perhaps).

    tdd gives developers comfort when they make a change to item A, that it won't impact expected functionality of item B that uses item A. there's also the possibility that a particular test doesn't exist and things slip through, but we're only human.
  24. Frameworks are unit testable, but...[ Go to top ]

    business applications are MOST certainly unit testable


    No they are usually not. Not only do you lack the environment for unit testing (stable, repeatable datasources to name only the obvious), the test descriptions are enormous because for a lot of applications there is no formalized description. The software itself more often than not is the only formal description available.
      
    > tdd gives developers comfort when they make a change to item A, that it won't > impact expected functionality of item B that uses item A. there's also the
    > possibility that a particular test doesn't exist and things slip through, but > we're only human.

    Right, this is what testing is all about. It ensures that an application in a certain scenario behaves as the developer expects. "A particular test does not exist" is some statement. In a normally complex application about 98% of the tests will not exist. It is probably much better to use assertions because this makes sure that an application always behaves in the expected way. And they are far easier to maintain as well...
  25. Frameworks are unit testable, but...[ Go to top ]

    No they are usually not. Not only do you lack

    >the environment for unit testing (stable,
    >repeatable datasources to name only the obvious),

    Mock Objects and the ObjectMother pattern are solutions for providing your unit tests with "stable, repeatable datasources".

    Saying business applications are usually not unit testable is an odd statement. If you're doing OO you're creating objects with public methods. The public methods have return values or they modify the state of the object or the state of contained objects. All of this behavior can be tested with unit tests.

    Assertions have their place. For that matter, so do logging statements. However, neither is a substitute for unit testing.
  26. For some value of 'unit'[ Go to top ]

    business applications are MOST certainly unit testable

    >
    > No they are usually not. Not only do you lack the environment for unit testing (stable, repeatable datasources to name only the obvious), the test descriptions are enormous because for a lot of applications there is no formalized description. The software itself more often than not is the only formal description available.

    TDD does not address that the code does the right thing, merely that it does the thing right. 'The thing' being what the developer intended it to do. The QA and formal acceptance testing process is there to ensure that what the developer thought it was supposed to do is what it actually is supposed to do.

    As an aside, bugfixing TDD'ed code is a truly joyful experience. Its usually possible to write one extra test case that highlights the bug that QA found, write the code that fixes the bug, and run the entire unit test suite to make sure you didn't break anything else, then check in. By comparison, bugfixing tightly coupled non-TDD code is more akin to a jaunt through the seventh level of hell.
  27. Frameworks are unit testable, but...[ Go to top ]

    Yes, frameworks and libraries are very much 'unit testable', but the applications that are using them are much less 'unit testable' and much less suited for TDD. For business applications you can write mich less usefull unit tests then for frameworks and you need a separate QA team for any serious app.
    It depends on the framework. A major goal of framework design should be to maximize testability of applications built on the framework, and (ideally) minimize dependence of application code on framework APIs. Inversion of Control is a powerful way of doing this. A good application framework such as Spring helps make application code more testable than doing without a framework. As Howard pointed out, so does HiveMind (his IoC solution).

    In my experience, a framework that reduces testability of application code (and I've seen a few, especially developed in house) is a bad framework.

    Regards,
    Rod
  28. Raj,

    We claim to use JUnit and eXtreme Programming where I work, but that's mostly lip service. I've tried it at home, however, with some little test projects I'm working on and I love it. Writing the tests first from a "client" point of view forces you to think about writing testable code. If you can't test it, what good is it? Plus building up that suite of unit tests as the project progresses is very powerful. I made what I thought was a subtle change, reran the test suite and quickly saw that my change had many other implications. Without that suite of unit tests, and the idea of refactor/retest frequently, I would not have found that out until much later.

    But, as you mentioned, it doesn't provide much guidance on overall architecture. So that's a bit of a disconnect. I guess you would need to start with a roughed out object model complete with sequence diagrams for object interactions and work from there?

    Thanks,

    Paul Carter
  29. As Dan says, TDD is a tactical thing. When you're in the middle of coding, it helps keep you from adding unnecessary complexity, and it let's you see how your code is going to be used even before it is.

    However, that's only part of the story. The other pieces are _enough_ design (up front or otherwise), and merciless refactoring later.

    If you don't continuously think about where you're going and how things fit together, and if you don't keep your code clean by refactoring constantly, all the testing in the world is not going to give you a good architecture. To come up with a good overall system architecture, especially on large projects, you do have to think at least a little about what direction you're going to go in ahead of time. The beauty of good unit testing is that it's not a big deal when you're wrong (because you will be, even if you're right today, you'll be wrong when you get those new requirements.)

    TDD helps you design the detailed stuff right the first time, and enables you to fix the bigger stuff later.

    In a system with very low duplication and high automated test coverage, you don't have to get it right the first time, because changing your mind later is about as expensive as changing it earlier. If you haven't worked on such a system, you should, it's such a different experience. No matter how dramatic a change you make, in a few seconds you know if you broke anything.

    Only unit testing can give you a system like this, because any other testing introduces duplication. Say you have 500 automated end to end tests with 95% test coverage. Let's forget the fact that it takes you an hour to run them all. Suddenly there are changes to the code that you can no longer make, because they would break so many of the tests that the team couldn't afford the time to fix them all. So crud in the code builds up, or the tests get thrown away. Either way, BAD.

    Unit tests on the other hand can be and should be as fine grained as possible so that most changes no matter how radical don't affect many tests. You can use techniques like mocking to help in this. Ideally, a test should test exactly one method/class/small chunk of functionality, and it shouldn't break if anything else changes.

    And TDD _does_ scale. I've seen teams of 30 developers with suites of 1000's of unit tests that run in less than a minute. And their code stays maleable. When people think of a way to make it better, they can. In fact I wouldn't work on a team that size if people weren't writing tests, because the code would deteriorate into spaghetti.

    For the record, I also am a huge advocate of having testers and other types of testing: acceptance, integration, end-to-end, functional, exploratory, etc. These all can come in very handy and find holes in your product, unit tests and requirements.
  30. I can't quite get to try TDD (i.e., write the test, then write the code to make the test compile and execute clean).

    However, I do stay disciplined: write the code, write the test, execute, check code coverage, add more tests, etc.

    I have long maintained (back even into my PL/1 coding days) that "coding for testability" is a good thing; you create better code with better seperation of concerns and abstraction, when you design it to be tested/testable.

    I always take code coverage figures as a challenge.

    I've found that coding and using HiveMind, I've been able to reach fairly high code coverage numbers (> 90%) relatively easily. And I *always* find bugs when I increase CC numbers. Really stupid braindead errors. I consider myself an A+ coder, but even in my code: you write tests, you find bugs. It's motivating.

    Last bit o' plug: because HiveMind creates proxies for services, it is possible to take a monolithic service and divide it into two mutually dependent pieces, and not get tripped up with a "chicken-or-the-egg" instantiation order problem. Service A will get a proxy to service B. Service B will get a proxy to service A. Which service actually gets instantiated and when is determined at runtime.
  31. That should say "I can't quite get to TRUE TDD", not "try" (which is just about grammatical, but confusing).
  32. What's the need?[ Go to top ]

    If i understand correctly, this says write a model client as if though the whole piece of software exists. But when you do so, your code will go through lot's of changes including design/architecture. Also sometimes, when you integrate your components with other external components you feel the necessity to change the interfaces. I know by doing proper planning/design you can avoid it, but in real world ???

    So, you finally end up modifying your model client all the time, whenever there is a change in design. I can't really see how this will benefit me. Why can't we write this model client once your design/architecture becomes stable?, That's what e'body normally does in a software development.

    Regards
    Zulfi
  33. In my opinion (experienced coder) TDD is a good thing for fast development of medium projects with little number of experienced developers (especially in OOP). They do know how to apply patterns, desing by contract and many other things because they are experienced.

    This way it´s easy to refactor the code and when the project is well defined or well known (looks like something you´ve done before) little problems are found at the end of each phase.

    But in large projects or with not so good developers you need to divide the work clerver or you will find that big "chicken-or-the-egg" problems will appear when you discover the things that fell in nobody´s land or known OO problems undiscovered by inexperience.

    That´s why TDD is so apreciated by XP people, because they usually belong to this group.

    You go back to the "old way" of development but knowing how to do "things right" and that´s too much difference.
  34. I've found that even under tight timescales TDD is the way to go. In the case of applications requiring an application server this is especially true. Most times I that I skip writing the test I find that I just replace running the unit test over and over with a whole build, deploy, restart (or reload if you have hotdeploy), along with navigating to the right part of the application.

    Combine this with the time saved in reduction of bugs in the code that is released (which take a silly amount of time to track down / ( build / deploy / restart / navigation )+ rerelease) and TDD makes for even more rapid development than rapid development.
  35. How do you unit test a GUI?[ Go to top ]

    Does anyone have a good answer to testing and replicating a user interacting with a GUI?
  36. RE: How do you unit test a GUI?[ Go to top ]

    Does anyone have a good answer to testing and replicating a user interacting with a GUI? <

    1) Put as little code as possible in the GUI. Create a class for all the other client code (lets call it the Presenter). Create an interface to the GUI (let's call it the View). Have the GUI implement the View interface. The Presenter can only access the GUI thru the View. When testing the code, have your unit test implement the View and verify the Presenter is updating it correctly when you call methods on the Presenter

    2) For the GUI stuff you just have to test, use JFCUnit

    Regards,
    John
  37. I think the concept of TDD is solid, in the sense that it helps to flesh out use cases for whatever level you're working at.

    If you're writing APIs, then TDD helps capture the ways that someone might use that API both in normal and exceptional cases. This does tend to produce a cleaner API.

    If you're writing business application code, TDD helps act a a first "user" of the system, which again helps to crystalize specs or high level designs into the simplest, most effective application design.

    I think this is just a different way of developing failry low-level use cases for a system (usually done by writing documents). It has the benefit of actually providing some level of confidence for making changes (change detection), as well as helping you think about the design of the classes.

    This isn't an accident: incrementally building a system forces us to think in terms of how it will be used, and what can go wrong. Too often as developers, we focus too much on the technology and architecture that will be used (which frameworks, EJB containers, caching mechanisms, OR mapping tools, etc.), and too little on the actual system and its business use cases.

    Having said all of that, I think TDD is misnamed: it should be Use-Driven Development (UDD). Testing is only one aspect - its more about an approach to development which has side benefits of having change detectors, and encoding your intentions for use by other developers.