Article: One Assertion Per Test

Discussions

News: Article: One Assertion Per Test

  1. Article: One Assertion Per Test (30 messages)

    Dave Astels writes about "what it's like to make tests as simple and decomposed as possible: aiming for a single assertion in each test." He takes an example that traditionally uses multiple assert*()'s within a test, and refactors it to give multiple tests with one assert each.

    Excerpt

    "I'm convinced writing tests like this is a useful approach. One advantage is that the resulting tests simpler and easier to understand. Just as important, and maybe more so, is that by adding the specification of the behavior one tiny piece at a time, you drive toward evolving the code in small, controllable, understandable steps."

    Do you agree with the author?

    An interesting aside, is that he gives examples in Squeak Smalltalk as well as Java!

    Read One Assertion Per Test

    Threaded Messages (30)

  2. Article: One Assertion Per Test[ Go to top ]

    I wonder if this person has ever written real code out there.

    One test assertion per test... right. What's next... only one method call per method? One only one variable?

    Suites, Classes and Methods are three important pieces of decomposition of your unit testing, and they are all important. Use them wisely, but use them all.

    A quick look at my own test shows an average of five-ten asserts per method, but it's sometimes much more and sometimes much less. Whatever makes sense.

    <sigh>

    --
    Cedric
  3. Article: One Assertion Per Test[ Go to top ]

    Cedric: I wonder if you have ever written some real tests out there ;-)

    I'm not surprised that you dismiss Dave Astels' ideas and his credibility in the field. I think you missed the point of the whole article:

    It is quite common to have unreadable tests. This is usually because they are not focused enough. They test a little bit of this and a little bit of that and are not coherent.

    A bad side effect of this is duplicate tests. Many test methods end up testing the same thing. Your tests become hard to maintain.

    With Astels' one assertion per test you don't get duplication in tests. And it makes you even more focused on your current task when you do TDD.

    I think one assertion per test is a great goal. You don't have to follow it religiously. Just strive for it.

    As for your out-of-the-air comparison with number of variables and method calls per method, here is my rule of thumb:

    7 lines of code per method.
  4. Article: One Assertion Per Test[ Go to top ]

    Aslak:
    With Astels' one assertion per test you don't get duplication in tests.

    How does "one assert per test method" guarantee you don't have duplication?

    For example, imagine I have about five asserts per method. I refactor this and I now have five methods. Multiply this by the number of tests I have and suddenly I now have hundreds of test methods.

    Do I have a better grasp on the design of my tests? Nope.

    You don't make design happen by applying automatic rules like this one.

    Testing is no different than standard development: only judgment and design skills will help you make your code better architected, not simple and baseless syntactic rules like the one suggested by Astel.

    --
    Cedric
  5. Article: One Assertion Per Test[ Go to top ]

    Aslak:

    > With Astels' one assertion per test you don't get duplication in tests.
    >

    > How does "one assert per test method" guarantee you don't have duplication?
    >

    I didn't mean to say that it guarantees non duplication. I meant to say it reduces the likelyhood of it happening.

    Here are some simple guidelines to help you reduce the risk of dupes:

    (Recap of the TDD rythm):
    1) Write a test
    2) Make the code compile (the simplest possible way)
    3) Make the test pass (the simplest possible way)
    4) Refactor
    (start again on 1)

    If you follow this, and accidentally write a passing test, you will get a green bar on 2. This is an indication that your recent test was a dupe and maybe should be deleted.

    > For example, imagine I have about five asserts per method. I refactor this and I now have five methods. Multiply this by the number of tests I have and suddenly I now have hundreds of test methods.
    >
    > Do I have a better grasp on the design of my tests? Nope.
    >
    > You don't make design happen by applying automatic rules like this one.
    >
    > Testing is no different than standard development: only judgment and design skills will help you make your code better architected, not simple and baseless syntactic rules like the one suggested by Astel.
    >

    Experience is important, but don't neglect the importance of sound guidelines and best practices. Sometime guidelines have to be a bit extreme in order to make people think differently.

    (It's Astels, not Astel).

    Aslak

    > --
    > Cedric
  6. Article: One Assertion Per Test[ Go to top ]

    Cedric: I wonder if you have ever written some real tests out there ;-)

    >
    > I'm not surprised that you dismiss Dave Astels' ideas and his credibility in the field.

    Yes, I've written real tests. I've been playing this game for close to 25 years.

    > With Astels' one assertion per test you don't get duplication in tests. And it
    > makes you even more focused on your current task when you do TDD.

    Yes, very much so.

    > I think one assertion per test is a great goal. You don't have to follow it
    > religiously. Just strive for it.

    Exactly.

    > As for your out-of-the-air comparison with number of variables and method
    > calls per method, here is my rule of thumb:
    >
    > 7 lines of code per method.

    I start getting edgy at 5 :)

    Dave
  7. Article: One Assertion Per Test[ Go to top ]

    Dave:
    Yes, I've written real tests. I've been playing this game for close to 25 years.

    Okay great, so how about commenting on my post instead of Aslak's?

    I am not the only one having a few issues with your post, so why don't you take this chance to clarify your position?

    --
    Cedric
  8. Aslak: As for your out-of-the-air comparison with number of variables and method calls per method, here is my rule of thumb: 7 lines of code per method.

    Hmm .. have you ever written any code in the real world? Who is this "Aslak Hellesoy" guy anyway?

    ;-)

    Peace,

    Cameron Purdy
    Tangosol, Inc.
    Coherence: Clustered JCache for Grid Computing!
  9. Article: One Line of Code Per Method[ Go to top ]

    Aslak: As for your out-of-the-air comparison with number of variables and method calls per method, here is my rule of thumb: 7 lines of code per method.


    :-) 7 lines of code/method?!? 5 lines of code/method?!? I don't know how realistic this is but this makes me smile. Or... maybe... you use multi-statement lines :-)

    Have you ever caught exceptions when you write code? I'm asking because I tried but I couldn't find a way to write this in less than 7 lines:

    try {
      goSomething();
    } catch(SomeException e) {
      doSomethingElse();
    } finally {
      doSomethingFinally();
    }//try-catch-finally

    If I can find some spare time, I'll try to write 1-line-unit-tests testing 5-line-methods...
  10. Article: One Line of Code Per Method[ Go to top ]

    maybe he writes it as:

    try {goSomething();} catch(SomeException e) { doSomethingElse(); } finally { doSomethingFinally();}//try-catch-finally

    Suweet.

    Jason McKerr
    The Open Source Lab
    "Open Minds. Open Doors. Open Source."
  11. Article: One Assertion Per Test[ Go to top ]

    A bad side effect of this is duplicate tests. Many test methods end up testing

    > the same thing. Your tests become hard to maintain.

    Hmm, I consider redundancy in my unit tests to be perfectly acceptable and even a good thing.
  12. Article: One Assertion Per Test[ Go to top ]

    Using more than one assert per test quickly transforms "unit tests" into "system tests". When I run my test suites, I want to be able to look at the method that failed and instantly know the gist of the issue. Without restricting each unit test to a single assert, this isn't possible.

    This is a very similar to a new habit I've formed - avoiding complex lines of code. There is nothing worse than getting a NullPointerException on a line like:

    String s = new String(foo.getBar() + bar.getBlah() + crap.whichOneFailed());

    A NPE here means I have to open up the debugger. If I just restructure the line above to be:

    String bar = foo.getBar();
    String blah = bar.getBlah();
    String knowExactlyWhichOneFailed = crap.whichOneFailed();
    String s = bar + blah + knowExactlyWhichOneFailed;

    I can more easily diagnose the problem. Granted, I might still have to turn my debugger on to find out why an object I expect to be around is null, but at least I know where to look the first time through.

    I suspect that if you try the single assert per test, you'll find your life and the lives of those that work w/ you are much easier.

    KISS strikes again.

    - Ryan
  13. Article: One Assertion Per Test[ Go to top ]

    Ryan:
    Using more than one assert per test quickly transforms "unit tests" into "system tests". When I run my test suites, I want to be able to look at the method that failed and instantly know the gist of the issue. Without restricting each unit test to a single assert, this isn't possible.

    Notice that you subtly changed the point of the original article.

    Astel thinks his design rule makes your test design better. You are saying that it makes the output easier to diagnose.

    These are very different things, but once again, I argue that good diagnostic has nothing to do with the number of asserts per method. Just take the time to write a meaningful failure message when you write your assert:

    Don't use:

    assertEquals(c.size(), 1);
    assertTrue("Cedric".equals(((Employee) c.iterator().next()).geFirstName());

    use:

    assertEquals("Expected a different number of employees",
                  c.size(), 1);
    assertTrue("The employee should be named Cedric",
           "Cedric".equals(((Employee) c.iterator().next()).geFirstName());

    Notice that in the first case, splitting the two asserts in two methods will certainly not help your understanding of the failure, except maybe with the vague hint of the test method name. But seriously, what do you prefer to see, "testFirtName() method failed" or "The employee should be named Cedric"?


    This is a very similar to a new habit I've formed - avoiding complex lines of code. There is nothing worse than getting a NullPointerException on a line like:

    For your information, this is typically referred to as the Demeter Principle.

    --
    Cedric
  14. Article: One Assertion Per Test[ Go to top ]

    Cedric:

    I don't have a problem with meaningful failure messages. My issue is with several different test cases being written within one method. You example is perfectly fine in my view. I run into issues where people combine 12 different test cases into a single method and then do not realize the extent of a problem becuase they see one failure. This is very poor test design, but is very common.

    I do think you have an excellent point via your example. The goal should not be the number of asserts as a hard and fast rule but in the scoping of the test case.

    - Ryan
  15. Article: One Assertion Per Test[ Go to top ]

    Ryan, I think you're missing a point here: the vast majority of the time, your tests will pass (or else you have much bigger problems). Testing at too fine a granularity means you're wasting time that could be better spent elsewhere.

    Put another way - most of the time, a test won't fail (even an "aggregate" tests). Most of the time, when a test fails, it's obvious why. In the few cases where it's not obvious, well then you've got some work to do :-)

    As with everything else, you want to target the common cases, not the corner cases. It sounds to me like you're working extra-hard to accomodate a corner case, and as a result you're working extra-hard when you generally don't need to.

         -Mike
  16. Article: One Assertion Per Test[ Go to top ]

    Mike:

    You and I have different ideas about what type of testing is required! It is time to agree to disagree.

    But for the record, as someone that develops applications that have to run on both Websphere and Weblogic, it would make my life much easier if the "corner cases" of the J2EE specs were tested consistently. It is nearly impossible to really use J2EE w/o application server-specific logic - probably in a large part to a "target the common cases, not the corner cases" attitude.

    - Ryan
  17. Article: One Assertion Per Test[ Go to top ]

    \Ryan Ashcraft\
    You and I have different ideas about what type of testing is required! It is time to agree to disagree.
    \Ryan Ashcraft\

    OK. But, as I said, I don't understand the value in artificial testing boundaries and nomenclature.

    \Ryan Ashcraft\
    But for the record, as someone that develops applications that have to run on both Websphere and Weblogic, it would make my life much easier if the "corner cases" of the J2EE specs were tested consistently. It is nearly impossible to really use J2EE w/o application server-specific logic - probably in a large part to a "target the common cases, not the corner cases" attitude.
    \Ryan Ashcraft\

    Most of the problems I've seen in application servers would not be caught by the sort of unit tests you're talking about. They usually involve complex lifecycle issues and many objects interacting - specifically the type of thing that fine-grained unit tests do _not_ catch.

        -Mike
  18. As a followup...[ Go to top ]

    As a followup to show what I'm talking about. Consider a system with 200,000 SLOC's (this is the realm I work in) which is maintained over the course of several years. Using (admittedly naive) back of the napkin calculations, with your indicator of an average of 7 lines of code/method, you now have 28,000 methods (!).

    Each of those 28,000 methods would seem to have a corresponding unit tests - so we have 28,000 unit tests. This is _only_ unit tests - we haven't done any larger scale tests at all.

    Imagine now _just_ maintaining the unit test code base - how many man-years of effort are you talking about just for unit tests? When major changes hit the code base (as they always do), how much effort is there _just_ to update the unit tests? Again - this doesn't cover any other tests, just unit tests.

    That's where I'm coming from, and to an extent where I believe Cedric is coming from as well. If you reach for the ultimate small granularity testing, you'll find that as your code base hits 100,000 lines and more that you're literally swimming in unit test code, and that it becomes a burden more than a boon.

    This is why I talk about abstraction and aggregation at the test level as well as the code level. Past a certain threshold really fine grained tests will overwhelm any development team.

         -Mike
  19. Article: One Assertion Per Test[ Go to top ]

    Ryan, what you are describing isn't simplifying the system, it's trading off where your complexity lies. Effectively, you now have smaller, more granular "building blocks", which makes the building blocks individually easier to understand.

    BUT - now you've got more building blocks! You've hoisted the complexity up one level of abstraction. This makes it easier to unit test and understand the individual pieces, and makes it harder to understand the inter-relationships.

    In my own development philosophy, the point of coding is not to make unit tests easier. The point is to make integration tests easier, make it easier to understand the code, and to ensure _overall_ correctness.

    Unit tests are so low-level that they are a minor consideration. To me they're low-level enough that you shouldn't obsess over them too much.

         -Mike
  20. Article: One Assertion Per Test[ Go to top ]

    Mike:

    I never said that I changed my design to make my unit testing easier. Good design should always come first. My point is simply that a "unit test" should test a "unit" - something that is singular. I don't buy into testing multiple units/cases at once because then my test is effectively a system test. System tests are fine but serve a different purpose because they look at the system in a more complete form than unit tests. The way that unit tests help developers is that they cut down on the scope of the problem compared to system tests.

    I don't suspect that you or Cedric are challenging this point. I gather that the issue here is whether or not one believes in system testing or unit testing. Both have their pluses and minuses. In my judgement, both should be used.

     - Ryan
  21. Article: One Assertion Per Test[ Go to top ]

    Ryan:
    I don't suspect that you or Cedric are challenging this point. I gather that the issue here is whether or not one believes in system testing or unit testing. Both have their pluses and minuses. In my judgement, both should be used.

    True. I guess it just boils down to our respective definition of a unit.

    Typically, if I want to test that an object has the right values, I can't really understand what I gain by changing:

    testPerson() {
      assertTrue("Cedric".equals(person.getFirstName());
      assertTrue("Beust".equals(person.getLastName());
    }

    into two different methods "testFirstName()" and "testLastName()" (which also implies the "person" object will be calculated in a third method probably inherited by the test... seems overly complex to me.

    I totally agree with Mike: unit tests should stay structurally simple, no need to obsess over their design as long as you respect the DRY principle.

    --
    Cedric
  22. Article: One Assertion Per Test[ Go to top ]


    > testPerson() {
    >   assertTrue("Cedric".equals(person.getFirstName());
    >   assertTrue("Beust".equals(person.getLastName());
    > }
    >

    JUnit has an assertEquals method that will give you a more meaningful message in case of a failure:

    testPerson() {
      assertEquals("Cedric", person.getFirstName());
      assertEquals("Beust", person.getLastName());
    }
  23. Article: One Assertion Per Test[ Go to top ]

    Total agreement here, Cedric. I prefer to aggregate both functionality and tests into logical units that work in "chunks" that don't necessarily match up to Java language constructs. Java gives us methods, instance variables, classes, and packages. This does not mean that testing or coding should be _logically_ limited to those constructs. Often tests make the most sense when they cross boundaries - the logical "unit" is far more important than the code "unit". So some of what I would call "unit" tests will use multiple methods or (gasp!) multiple objects, and will even scale up and cross package/sub-system boundaries. Alot of people seem to want rigid boundaries between "unit" tests, "system" tests, integration tests, etc - to which I say "Bah!". Such boundaries are both artificial and, often, arbitrary.

    You should code and test at the abstraction level that makes sense to your problem domain and for the code in hand. If you're forcing yourself to call a "unit" the smallest expressable Java coding construct, then likely you're being too granular and not scaling your development effort as well as you could. My rules of thumb:

      - if your testing methods have one-to-one correspondence to real code methods, or:
      - if your number of test classes has a one-to-one correspondence to real classes, or:
      - if the overall size of your test harness equals the size of your real code base

      then:
       - you're too granular, you're spinning your wheels far too much in test code, and not as agile as you could be.


          -Mike
  24. Article: One Assertion Per Test[ Go to top ]

    While refactoring having high test case coverage aids me greatly in detecting when things go wrong.
    The number of tests that do occasionally break gives me a good "feel" for the difficulty/complexity of the refactoring at hand, and shows me quickly the areas in the codebase that are impacted.

    Thus, if I have a method with more than 1 assert, any number may be broken, and I only get 1 failure (as the test method exits). For this reason alone, I prefer one assert per test method. It maximises the visibility of the impact of my change (in that a larger number of tests break).

    The error messages are also a very good indicator of what is going wrong. This helps me re-plan my refactoring or design -> The tests are driving my design....

    Cheers
  25. Article: One Assertion Per Test[ Go to top ]

    Permit me to slightly elaborate on Dave Astel's principle (or what I believe his intention to be): Assert on one object / action per Test Method. What I understood Dave to be promoting is testing one responsibility per method (which will often, but not always, be one assertion).

    public void testMethodThatReturnsCollectionPopulated() throws Exception {
        Collection populatedCollection = someObjectAlreadySetup.getSomeCollection();
        //There are three asserts here but testing only one responsibility
        assertNotNull("Expected populated collection", populatedCollection);
        assertEquals("Expection collection to have 5 elements", 5, populatedCollection.size());
        assertTrue("Expected blah in the collection", populatedCollection.iterator().next() instanceof AClass);
    }

    public void testMethodThatReturnsCollectionEmpty() ...
    public void testMethodThatReturnsCollectionNull() ...

    public void testMethodThatReturnsSomeObjectPopulated() throws Exception {
        SomeObject anObject = someObjectAlreadySetup.getSomeObject();
        //Two asserts here on a different responsbility of the same object that was being tested above
        assertNotNull("Expected not null object", anObject);
        assertEquals("Expected something else", objectExpected, anObject);
    }
    ...

    In this example, someObjectAlreadySetup has a number of methods. Each of these methods are tested in a different test method, rather than asserting everything in one test method; however, a number of assertions are made about each action / responsibility. I think the null check is important in some scenarios so as to prevent NPEs in test methods as this does not pinpoint the failure succinctly - though this should mostly be avoided if adhering to the Law of Demeter. (Yes I realise it's a very thorough unit test; yes I realise that Generics in JDK1.5 will remove the need to do instanceof on Collection elements; it's just an example)

    I don't think Dave was asserting (no pun intended) that all test methods should have one assertion only as a hard-and-fast rule, more that each test method should have one responsibility - one assertion per test guarantees this, if Dave's process is follwed. The underlying concept of simple, decomposed tests is not new. Of course, this should automatically occur when practicing TDD; testing after-the-fact requires more thought to obtain the same sort of cleanliness.

    Coding is not a science, if it were there would be a formula for success. There are no hard-and-fast rules, only guidelines. We as developers have to work out what the best design is, knowing the guidelines, for any given situation. I think this is a good guideline...

    Cheers
  26. Article: One Assertion Per Test[ Go to top ]

    This raises the question again: is it possible to have too much testing code? More specifically - is it possible to have too much unit testing code at this level? I get a distinct sense on this thread (and others :-) that there's a general perception that there's "no such thing as too many unit tests", and I find that rather disturbing.

    Let me approach this with a modest example (for a change) - let's say you have 500 classes with an aggregate total of 2,500 methods. Do you really think having 2,500 individual unit tests (each with one responsibility assert apiece) is the best expenditure of developer time?

    More telling - how much is it costing a company for developers to be saying "I spent 25% of my time asserting that my collections have X elements" :-?

    [The rest isn't directed at any particular person, but should be read as a generic rant :/).

    Further - what is the true worth of these tests to the project as a whole? Forget TDD for a moment - that's initial development. Go down the road 12 months after the project is in production - what is the worth of these 2,500 super-fine-grained tests? What is the cost of maintaining them? Will they honestly be maintained?

    I am not against unit tests, per se - but I see an enormous potential for abuse and excessiveness in them. Most troubling is that unit tests get the spotlight in so many articles and forums, just like this one. And you see damn few articles on performance testing, integration testing, multi-threaded testing. In case the industry was asleep and missed it, _this_ is where the bugs are that haunt developers. You don't see an all-hands emergency meeting because Joe in cube 4.105D missed an assignment and got an NPE. Usually, the problems that take up the most time and effort and angst are just a wee bit more complicated than that.

    A mild observation/question: do developers obsess over Unit Tests because they give cheap and easy satisfaction? It sounds trite until you see it, but I have seen quite a number of people crowing about passing all the unit tests, while the system as a whole is crumbling around them. I've also seen one too many people blithely saying "Oh, of course we don't _just_ unit test, we do those other things as well". In reality, the "_just_" is quite telling. Just as often, the same people can give you a 19-page treatise on their unit testing strategy, and their explanation of functional/stress/performance/integration testing is summed up as "yeah, we do that too". In fact that is quite common on TSS - how often do you see long posts on JUnit nuances? Now, compare and contrast, how often do you see long posts on stress testing, failure/recovery testing, intricate threading and deadlock detection tests? From where I sit, so many people are ignorant and/or terrified of real testing that they find it a relief to be able to retreat into something warm and comfy and easy like unit testing.

        -Mike
  27. Article: One Assertion Per Test[ Go to top ]

    This raises the question again: is it possible to have too much testing code? More specifically - is it possible to have too much unit testing code at this level? ...


    Mike,

    I used to think the way you do about the trade-off between testing everything and having thousands of unit tests to maintain. But then I started TDD... TDD caused a shift not only in the way I write tests and code but also in the way I value unit tests. If you look at unit tests as not merely validation and verification but also the design for your system and the best documentation of your codebase there is, unit tests become much more valuable. If you also consider unit tests to be part of your code-base there is no concept of "maintaining" them - any code that gets refactored directly affects the tests.

    Of course you have to be pragmatic about writing your unit tests - i.e. are individual tests needed for simple getters and setters? Probably not.

    I agree with you on one point - unit tests are only one form of testing. They should absolutely be accompanied by a good set of integration, functional, performance and acceptance tests, etc. I think the reason that unit tests are talked about so much is there close correllation with the code and by people who code test-first where the unit tests become their design.

    Cheers
  28. Article: One Assertion Per Test[ Go to top ]

    Permit me to slightly elaborate on Dave Astel's principle (or what I believe

    > his intention to be): Assert on one object / action per Test Method. What I
    > understood Dave to be promoting is testing one responsibility per method
    > (which will often, but not always, be one assertion).

    Very good. That is exactly the point.

    The goal was to make you think about what happens or what's possible when you have one assertion per test.

    > I don't think Dave was asserting (no pun intended) that all test methods
    > should have one assertion only as a hard-and-fast rule, more that each test
    > method should have one responsibility - one assertion per test guarantees
    > this, if Dave's process is followed.

    Yes.

    > The underlying concept of simple, decomposed tests is not new. Of course,
    > this should automatically occur when practicing TDD; testing after-the-fact
    > requires more thought to obtain the same sort of cleanliness.

    Yes. I'm coming from a TDD point of view as that's how I work (and what I preach and what I teach).

    > There are no hard-and-fast rules, only guidelines.

    Anyone here see "Pirates of the Carribean"? ;-)

    > We as developers have to work out what the best design is, knowing the
    > guidelines, for any given situation. I think this is a good guideline...

    Thanks. Working test-first, with fine grained tests is a handy way to "work out what the best design is", IMHO

    Dave
  29. One test scenario per test method[ Go to top ]

    This is the opposite extreme of one test method per tested method where you may have several asserts per test method (each assert testing a specific test scenario).

    I favor something in the middle: One test method per test scenario.

    For example, in Robert Martin's "Bowling Game" example, you would test a score() method by writing tests like testGutterGame(), testOneStrikeGame(), or testRandomRollsGame().

    If that ends up meaning that a test method has a single assert() then that's fine, but you shouldn't limit yourself to a single assert(), nor should you limit yourself to one monster test method per tested method.
  30. One test scenario per test method[ Go to top ]

    Forgot to mention...for those unfamiliar with Robert Martin's bowling game, here's the link: http://www.objectmentor.com/resources/articles/xpepisode.htm
  31. Testing a Scenario is a Good Approach[ Go to top ]

    I favor something in the middle: One test method per test scenario.


    This is my approach to unit testing.