Dave Astels writes about "what it's like to make tests as simple and decomposed as possible: aiming for a single assertion in each test." He takes an example that traditionally uses multiple assert*()'s within a test, and refactors it to give multiple tests with one assert each.
Excerpt
"I'm convinced writing tests like this is a useful approach. One advantage is that the resulting tests simpler and easier to understand. Just as important, and maybe more so, is that by adding the specification of the behavior one tiny piece at a time, you drive toward evolving the code in small, controllable, understandable steps."
Do you agree with the author?
An interesting aside, is that he gives examples in Squeak Smalltalk as well as Java!
Read One Assertion Per Test
-
Article: One Assertion Per Test (30 messages)
- Posted by: Dion Almaer
- Posted on: February 23 2004 21:16 EST
Threaded Messages (30)
- Article: One Assertion Per Test by Cedric Beust on February 24 2004 10:38 EST
- Article: One Assertion Per Test by Aslak Hellesøy on February 24 2004 11:05 EST
-
Article: One Assertion Per Test by Cedric Beust on February 24 2004 11:20 EST
- Article: One Assertion Per Test by Aslak Hellesøy on February 24 2004 12:09 EST
-
Article: One Assertion Per Test by Dave Astels on February 24 2004 02:30 EST
- Article: One Assertion Per Test by Cedric Beust on February 24 2004 03:16 EST
-
Article: One Line of Code Per Method by Cameron Purdy on February 24 2004 05:18 EST
-
Article: One Line of Code Per Method by Yagiz Erkan on February 25 2004 12:38 EST
- Article: One Line of Code Per Method by Jason McKerr on February 25 2004 03:11 EST
-
Article: One Line of Code Per Method by Yagiz Erkan on February 25 2004 12:38 EST
- Article: One Assertion Per Test by Thomas Mattson on February 25 2004 07:44 EST
-
Article: One Assertion Per Test by Cedric Beust on February 24 2004 11:20 EST
- Article: One Assertion Per Test by Ryan Ashcraft on February 24 2004 11:27 EST
-
Article: One Assertion Per Test by Cedric Beust on February 24 2004 11:33 EST
-
Article: One Assertion Per Test by Ryan Ashcraft on February 24 2004 12:26 EST
-
Article: One Assertion Per Test by Mike Spille on February 24 2004 12:35 EST
-
Article: One Assertion Per Test by Ryan Ashcraft on February 24 2004 02:17 EST
-
Article: One Assertion Per Test by Mike Spille on February 24 2004 02:53 EST
- As a followup... by Mike Spille on February 24 2004 03:19 EST
-
Article: One Assertion Per Test by Mike Spille on February 24 2004 02:53 EST
-
Article: One Assertion Per Test by Ryan Ashcraft on February 24 2004 02:17 EST
-
Article: One Assertion Per Test by Mike Spille on February 24 2004 12:35 EST
-
Article: One Assertion Per Test by Ryan Ashcraft on February 24 2004 12:26 EST
-
Article: One Assertion Per Test by Mike Spille on February 24 2004 11:37 EST
-
Article: One Assertion Per Test by Ryan Ashcraft on February 24 2004 11:55 EST
-
Article: One Assertion Per Test by Cedric Beust on February 24 2004 12:13 EST
- Article: One Assertion Per Test by Aslak Hellesøy on February 24 2004 12:24 EST
- Article: One Assertion Per Test by Mike Spille on February 24 2004 12:33 EST
-
Article: One Assertion Per Test by Cedric Beust on February 24 2004 12:13 EST
-
Article: One Assertion Per Test by Ryan Ashcraft on February 24 2004 11:55 EST
-
Article: One Assertion Per Test by Cedric Beust on February 24 2004 11:33 EST
- Article: One Assertion Per Test by Jason King on February 25 2004 08:24 EST
-
Article: One Assertion Per Test by Pamela Jones on February 25 2004 07:33 EST
-
Article: One Assertion Per Test by Mike Spille on February 25 2004 08:26 EST
- Article: One Assertion Per Test by Pamela Jones on February 26 2004 06:12 EST
- Article: One Assertion Per Test by Dave Astels on February 26 2004 12:54 EST
-
Article: One Assertion Per Test by Mike Spille on February 25 2004 08:26 EST
-
Article: One Assertion Per Test by Pamela Jones on February 25 2004 07:33 EST
- Article: One Assertion Per Test by Aslak Hellesøy on February 24 2004 11:05 EST
- One test scenario per test method by Craig Walls on February 24 2004 11:03 EST
- One test scenario per test method by Craig Walls on February 24 2004 11:09 EST
- Testing a Scenario is a Good Approach by T Wilson on February 25 2004 09:15 EST
-
Article: One Assertion Per Test[ Go to top ]
- Posted by: Cedric Beust
- Posted on: February 24 2004 10:38 EST
- in response to Dion Almaer
I wonder if this person has ever written real code out there.
One test assertion per test... right. What's next... only one method call per method? One only one variable?
Suites, Classes and Methods are three important pieces of decomposition of your unit testing, and they are all important. Use them wisely, but use them all.
A quick look at my own test shows an average of five-ten asserts per method, but it's sometimes much more and sometimes much less. Whatever makes sense.
<sigh>
--
Cedric -
Article: One Assertion Per Test[ Go to top ]
- Posted by: Aslak Hellesøy
- Posted on: February 24 2004 11:05 EST
- in response to Cedric Beust
Cedric: I wonder if you have ever written some real tests out there ;-)
I'm not surprised that you dismiss Dave Astels' ideas and his credibility in the field. I think you missed the point of the whole article:
It is quite common to have unreadable tests. This is usually because they are not focused enough. They test a little bit of this and a little bit of that and are not coherent.
A bad side effect of this is duplicate tests. Many test methods end up testing the same thing. Your tests become hard to maintain.
With Astels' one assertion per test you don't get duplication in tests. And it makes you even more focused on your current task when you do TDD.
I think one assertion per test is a great goal. You don't have to follow it religiously. Just strive for it.
As for your out-of-the-air comparison with number of variables and method calls per method, here is my rule of thumb:
7 lines of code per method. -
Article: One Assertion Per Test[ Go to top ]
- Posted by: Cedric Beust
- Posted on: February 24 2004 11:20 EST
- in response to Aslak Hellesøy
Aslak:
With Astels' one assertion per test you don't get duplication in tests.
How does "one assert per test method" guarantee you don't have duplication?
For example, imagine I have about five asserts per method. I refactor this and I now have five methods. Multiply this by the number of tests I have and suddenly I now have hundreds of test methods.
Do I have a better grasp on the design of my tests? Nope.
You don't make design happen by applying automatic rules like this one.
Testing is no different than standard development: only judgment and design skills will help you make your code better architected, not simple and baseless syntactic rules like the one suggested by Astel.
--
Cedric -
Article: One Assertion Per Test[ Go to top ]
- Posted by: Aslak Hellesøy
- Posted on: February 24 2004 12:09 EST
- in response to Cedric Beust
Aslak:
> With Astels' one assertion per test you don't get duplication in tests.
>
> How does "one assert per test method" guarantee you don't have duplication?
>
I didn't mean to say that it guarantees non duplication. I meant to say it reduces the likelyhood of it happening.
Here are some simple guidelines to help you reduce the risk of dupes:
(Recap of the TDD rythm):
1) Write a test
2) Make the code compile (the simplest possible way)
3) Make the test pass (the simplest possible way)
4) Refactor
(start again on 1)
If you follow this, and accidentally write a passing test, you will get a green bar on 2. This is an indication that your recent test was a dupe and maybe should be deleted.
> For example, imagine I have about five asserts per method. I refactor this and I now have five methods. Multiply this by the number of tests I have and suddenly I now have hundreds of test methods.
>
> Do I have a better grasp on the design of my tests? Nope.
>
> You don't make design happen by applying automatic rules like this one.
>
> Testing is no different than standard development: only judgment and design skills will help you make your code better architected, not simple and baseless syntactic rules like the one suggested by Astel.
>
Experience is important, but don't neglect the importance of sound guidelines and best practices. Sometime guidelines have to be a bit extreme in order to make people think differently.
(It's Astels, not Astel).
Aslak
> --
> Cedric -
Article: One Assertion Per Test[ Go to top ]
- Posted by: Dave Astels
- Posted on: February 24 2004 14:30 EST
- in response to Aslak Hellesøy
Cedric: I wonder if you have ever written some real tests out there ;-)
>
> I'm not surprised that you dismiss Dave Astels' ideas and his credibility in the field.
Yes, I've written real tests. I've been playing this game for close to 25 years.
> With Astels' one assertion per test you don't get duplication in tests. And it
> makes you even more focused on your current task when you do TDD.
Yes, very much so.
> I think one assertion per test is a great goal. You don't have to follow it
> religiously. Just strive for it.
Exactly.
> As for your out-of-the-air comparison with number of variables and method
> calls per method, here is my rule of thumb:
>
> 7 lines of code per method.
I start getting edgy at 5 :)
Dave -
Article: One Assertion Per Test[ Go to top ]
- Posted by: Cedric Beust
- Posted on: February 24 2004 15:16 EST
- in response to Dave Astels
Dave:
Yes, I've written real tests. I've been playing this game for close to 25 years.
Okay great, so how about commenting on my post instead of Aslak's?
I am not the only one having a few issues with your post, so why don't you take this chance to clarify your position?
--
Cedric -
Article: One Line of Code Per Method[ Go to top ]
- Posted by: Cameron Purdy
- Posted on: February 24 2004 17:18 EST
- in response to Aslak Hellesøy
Aslak: As for your out-of-the-air comparison with number of variables and method calls per method, here is my rule of thumb: 7 lines of code per method.
Hmm .. have you ever written any code in the real world? Who is this "Aslak Hellesoy" guy anyway?
;-)
Peace,
Cameron Purdy
Tangosol, Inc.
Coherence: Clustered JCache for Grid Computing! -
Article: One Line of Code Per Method[ Go to top ]
- Posted by: Yagiz Erkan
- Posted on: February 25 2004 12:38 EST
- in response to Cameron Purdy
Aslak: As for your out-of-the-air comparison with number of variables and method calls per method, here is my rule of thumb: 7 lines of code per method.
:-) 7 lines of code/method?!? 5 lines of code/method?!? I don't know how realistic this is but this makes me smile. Or... maybe... you use multi-statement lines :-)
Have you ever caught exceptions when you write code? I'm asking because I tried but I couldn't find a way to write this in less than 7 lines:
try {
goSomething();
} catch(SomeException e) {
doSomethingElse();
} finally {
doSomethingFinally();
}//try-catch-finally
If I can find some spare time, I'll try to write 1-line-unit-tests testing 5-line-methods... -
Article: One Line of Code Per Method[ Go to top ]
- Posted by: Jason McKerr
- Posted on: February 25 2004 15:11 EST
- in response to Yagiz Erkan
maybe he writes it as:
try {goSomething();} catch(SomeException e) { doSomethingElse(); } finally { doSomethingFinally();}//try-catch-finally
Suweet.
Jason McKerr
The Open Source Lab
"Open Minds. Open Doors. Open Source." -
Article: One Assertion Per Test[ Go to top ]
- Posted by: Thomas Mattson
- Posted on: February 25 2004 07:44 EST
- in response to Aslak Hellesøy
A bad side effect of this is duplicate tests. Many test methods end up testing
> the same thing. Your tests become hard to maintain.
Hmm, I consider redundancy in my unit tests to be perfectly acceptable and even a good thing. -
Article: One Assertion Per Test[ Go to top ]
- Posted by: Ryan Ashcraft
- Posted on: February 24 2004 11:27 EST
- in response to Cedric Beust
Using more than one assert per test quickly transforms "unit tests" into "system tests". When I run my test suites, I want to be able to look at the method that failed and instantly know the gist of the issue. Without restricting each unit test to a single assert, this isn't possible.
This is a very similar to a new habit I've formed - avoiding complex lines of code. There is nothing worse than getting a NullPointerException on a line like:
String s = new String(foo.getBar() + bar.getBlah() + crap.whichOneFailed());
A NPE here means I have to open up the debugger. If I just restructure the line above to be:
String bar = foo.getBar();
String blah = bar.getBlah();
String knowExactlyWhichOneFailed = crap.whichOneFailed();
String s = bar + blah + knowExactlyWhichOneFailed;
I can more easily diagnose the problem. Granted, I might still have to turn my debugger on to find out why an object I expect to be around is null, but at least I know where to look the first time through.
I suspect that if you try the single assert per test, you'll find your life and the lives of those that work w/ you are much easier.
KISS strikes again.
- Ryan -
Article: One Assertion Per Test[ Go to top ]
- Posted by: Cedric Beust
- Posted on: February 24 2004 11:33 EST
- in response to Ryan Ashcraft
Ryan:
Using more than one assert per test quickly transforms "unit tests" into "system tests". When I run my test suites, I want to be able to look at the method that failed and instantly know the gist of the issue. Without restricting each unit test to a single assert, this isn't possible.
Notice that you subtly changed the point of the original article.
Astel thinks his design rule makes your test design better. You are saying that it makes the output easier to diagnose.
These are very different things, but once again, I argue that good diagnostic has nothing to do with the number of asserts per method. Just take the time to write a meaningful failure message when you write your assert:
Don't use:
assertEquals(c.size(), 1);
assertTrue("Cedric".equals(((Employee) c.iterator().next()).geFirstName());
use:
assertEquals("Expected a different number of employees",
c.size(), 1);
assertTrue("The employee should be named Cedric",
"Cedric".equals(((Employee) c.iterator().next()).geFirstName());
Notice that in the first case, splitting the two asserts in two methods will certainly not help your understanding of the failure, except maybe with the vague hint of the test method name. But seriously, what do you prefer to see, "testFirtName() method failed" or "The employee should be named Cedric"?
This is a very similar to a new habit I've formed - avoiding complex lines of code. There is nothing worse than getting a NullPointerException on a line like:
For your information, this is typically referred to as the Demeter Principle.
--
Cedric -
Article: One Assertion Per Test[ Go to top ]
- Posted by: Ryan Ashcraft
- Posted on: February 24 2004 12:26 EST
- in response to Cedric Beust
Cedric:
I don't have a problem with meaningful failure messages. My issue is with several different test cases being written within one method. You example is perfectly fine in my view. I run into issues where people combine 12 different test cases into a single method and then do not realize the extent of a problem becuase they see one failure. This is very poor test design, but is very common.
I do think you have an excellent point via your example. The goal should not be the number of asserts as a hard and fast rule but in the scoping of the test case.
- Ryan -
Article: One Assertion Per Test[ Go to top ]
- Posted by: Mike Spille
- Posted on: February 24 2004 12:35 EST
- in response to Ryan Ashcraft
Ryan, I think you're missing a point here: the vast majority of the time, your tests will pass (or else you have much bigger problems). Testing at too fine a granularity means you're wasting time that could be better spent elsewhere.
Put another way - most of the time, a test won't fail (even an "aggregate" tests). Most of the time, when a test fails, it's obvious why. In the few cases where it's not obvious, well then you've got some work to do :-)
As with everything else, you want to target the common cases, not the corner cases. It sounds to me like you're working extra-hard to accomodate a corner case, and as a result you're working extra-hard when you generally don't need to.
-Mike -
Article: One Assertion Per Test[ Go to top ]
- Posted by: Ryan Ashcraft
- Posted on: February 24 2004 14:17 EST
- in response to Mike Spille
Mike:
You and I have different ideas about what type of testing is required! It is time to agree to disagree.
But for the record, as someone that develops applications that have to run on both Websphere and Weblogic, it would make my life much easier if the "corner cases" of the J2EE specs were tested consistently. It is nearly impossible to really use J2EE w/o application server-specific logic - probably in a large part to a "target the common cases, not the corner cases" attitude.
- Ryan -
Article: One Assertion Per Test[ Go to top ]
- Posted by: Mike Spille
- Posted on: February 24 2004 14:53 EST
- in response to Ryan Ashcraft
\Ryan Ashcraft\
You and I have different ideas about what type of testing is required! It is time to agree to disagree.
\Ryan Ashcraft\
OK. But, as I said, I don't understand the value in artificial testing boundaries and nomenclature.
\Ryan Ashcraft\
But for the record, as someone that develops applications that have to run on both Websphere and Weblogic, it would make my life much easier if the "corner cases" of the J2EE specs were tested consistently. It is nearly impossible to really use J2EE w/o application server-specific logic - probably in a large part to a "target the common cases, not the corner cases" attitude.
\Ryan Ashcraft\
Most of the problems I've seen in application servers would not be caught by the sort of unit tests you're talking about. They usually involve complex lifecycle issues and many objects interacting - specifically the type of thing that fine-grained unit tests do _not_ catch.
-Mike -
As a followup...[ Go to top ]
- Posted by: Mike Spille
- Posted on: February 24 2004 15:19 EST
- in response to Mike Spille
As a followup to show what I'm talking about. Consider a system with 200,000 SLOC's (this is the realm I work in) which is maintained over the course of several years. Using (admittedly naive) back of the napkin calculations, with your indicator of an average of 7 lines of code/method, you now have 28,000 methods (!).
Each of those 28,000 methods would seem to have a corresponding unit tests - so we have 28,000 unit tests. This is _only_ unit tests - we haven't done any larger scale tests at all.
Imagine now _just_ maintaining the unit test code base - how many man-years of effort are you talking about just for unit tests? When major changes hit the code base (as they always do), how much effort is there _just_ to update the unit tests? Again - this doesn't cover any other tests, just unit tests.
That's where I'm coming from, and to an extent where I believe Cedric is coming from as well. If you reach for the ultimate small granularity testing, you'll find that as your code base hits 100,000 lines and more that you're literally swimming in unit test code, and that it becomes a burden more than a boon.
This is why I talk about abstraction and aggregation at the test level as well as the code level. Past a certain threshold really fine grained tests will overwhelm any development team.
-Mike -
Article: One Assertion Per Test[ Go to top ]
- Posted by: Mike Spille
- Posted on: February 24 2004 11:37 EST
- in response to Ryan Ashcraft
Ryan, what you are describing isn't simplifying the system, it's trading off where your complexity lies. Effectively, you now have smaller, more granular "building blocks", which makes the building blocks individually easier to understand.
BUT - now you've got more building blocks! You've hoisted the complexity up one level of abstraction. This makes it easier to unit test and understand the individual pieces, and makes it harder to understand the inter-relationships.
In my own development philosophy, the point of coding is not to make unit tests easier. The point is to make integration tests easier, make it easier to understand the code, and to ensure _overall_ correctness.
Unit tests are so low-level that they are a minor consideration. To me they're low-level enough that you shouldn't obsess over them too much.
-Mike -
Article: One Assertion Per Test[ Go to top ]
- Posted by: Ryan Ashcraft
- Posted on: February 24 2004 11:55 EST
- in response to Mike Spille
Mike:
I never said that I changed my design to make my unit testing easier. Good design should always come first. My point is simply that a "unit test" should test a "unit" - something that is singular. I don't buy into testing multiple units/cases at once because then my test is effectively a system test. System tests are fine but serve a different purpose because they look at the system in a more complete form than unit tests. The way that unit tests help developers is that they cut down on the scope of the problem compared to system tests.
I don't suspect that you or Cedric are challenging this point. I gather that the issue here is whether or not one believes in system testing or unit testing. Both have their pluses and minuses. In my judgement, both should be used.
- Ryan -
Article: One Assertion Per Test[ Go to top ]
- Posted by: Cedric Beust
- Posted on: February 24 2004 12:13 EST
- in response to Ryan Ashcraft
Ryan:
I don't suspect that you or Cedric are challenging this point. I gather that the issue here is whether or not one believes in system testing or unit testing. Both have their pluses and minuses. In my judgement, both should be used.
True. I guess it just boils down to our respective definition of a unit.
Typically, if I want to test that an object has the right values, I can't really understand what I gain by changing:
testPerson() {
assertTrue("Cedric".equals(person.getFirstName());
assertTrue("Beust".equals(person.getLastName());
}
into two different methods "testFirstName()" and "testLastName()" (which also implies the "person" object will be calculated in a third method probably inherited by the test... seems overly complex to me.
I totally agree with Mike: unit tests should stay structurally simple, no need to obsess over their design as long as you respect the DRY principle.
--
Cedric -
Article: One Assertion Per Test[ Go to top ]
- Posted by: Aslak Hellesøy
- Posted on: February 24 2004 12:24 EST
- in response to Cedric Beust
> testPerson() {
> assertTrue("Cedric".equals(person.getFirstName());
> assertTrue("Beust".equals(person.getLastName());
> }
>
JUnit has an assertEquals method that will give you a more meaningful message in case of a failure:
testPerson() {
assertEquals("Cedric", person.getFirstName());
assertEquals("Beust", person.getLastName());
} -
Article: One Assertion Per Test[ Go to top ]
- Posted by: Mike Spille
- Posted on: February 24 2004 12:33 EST
- in response to Cedric Beust
Total agreement here, Cedric. I prefer to aggregate both functionality and tests into logical units that work in "chunks" that don't necessarily match up to Java language constructs. Java gives us methods, instance variables, classes, and packages. This does not mean that testing or coding should be _logically_ limited to those constructs. Often tests make the most sense when they cross boundaries - the logical "unit" is far more important than the code "unit". So some of what I would call "unit" tests will use multiple methods or (gasp!) multiple objects, and will even scale up and cross package/sub-system boundaries. Alot of people seem to want rigid boundaries between "unit" tests, "system" tests, integration tests, etc - to which I say "Bah!". Such boundaries are both artificial and, often, arbitrary.
You should code and test at the abstraction level that makes sense to your problem domain and for the code in hand. If you're forcing yourself to call a "unit" the smallest expressable Java coding construct, then likely you're being too granular and not scaling your development effort as well as you could. My rules of thumb:
- if your testing methods have one-to-one correspondence to real code methods, or:
- if your number of test classes has a one-to-one correspondence to real classes, or:
- if the overall size of your test harness equals the size of your real code base
then:
- you're too granular, you're spinning your wheels far too much in test code, and not as agile as you could be.
-Mike -
Article: One Assertion Per Test[ Go to top ]
- Posted by: Jason King
- Posted on: February 25 2004 08:24 EST
- in response to Cedric Beust
While refactoring having high test case coverage aids me greatly in detecting when things go wrong.
The number of tests that do occasionally break gives me a good "feel" for the difficulty/complexity of the refactoring at hand, and shows me quickly the areas in the codebase that are impacted.
Thus, if I have a method with more than 1 assert, any number may be broken, and I only get 1 failure (as the test method exits). For this reason alone, I prefer one assert per test method. It maximises the visibility of the impact of my change (in that a larger number of tests break).
The error messages are also a very good indicator of what is going wrong. This helps me re-plan my refactoring or design -> The tests are driving my design....
Cheers -
Article: One Assertion Per Test[ Go to top ]
- Posted by: Pamela Jones
- Posted on: February 25 2004 19:33 EST
- in response to Jason King
Permit me to slightly elaborate on Dave Astel's principle (or what I believe his intention to be): Assert on one object / action per Test Method. What I understood Dave to be promoting is testing one responsibility per method (which will often, but not always, be one assertion).
public void testMethodThatReturnsCollectionPopulated() throws Exception {
Collection populatedCollection = someObjectAlreadySetup.getSomeCollection();
//There are three asserts here but testing only one responsibility
assertNotNull("Expected populated collection", populatedCollection);
assertEquals("Expection collection to have 5 elements", 5, populatedCollection.size());
assertTrue("Expected blah in the collection", populatedCollection.iterator().next() instanceof AClass);
}
public void testMethodThatReturnsCollectionEmpty() ...
public void testMethodThatReturnsCollectionNull() ...
public void testMethodThatReturnsSomeObjectPopulated() throws Exception {
SomeObject anObject = someObjectAlreadySetup.getSomeObject();
//Two asserts here on a different responsbility of the same object that was being tested above
assertNotNull("Expected not null object", anObject);
assertEquals("Expected something else", objectExpected, anObject);
}
...
In this example, someObjectAlreadySetup has a number of methods. Each of these methods are tested in a different test method, rather than asserting everything in one test method; however, a number of assertions are made about each action / responsibility. I think the null check is important in some scenarios so as to prevent NPEs in test methods as this does not pinpoint the failure succinctly - though this should mostly be avoided if adhering to the Law of Demeter. (Yes I realise it's a very thorough unit test; yes I realise that Generics in JDK1.5 will remove the need to do instanceof on Collection elements; it's just an example)
I don't think Dave was asserting (no pun intended) that all test methods should have one assertion only as a hard-and-fast rule, more that each test method should have one responsibility - one assertion per test guarantees this, if Dave's process is follwed. The underlying concept of simple, decomposed tests is not new. Of course, this should automatically occur when practicing TDD; testing after-the-fact requires more thought to obtain the same sort of cleanliness.
Coding is not a science, if it were there would be a formula for success. There are no hard-and-fast rules, only guidelines. We as developers have to work out what the best design is, knowing the guidelines, for any given situation. I think this is a good guideline...
Cheers -
Article: One Assertion Per Test[ Go to top ]
- Posted by: Mike Spille
- Posted on: February 25 2004 20:26 EST
- in response to Pamela Jones
This raises the question again: is it possible to have too much testing code? More specifically - is it possible to have too much unit testing code at this level? I get a distinct sense on this thread (and others :-) that there's a general perception that there's "no such thing as too many unit tests", and I find that rather disturbing.
Let me approach this with a modest example (for a change) - let's say you have 500 classes with an aggregate total of 2,500 methods. Do you really think having 2,500 individual unit tests (each with one responsibility assert apiece) is the best expenditure of developer time?
More telling - how much is it costing a company for developers to be saying "I spent 25% of my time asserting that my collections have X elements" :-?
[The rest isn't directed at any particular person, but should be read as a generic rant :/).
Further - what is the true worth of these tests to the project as a whole? Forget TDD for a moment - that's initial development. Go down the road 12 months after the project is in production - what is the worth of these 2,500 super-fine-grained tests? What is the cost of maintaining them? Will they honestly be maintained?
I am not against unit tests, per se - but I see an enormous potential for abuse and excessiveness in them. Most troubling is that unit tests get the spotlight in so many articles and forums, just like this one. And you see damn few articles on performance testing, integration testing, multi-threaded testing. In case the industry was asleep and missed it, _this_ is where the bugs are that haunt developers. You don't see an all-hands emergency meeting because Joe in cube 4.105D missed an assignment and got an NPE. Usually, the problems that take up the most time and effort and angst are just a wee bit more complicated than that.
A mild observation/question: do developers obsess over Unit Tests because they give cheap and easy satisfaction? It sounds trite until you see it, but I have seen quite a number of people crowing about passing all the unit tests, while the system as a whole is crumbling around them. I've also seen one too many people blithely saying "Oh, of course we don't _just_ unit test, we do those other things as well". In reality, the "_just_" is quite telling. Just as often, the same people can give you a 19-page treatise on their unit testing strategy, and their explanation of functional/stress/performance/integration testing is summed up as "yeah, we do that too". In fact that is quite common on TSS - how often do you see long posts on JUnit nuances? Now, compare and contrast, how often do you see long posts on stress testing, failure/recovery testing, intricate threading and deadlock detection tests? From where I sit, so many people are ignorant and/or terrified of real testing that they find it a relief to be able to retreat into something warm and comfy and easy like unit testing.
-Mike -
Article: One Assertion Per Test[ Go to top ]
- Posted by: Pamela Jones
- Posted on: February 26 2004 18:12 EST
- in response to Mike Spille
This raises the question again: is it possible to have too much testing code? More specifically - is it possible to have too much unit testing code at this level? ...
Mike,
I used to think the way you do about the trade-off between testing everything and having thousands of unit tests to maintain. But then I started TDD... TDD caused a shift not only in the way I write tests and code but also in the way I value unit tests. If you look at unit tests as not merely validation and verification but also the design for your system and the best documentation of your codebase there is, unit tests become much more valuable. If you also consider unit tests to be part of your code-base there is no concept of "maintaining" them - any code that gets refactored directly affects the tests.
Of course you have to be pragmatic about writing your unit tests - i.e. are individual tests needed for simple getters and setters? Probably not.
I agree with you on one point - unit tests are only one form of testing. They should absolutely be accompanied by a good set of integration, functional, performance and acceptance tests, etc. I think the reason that unit tests are talked about so much is there close correllation with the code and by people who code test-first where the unit tests become their design.
Cheers -
Article: One Assertion Per Test[ Go to top ]
- Posted by: Dave Astels
- Posted on: February 26 2004 12:54 EST
- in response to Pamela Jones
Permit me to slightly elaborate on Dave Astel's principle (or what I believe
> his intention to be): Assert on one object / action per Test Method. What I
> understood Dave to be promoting is testing one responsibility per method
> (which will often, but not always, be one assertion).
Very good. That is exactly the point.
The goal was to make you think about what happens or what's possible when you have one assertion per test.
> I don't think Dave was asserting (no pun intended) that all test methods
> should have one assertion only as a hard-and-fast rule, more that each test
> method should have one responsibility - one assertion per test guarantees
> this, if Dave's process is followed.
Yes.
> The underlying concept of simple, decomposed tests is not new. Of course,
> this should automatically occur when practicing TDD; testing after-the-fact
> requires more thought to obtain the same sort of cleanliness.
Yes. I'm coming from a TDD point of view as that's how I work (and what I preach and what I teach).
> There are no hard-and-fast rules, only guidelines.
Anyone here see "Pirates of the Carribean"? ;-)
> We as developers have to work out what the best design is, knowing the
> guidelines, for any given situation. I think this is a good guideline...
Thanks. Working test-first, with fine grained tests is a handy way to "work out what the best design is", IMHO
Dave -
One test scenario per test method[ Go to top ]
- Posted by: Craig Walls
- Posted on: February 24 2004 11:03 EST
- in response to Dion Almaer
This is the opposite extreme of one test method per tested method where you may have several asserts per test method (each assert testing a specific test scenario).
I favor something in the middle: One test method per test scenario.
For example, in Robert Martin's "Bowling Game" example, you would test a score() method by writing tests like testGutterGame(), testOneStrikeGame(), or testRandomRollsGame().
If that ends up meaning that a test method has a single assert() then that's fine, but you shouldn't limit yourself to a single assert(), nor should you limit yourself to one monster test method per tested method. -
One test scenario per test method[ Go to top ]
- Posted by: Craig Walls
- Posted on: February 24 2004 11:09 EST
- in response to Craig Walls
Forgot to mention...for those unfamiliar with Robert Martin's bowling game, here's the link: http://www.objectmentor.com/resources/articles/xpepisode.htm -
Testing a Scenario is a Good Approach[ Go to top ]
- Posted by: T Wilson
- Posted on: February 25 2004 09:15 EST
- in response to Craig Walls
I favor something in the middle: One test method per test scenario.
This is my approach to unit testing.