Discussions

News: TestNG crew takes a sneak peek at JUnit 4

  1. It was recently leaked that early code for JUnit4 is avaialable in a special branch of CVS. TestNG co-founders Cedric Beust and Alexandru Popescu took a look the code and have revealed at what's coming up in JUnit4 and how it compares to TestNG. JUnit4 has been refactored with annotations support and many TestNG-like improvements.

    Alexandru did a comparison of JUnit 4 with features already in TestNG, and Cedric itemized the new features in JUnit 4 he and Alexandru noticed:
    * Annotations. As we observed with TestNG, annotations let you do away with inheritance of a base class and make more sense than trying to cram test information into method names.
           
        * The @Test annotation has an expected attribute (probably similar to @ExpectedExceptions, but it's hard to tell at this point)=. I also note they only allow one class to be expected while TestNG lets you specify more than one).
           
        * There is a new @Ignore annotations, which we decided to specify as an attribute of @Test in TestNG (e.g. @Test(enabled = false)).
           
        * TestNG's timeout and invocationCount also seem to be planned for JUnit.
           
        * Finer grained configuration access. JUnit only allowed to setUp/tearDown around test methods, you can now do this at the suite level (TestNG offers even greater flexibility: around classes and around groups).
           
        * Suites are being deprecated and they are looking for a way to describe collections of tests (quite a surprising decision, but may I suggest a file called... I don't know, junit.xml?).
           
        * There will be a converter (I invite the JUnit team to start off with com.beust.testng.JUnitConverter and tweak the imports it generates :-)).
           
        * Run failures first (TestNG generates a testng-failures.xml for this).
           
        * Quoted from the todo file: "be able to specify the order of tests (ewwww....)". Now that is very interesting... They are just falling short of specifying test dependencies but as much as I find this feature crucial for non-unit testing, I question its use for unit testing.
           
        * Groups! Yes, that's right. A lot of people criticized the need for groups when they were introduced in TestNG a year ago, but we have definitely seen users like this feature and use it a lot. It's good to see JUnit follow the same path.
    JUnit 4 will bring a significant improvement over previous versions, but still seems to be philosophically different thatn TestNG in a few ways.

    Threaded Messages (35)

  2. Reply by Erich Gamma[ Go to top ]

    In the comments section of Cedric's blog entry (second link, http://beust.com/weblog/archives/000292.html), there is a reply by Erich Gamma.

    I just wanted to point it out because it's interesting and easy to miss.
  3. JDK 1.5 only?[ Go to top ]

    Does it require 1.5 or will it also work with 1.3/1.4

    Billy
    IBM
    http://www.billynewport.com
  4. JDK 1.5 only?[ Go to top ]

    Does it require 1.5 or will it also work with 1.3/1.4BillyIBM
    It's 1.5 only right now, but you would have to ask Erich directly to find out if they plan to support 1.4 as well.

    --
    Cedric
  5. joining ?[ Go to top ]

    Is it time for TestNG to contribute their work and adapt it to JUnit ?
    Apart those very tech. details floating around whatever annotation syntax, and the 'we were there first' claims throwned in contrast of early JUnit 4 work (that sounds to scary TestNG team that keeps JUnit in its radar), what is the TestNG value proposal ? How does it integrates within my IDEs etc ?

    Alex
  6. IDE support[ Go to top ]

    How does it integrates within my IDEs etc ?Alex
    There is an Eclipse plug-in:

    http://beust.com/testng/main.html#eclipse

    --
    Cedric
    http://beust.com/testng
  7. joining ?[ Go to top ]

    Is it time for TestNG to contribute their work and adapt it to JUnit ?Apart those very tech. details floating around whatever annotation syntax, and the 'we were there first' claims throwned in contrast of early JUnit 4 work (that sounds to scary TestNG team that keeps JUnit in its radar), what is the TestNG value proposal ? How does it integrates within my IDEs etc ?Alex


    +1. What compelling reason is there to switch to TestNG given that most modern IDE's solely support JUnit?

    Anyways, I never understood why such a simple framework needed rewriting. Don't you NG guys have anything better to do?
  8. Simple Framework[ Go to top ]

    Hi Bill,

    I think the main problem with JUnit is that some parts of its design and implementation do not neccessarily lend themselves to extension development though this has not stopped many companies and individuals creating extensions. JInspired recently announced an Ant/JUnit tracing extension: http://www.jinspired.com/products/jdbinsight/servletconfig.html

    I hope the following statement indicates that this will be addressed in JUnit4.

    "we want to make it simpler for both the user and junit tool providers to support suites." - Erich Gamma

    It would be nice to have a better mechanism for extensions that does not exclude other extensions from being used during test execution. For example the decorator approach adopted by extensions has its limitations: http://www.clarkware.com/software/JUnitPerf.html#limitations

    I would also like to see the JUnit framework supporting a more formal service provider interface for aspects of the test execution.

    Regards,


    William Louth
    JXInsight Product Architect
    JInspired

    "J2EE tuning, testing and tracing with JXInsight"
    http://www.jinspired.com
  9. Wrong Link.[ Go to top ]

    http://www.jinspired.com/products/jdbinsight/antjunittracing.html

    Sorry.

    William Louth
    JXInsight Product Architect
    JInspired

    "J2EE tuning, testing and tracing with JXInsight"
    http://www.jinspired.com
  10. Other IDEs and Tools support[ Go to top ]

    But is not only the eclipse that counts.

    And maven ? TestNG have a plugin for maven ?

    and other IDEs, IDEA, etc ?
  11. Other IDEs and Tools support[ Go to top ]

    Yes there is a Maven plugin too. And yes, there are plans to provide a plugin for TestNG in Idea (but have you looked at the Idea OpenAPI? :-))
  12. joining ?[ Go to top ]

    Anyways, I never understood why such a simple framework needed rewriting. Don't you NG guys have anything better to do?
    No, I live a sad and lonely life and I cry myself to sleep at night with a bottle of scotch.

    I can't believe that of all people, you, who created JBossAOP, would criticize an initiative that is trying to improve on something that already exists...

    --
    Cedric
    http://beust.com/testng
  13. joining ?[ Go to top ]

    Is it time for TestNG to contribute their work and adapt it to JUnit ?Apart those very tech. details floating around whatever annotation syntax, and the 'we were there first' claims throwned in contrast of early JUnit 4 work (that sounds to scary TestNG team that keeps JUnit in its radar), what is the TestNG value proposal ? How does it integrates within my IDEs etc ?Alex
    +1. What compelling reason is there to switch to TestNG given that most modern IDE's solely support JUnit? Anyways, I never understood why such a simple framework needed rewriting. Don't you NG guys have anything better to do?

    I'm not one of the "NG guys" but I have an answer for you...JUnit sucks in it's current form. Everyone knows it but almost everybody is affraid to say it.

    As for IDE integration, yeah JUnit is integrated with everything. Unfortunate but true. However, I think that's only because a lot of people still not have accepted the degree to which JUnit sucks. Only then, will the plugins for TestNG start to flow.

    Finally, I have to say that I'm shocked with this new direction with JUnit. The JUnit peeps have staunchly defended their pos library. JUnit4 seems like an admission of how much of a pile of steaming crap their current libraries are. That said, I'm glad they are taking this new direction because for some odd reason people can't seem to move away from JUnit.
  14. joining ?[ Go to top ]

    Is it time for TestNG to contribute their work and adapt it to JUnit ?Apart those very tech. details floating around whatever annotation syntax, and the 'we were there first' claims throwned in contrast of early JUnit 4 work (that sounds to scary TestNG team that keeps JUnit in its radar), what is the TestNG value proposal ? How does it integrates within my IDEs etc ?Alex
    +1. What compelling reason is there to switch to TestNG given that most modern IDE's solely support JUnit? Anyways, I never understood why such a simple framework needed rewriting. Don't you NG guys have anything better to do?

    My compelling reason to switch was that TestNG allowed me to easily perform dynamic tests on different sets of input data. In order to accomplish the same task in JUnit, it required me to override several methods within the TestCase class and extend from the new hacked version. Very ugly.

    With TestNG it required me to simply move my test fixture builder into a @Factory method and pass the test classes into the @Factory using TestNG.xml. It isn't perfect, but I didn't have to override a single method and it produces a much cleaner report.

    Granted I think my case is a bit different than what most are doing. I'd be quite curious to see how JUnit 4 would handle my test senario, but I think that TestNG can do other kinds of accpeptance or functional testing a bit better than JUnit 3.x can for my scenario.
  15. joining ?[ Go to top ]

    I see it as a fairly simple situation. JUnit has sucked for a very, very long time. It's "design" (quotes very intentional) is based more on dogma than the real needs of developers, and it's awkward to use in the small and scales horribly to the large.

    TestNG has a very neat, clean design. More importantly, it's clearly been built by people who have worked on large systems and have experience in testing them. The abstractions are right, it's absurdly easy to use, and it scales from simple stuff up to very complex projects.

    JUnit showed a couple of good ideas with a fairly lousy implementation, and then just stalled out several years ago. TestNG has taken up the gauntlet and is now the testing engine to beat. IMHO Gamma and Beck should adopt TestNG whole hog and just let the JUnit code lie in state. The work for a _modern_ testing framework has already been done by Cedric et al. If Beck and Gamma are smart they'll build on that work rather than try to build on the poor JUnit codebase.
  16. Don't oversell this thing...[ Go to top ]

    I see it as a fairly simple situation. JUnit has sucked for a very, very long time.

    ...so badly that no one stuck their necks out to create a replacement for how many years?
    It's "design" (quotes very intentional) is based more on dogma than the real needs of developers, and it's awkward to use in the small and scales horribly to the large.

    I can't speak to whether those guys have some dogma to advance. I can say that no other testing framework I've used was as easy to use as JUnit. TestNG's not so bad to use but I loathe having to describe suites in XML files.

    (Was the apostrophe in "it's" intentional? 8^))
    TestNG has a very neat, clean design.

    Sure, if you love getters galore.
    More importantly, it's clearly been built by people who have worked on large systems and have experience in testing them. The abstractions are right, it's absurdly easy to use, and it scales from simple stuff up to very complex projects.

    Cool, I'm ready for the success stories from those very complex projects to pour in.
    JUnit showed a couple of good ideas with a fairly lousy implementation, and then just stalled out several years ago. TestNG has taken up the gauntlet and is now the testing engine to beat. IMHO Gamma and Beck should adopt TestNG whole hog and just let the JUnit code lie in state. The work for a _modern_ testing framework has already been done by Cedric et al. If Beck and Gamma are smart they'll build on that work rather than try to build on the poor JUnit codebase.

    JUnit showed some /great/ ideas. The code could stand to be improved. For that matter, so could TestNG's code.

    I'm not here to be a shill for JUnit, or a rainer on TestNG's parade. I fail to understand why Kent and Erich are getting dogpiled on all of a sudden.

    Ah well.
  17. Don't oversell this thing...[ Go to top ]

    Actually, I would claim strongly that many people stuck their necks out to shore up JUnit. Many of the tools written on top of JUnit also go a long way to fix fudnamental JUnit problems.

    I also think for a long time people looked to Beck et al to fix and improve JUnit - and due to the stature of the people waited much longer than is perhaps normal. But JUnit stagnated for a very long time, and Cedric was the one who finally got sick of waiting and did something about it.

    To date TestNG's adoption rate has been somewhat low for 3 years that I can see. First of all, for awhile it required Java 1.5, which was a showstopper for many (me included). Second, people complained about lack of integration with various tools (as they have here). That seems to be rolling in over time.

    The third reason is people saying "JUnit rocks!" because they've never used anything else, or they believe that since Kent Beck wrote it it must be perfect. This last reason is probably the biggest one. There is a _huge_ contingent of people who get very, very angry if you criticize any aspect of JUnit at all. Go look at blogs which criticize JUnit and you'll see many comments from angry torch-bearing XPers who are outraged that their personal god has been affronted.

    As for great ideas created by JUnit, I'd like to see 'em. The tool looks for test* methods via reflection, has a bunch of assertXYZ() methods in a class you have to extend, and has some crippled composition capabilities bolted on. Some of these were decent ideas but I fail to see where the "great" comes in.

    With all this aside, people should give TestNG a serious try. They might be surprised at how much more it gives you over JUnit, and you might just look back a few months later wondering why you ever put up with all the JUnit silliness.
  18. I agree. Still, I feel there are "issues" with this
    particular form of presentation @root.
  19. I agree. Still, I feel there are "issues" with thisparticular form of presentation @root.
    Are you talking about the syntax for annotations (using the @ sign)?

    --
    Cedric
  20. I prefer € ;)
  21. joining ?[ Go to top ]

    From one TestNG fanboy
    ...JUnit sucks in it's current form.
    and now this fanboy:
    JUnit has sucked for a very, very long time.

    Yeah, real objective. That's usually when I stop reading, another slashdot kiddie post.
  22. Assumptions[ Go to top ]

    Michael, where did I claim to be objective? Why do you assume I am objective?

    You might also note that you don't have to be a "slashdot kiddie" to have an opinion that something sucks, and say so. If you prefer "objective" wording, then let's say that JUnit was developed with very limited and limiting capabilities, and that TestNG provides a superior testing experience. You don't have to subclass the framework's stuff, rely on naming conventions, or deal with the ridiculous one-object-per-test-method-invocation stuff. You're not forced into using statics to work around the tool. And I admit the annotations are kinda cool :-)

    I would be interested to hear in why you think JUnit doesn't suck. Or is your sole contribution going to calling "fanboy!"?
  23. JUnit is pretty bad...[ Go to top ]

    JUnit is fine for doing simple static tests, i.e. a test class with testXxx() methods, but try something a little more complex or even just look under the covers.... Ewww.

    No offense to Eric et al, but from what I can see, there's no rhyme or reason to the JUnit class hierarchy. It looks like it's crippled by backward compatability and short design foresight. Comparing it to TestNG certainly supports Cedric's XP comments.

    I used JUnit to write a dynamic test suite to validate some design conventions at build time. I wanted it to appear as one test per use of the convention. The test would fail if the convention wasn't used properly. First, the only way I could set the test name was to extend TestCase rather than simply implementing Test. Also the test life cycle was broken (it would look like my tests never finished) until I figured out the proper interaction with TestResult. Overall, the JUnit API was counterintuitive and undocumented.

    Some may argue that JUnit is only for "unit tests" and I was pushing it too far. First, JUnit worked out OK in the end despite some ugliness under the hood. Second, am I really supposed to use yet another testing framework for slightly more complicated tests? There's no reason for TestNG or JUnit to not form the foundation for any type of testing.
  24. joining ?[ Go to top ]

    For the record, I'm not a TestNG fanboy. I would think that, as a minimum, you would have to actually use it on a project to qualify as a "fanboy". Sadly, I must report that I have not had the opportunity to use it in "the real world". Why? Because companies insist on JUnit.

    However, if you'd like to refer to me as "bitter" or something similar, feel free. I won't disagree.

    As for my terse "it sucks" commentaries...well, JUnit's flaws are well documented. Need I enumerate them once again? Indeed, anyone who has used JUnit for any length of time does not need to ask for such information.

    In truth, I'm brutaly utilitarian when it comes to tools. I'm a fan of what's best (even if it's not OSS). I don't really care if it has a famous name associated with it or whether it's the most popular. Mr. Spille has done an excellent job pointing out why JUnit is where it is. People see the word "Beck" and it's game over. So, who's the real fanboy?
  25. joining?[ Go to top ]

    Afraid of being inventive? No, we are not. I think you may know what means to keep an eye on the competition. Moreover, we are not considering JUnit as a competitor, as our spectrum is a little larger than just unit testing.
  26. Re: joining?[ Go to top ]

    Is it time for TestNG to contribute their work and adapt it to JUnit?

    Yes! Of course, this may not be possible, what do I know...

    I can already see the worse-case scenario brewing... the people at Sun decide that unit testing is so popular that there needs to be a standard framework and end up throwing it into the JDK (right after they implement the minimal HTTP server).

    Then someone at Apache Commons decides there needs to be a unit testing wrapper framework out there to wrap all these disparate frameworks, and in two years we end up having to use 4 logging, err, unit testing framework jars in each new project... ;-)
  27. Usability is important. KISS!
  28. lol[ Go to top ]

    "TestNG co-founders Cedric Beust and Alexandru Popescu
    took a look the code and have revealed at what's coming up in JUnit4"

    thanks. keep it up. will recommend JUnit4 to everyone ( :-) )
  29. I wonder if the @Test annotation is an example of over-use of annotations? I certainly never felt constraint by having to "use up" my single inheritance relationship by inheriting test classes from junit.framework.TestCase...

      cheers,
      gerald
  30. over use of anotations?[ Go to top ]

    I need to combine dbunit tests with cactus. Well well multiple inheritence. So what? Funtional injection is the sollution, is´t it. So...

    @junit-test
    @cactus-remote
    @dbunit-clean-tables

    Any better sollution?
  31. I wonder if the @Test annotation is an example of over-use of annotations?
    On the contrary, I think @Test is a great example of annotation use.

    When you are calling your method testFoo(), you are actually adding metadata to the method "foo()". This metadata should have no impact on your Java code and shouldn't impact your callers or people who read your JavaDocs.

    Also, imagine you use a tool that generates RMI proxies to all methods that have "remote" in their name: now you find yourself having to call your methods testRemoteFoo()... This is clearly not going to scale, as opposed to

    @Test
    @Remote
    public void foo() {
      ...
    }
    I certainly never felt constraint by having to "use up" my single inheritance relationship by inheriting test classes from junit.framework.TestCase...
    Some people do, but the main argument is, as I said above, that annotations are a natural fit in this case. Not having to extend TestCase is a nice side benefit.

    --
    Cedric
  32. I wonder if the @Test annotation is an example of over-use of annotations?

    On the contrary, I think @Test is a great example of annotation use.When you are calling your method testFoo(), you are actually adding metadata to the method "foo()". This metadata should have no impact on your Java code and shouldn't impact your callers or people who read your JavaDocs.Also, imagine you use a tool that generates RMI proxies to all methods that have "remote" in their name: now you find yourself having to call your methods testRemoteFoo()... This is clearly not going to scale, as opposed to

    @Test
    @Remote
    public void foo() {  ...}

    I certainly never felt constraint by having to "use up" my single inheritance relationship by inheriting test classes from junit.framework.TestCase...
    Some people do, but the main argument is, as I said above, that annotations are a natural fit in this case. Not having to extend TestCase is a nice side benefit.

    -- Cedric

    I would second the use of "@Test" for unit tests. NUnit already does this and I find it makes more sense to add annotation to my unit tests. Sometimes i don't like to label my methods "testBlah", but other times it makes sense. Adding support for annotations seems like a natural progression to me. If the feature is configurable, I see no harm having support for annotations.

    peter
  33. JUnit's goal always was to make writing a test as easy as "adding a debug statement to your code". We have found that using annotations (as many other test frameworks have done this by now) helps us to simplify JUnit. Both Kent and I were skceptical initially but this ended up as a pleasant journey:

    * @Test: it isn't really important that you no longer have to inherit from TestCase. What is cool is that you no longer have to know about the TestCase base class at all (OK now you have to add some static import of the Assert class).

    * @Test(expected= ExceptionClass) finally allows simplifying tests for expected exceptions. In the "classic" JUnit we have tried to address this but we failed. ExceptionTestCase is the trace of this. We could never make this case as simple as with an "expect: block" you could write in Smalltalk.

    * @Ignore("reason") is a good example for making a convention explicit. I have seen too many conventions to disable tests. The @Ignore annotation makes the convention explicit. As a benevolent side effect you can now easily find all disabled tests with a simple reference search for Ignore.

    --erich
  34. I can now see your point that establishing a naming convention for methods (such as test*(void) ) is actually a primordial way of adding meta-data to methods, that is better expressed using annotations. Very convincing.

      thanks,
      gerald
  35. I can now see your point that establishing a naming convention for methods (such as test*(void) ) is actually a primordial way of adding meta-data to methods, that is better expressed using annotations. Very convincing.  thanks,  gerald

    If a method is supposed to tell a reader what the method is for/does then test is an essential part of the name. It's no more meta data then any other part of the name. You might as well have a annotation called @Method to set the root part of the method name.
  36. If a method is supposed to tell a reader what the method is for/does then test is an essential part of the name. It's no more meta data then any other part of the name. You might as well have a annotation called @Method to set the root part of the method name.

    If a method name is chose to communicate to a human, then you are right and the method name is "just" that: a name.

    If, however, a method name must adhere to a naming convention because some software depends on that naming convention to do something special with that method, then you are essentially adding meta-data to the method: you are expressing the fact that the method is a member of a special set of methods. Examples that spring to mind:
      - said "void test*()" for test cases
      - JavaBeans getters/setters
      - "void ejbCreate(*)" for EJB "initialisers"

    Membership to a set of methods can be expressed more explicitly and precisely through annotations on the method than through a naming convention. Annotations also make membership to different sets of methods orthogonal without completely obfuscating method names (see Cedric's above example of remote test methods).

    And, as Erich pointed out above, there is other semantics that can piggyback on such an annotation, because the annotation allows you to pass attributes (such as expected exceptions for a test case) to the annotation handler.

    It all makes a lot of sense, I think.

      cheers,
      gerald

    http://www.gerald-loeffler.net