Discussions

News: TestNG: Testing, the Next Generation

  1. TestNG: Testing, the Next Generation (26 messages)

    Cedric is at it again :) This time he has take a pinch of JUnit, a touch of NUnit, mixed them together with his special sauce and out came TestNG. This new testing framework features: JSR 175 Annotations, flexible test configuration, and a powerful execution model (no more TestSuite).

    Excerpt
    A TestNG test class doesn't need to extend a class or implement an interface. It is a standard Java class annotated with annotations from the com.beust.testng.annotations.* package (only four annotations as I write this document).
    Read Cedric's TestNG announcement

    Learn more at the TestNG home page

    Threaded Messages (26)

  2. This seems like a good academic exercise but I think if we are to replace JUnit there is a lot more thinking to do. I too am frustrated with a few things in JUnit and I think the ideas in TestNG are interesting. Not needing to extend a class is nice. Using JDK1.5's annotations is a great idea, but it will take some time before everyone can switch to JDK. I think there is a typo with the setup/teardown:

    public class BaseTest {
      @Configuration(beforeTestClass = true)
      public void setUp() {
      }

      @Configuration(beforeTestClass = true)
      public void tearDown() {
      }

    Shouldn't the second method's annotation be false? It looks like setUp & tearDown would be run at the same time. Regarding the groups, I don't like the separation of "Functional" and "Check-in" tests (especially saying checkin tests are a subset of functional tests).. How about: Unit tests (run constantly while in development and before any check-in, integration tests (similar to unit tests but they test several units together), and functional tests (run during a daily build but can be run any time, may require a deployment). Finally, I really like the Assert class in JUnit. I like having Assert.assertNotNull(), Assert.assertNotEqual(), etc. I think it's more readable.

    In summary, there are some good ideas here but I think replacing JUnit will require broad support from JUnit users. Lots of work to do here and TestNG has some good ideas to get started.

    Michael Mattox
  3. Using JDK1.5's annotations is a great idea, but it will take some time before everyone can switch to JDK.
    True, but as I said in the documentation, you only need JDK 1.5 to write and run the tests, but you can test Java programs written with any JDK.
      I think there is a typo with the setup/teardown:public class BaseTest { @Configuration(beforeTestClass = true) public void setUp() { } @Configuration(beforeTestClass = true) public void tearDown() { }Shouldn't the second method's annotation be false?
    Oops, you are right, typo, except that it should read "afterTestClass = true" for the tearDown() method. I fixed the documentation.
     Regarding the groups, I don't like the separation of "Functional" and "Check-in" tests (especially saying checkin tests are a subset of functional tests).. How about: Unit tests (run constantly while in development and before any check-in, integration tests (similar to unit tests but they test several units together), and functional tests (run during a daily build but can be run any time, may require a deployment).
    Well, that's really up to you, TestNG doesn't impose any methodology or terminology. Place your classes and methods into groups that make sense to you and invoke them as you see fit.
      Finally, I really like the Assert class in JUnit. I like having Assert.assertNotNull(), Assert.assertNotEqual(), etc. I think it's more readable.
    These are convenient methods but I think the price you have to pay to get them (extending a base class) is a little too expensive. It would certainly be easy to provide such methods in a static helper class.
    In summary, there are some good ideas here but I think replacing JUnit will require broad support from JUnit users. Lots of work to do here and TestNG has some good ideas to get started.Michael Mattox
    Thanks, that was my only intention at this early stage: get the community interested and throw a few new ideas around.

    I have a lot of ideas for improvement for TestNG, they will hopefully come together in the next few months.

    --
    Cedric
  4. utility class[ Go to top ]

      Finally, I really like the Assert class in JUnit. I like having Assert.assertNotNull(), Assert.assertNotEqual(), etc. I think it's more readable.
    These are convenient methods but I think the price you have to pay to get them (extending a base class) is a little too expensive. It would certainly be easy to provide such methods in a static helper class.
    One could also provide a utility class, e.g. Assertions.assertNotNull(), which does not require extending.
  5. Assert.assertNotNull(), etc.[ Go to top ]

    I don't think anyone else pointed this out, but if someone did, oops.

    You don't have to lose Assert.assertNull(), etc., if you stop inheriting from a base class. You could use the new Java 1.5 feature, static imports (http://www.jcp.org/en/jsr/detail?id=201), to just import those handy methods. That feature is still going to be a part of 1.5, right?

    Moses
  6. ahhhh, ok...[ Go to top ]

    I've been using JUnit for some time now, but was never sure WHY (and still am not). I knew I wanted/needed unit tests, so hey, why not use the thingy that everyone seems to be using.
    Like Michael I also like the JUnit assertion methods. And the fact that each added Test method is executed automatically, also saves me the trouble of extending a main. So I have no beef with JUnit, but I also could be writing my own standalone unit testing classes (I really don't need any progressbar GUI for unit testing).

    The same goes for TestNG. A plus is the addon to the "test methods are run automatically", by making that selective, but I could also enable that behaviour with a few if statements at the start of a JUnit test method checking a system property.

    So for me this is not a "killer app", but I probably would use TestNG when I switch to JDK1.5.
  7. ahhhh, ok...[ Go to top ]

    I've been using JUnit for some time now, but was never sure WHY (and still am not). I knew I wanted/needed unit tests, so hey, why not use the thingy that everyone seems to be using. Like Michael I also like the JUnit assertion methods. And the fact that each added Test method is executed automatically, also saves me the trouble of extending a main. So I have no beef with JUnit, but I also could be writing my own standalone unit testing classes (I really don't need any progressbar GUI for unit testing).The same goes for TestNG. A plus is the addon to the "test methods are run automatically", by making that selective, but I could also enable that behaviour with a few if statements at the start of a JUnit test method checking a system property. So for me this is not a "killer app", but I probably would use TestNG when I switch to JDK1.5.
    The "killer app" part is definitely tongue in cheek, kind of my signature :-)

    You are making a good point about the "automatic" part. Right now, you need to add a @TestMethod annotation on top of a method in order to make it part of the test run, but I could actually make this optional by having all the methods in a TestClass part of a default group, say "test", which is run automatically when no include-group is specified.

    This way, all the methods in your test class will by default be invoked even if they don't have any annotation.

    I wonder if this is something desirable?

    --
    Cedric
  8. All methods in a default test group[ Go to top ]

    I wonder if this is something desirable?
    Yes, it is!
    To run all tests in a test class requires that all public methods without an annotation and all annotated test methods are members in an implicit default group.

    One more suggestion: It would be nice if you could select the groups to run by a regexp.

    --
    Ole
  9. assert is static[ Go to top ]

    Right now, you need to add a @TestMethod annotation on top of a method in order to make it part of the test run, but I could actually make this optional by having all the methods in a TestClass part of a default group, say "test", which is run automatically when no include-group is specified.This way, all the methods in your test class will by default be invoked even if they don't have any annotation.I wonder if this is something desirable?-- Cedric
    I think this is a good approach. In general, I think if you can provide the same behavior as JUnit as a default, that would enable very easy migration and lessen resistence. The power will be to go beyond JUnit functionality.

    BTW the Assert in JUnit *is* static:

    http://www.junit.org/junit/javadoc/3.8.1/index.htm

    Michael
  10. Finally, I really like the Assert class in JUnit. I like having Assert.assertNotNull(), Assert.assertNotEqual(), etc. I think it's more readable.
    In NUnit these are Assert.notNull(), Assert.notEqual() etc. which in my opinion is very nice and readable.
  11. I am using junit to do unit test. I have try to use testNG, but I give it up. I like the style of junit and I think junit is simple. But junit can not test several methods or classes in order that is why I try testNG. I hope there is tool based on junit and can provide rules testing serveral methods or classes in order.That will be perfect.
  12. I like it. Obviously a long way to go--Jon's point about tool support for JUnit is a good one--but I think the use of attributes is very promising, as NUnit has already shown.

    Regards,
    Rod
  13. I'm not a big admirer of NUnit's style tests, though I like some features of NUnit. Most interesting is ExpectedException Attribute. It is used to specify "expected" exception(s) that should/can be thrown by given test. It could be nice to have this feature in JUnit, it could simplify writing tests a lot for me. I will stay with JUnit for now (as it is rock solid) and will wait until JUnit authors incorporate new features using annotations.

    Artur
  14. Most interesting is ExpectedException Attribute. It is used to specify "expected" exception(s) that should/can be thrown by given test. It could be nice to have this feature in JUnit, it could simplify writing tests a lot for me.
    A while back I wrote a small utility class called ThrowableAssert which provides a set of JUnit-style assert methods:

      ThrowableAssert.assertThrows(ThrowableOperation op, Throwable t);
      ThrowableAssert.assertThrowsIllegalArgumentException(ThrowableOperation op);
      etc...

    Where ThrowableOperation is defined as an interface with a single method:

      public void run() throws Throwable;

    In case of a successful test, the thrown Throwable can be retrieved for further inspection, for example to assert that the expected error offset is provided by a thrown ParseException. It didn't take long to write and has been very useful. My point being that its very easy to extend JUnit to provide additional functionality.
  15. A while back I wrote a small utility class called ThrowableAssert which provides a set of JUnit-style assert methods:  ThrowableAssert.assertThrows(ThrowableOperation op, Throwable t);  ThrowableAssert.assertThrowsIllegalArgumentException(ThrowableOperation op);  etc...Where ThrowableOperation is defined as an interface with a single method:  public void run() throws Throwable;In case of a successful test, the thrown Throwable can be retrieved for further inspection, for example to assert that the expected error offset is provided by a thrown ParseException. It didn't take long to write and has been very useful. My point being that its very easy to extend JUnit to provide additional functionality.
    Isn't it more difficult to create a new "command" class than to just do the following:
    try {
      doSomethingThatShouldThrow();
      fail("Successfully did something wrong!");
    } catch (MyException expected) {
    }
  16. We found that the problem with that structure is that developers can (and do) make mistakes - the classic being the test never fails:
      try {
        somethingThatShouldThrow();
      } catch (MyException expected) {
      }
    With ThrowableAssert, the above test becomes:
      ThrowableAssert.assertThrows(new ThrowableOperation() {
        public void run() throws MyException {
          somethingThatShouldThrow();
        }
      }, MyException.class);
    Or for supported common exceptions:
      ThrowableAssert.assertThrowsIllegalStateException(new ThrowableOperation() {
        public void run() {
          myObj.doB(); // Oops, wrong way round ...
          myObj.doA(); // ... should throw an IllegalStateException
        }
      });
    IMHO, its more difficult to track down a bug caused by the first flawed test case than it is to write a test case using ThrowableAssert, which by design, makes it far harder to write bad unit tests - w.r.t exceptions, of course :)
  17. IMHO, its more difficult to track down a bug caused by the first flawed test case than it is to write a test case using ThrowableAssert, which by design, makes it far harder to write bad unit tests - w.r.t exceptions, of course :)
    This seems to be a matter of taste. I see your point but I still prefer my way for these reasons:
    - missing fail statements are easy to spot (maybe even automatically?)
    - the code is more clear (less verbose)
    - tests need to be written, checked and rechecked with great care anyway

    How do you focus the exception expectations (funny expression:-) to a small piece of code? Do you pass the current state as constructor parameters to your ThrowableOperation object? That's quite verbose IMHO. (Alternatively you could put everything else in setUp and tearDown but that makes it impossible to use the test class for anything else.)

    For me your pattern would make it harder not only to write bad unit tests but unit tests period:)

    As tests are often a kind of specification, they need to be as clear as possible so it's easy to check if they are sane.

    Anyway, let's just use whatever way that makes us most productive.

    Ville Oikarinen
  18. If you don't like it, it does not mean it is not good ( I am reffeing to JUnit).
    This 'Next Generation' thing is only used by ms like companies. I would improve what we have.
    Junit is a framework used my real programmers.
    You are just trying to make java/junit complicated. If you like complicated things, I think MS will be happy to hire you.
    I would keep things that work in place. Java succeeded being simple.
    Please stop pushing for changes that make sense to you only. I hope you hear the crowd - stop killing java.
  19. What?[ Go to top ]

    This is just bad thinking. People shouldn't innovate?

    Cedric is not trying to complicate things; he's trying to make things simpler. If you don't think he has done that, then say why. It seems some people think he's going in an interesting direction.

    I certainly hope people continue to experiment and try to improve Java. I like JUnit a lot, but if I can have a framework that does the same things in fewer classes and more flexibly, I'm interested! Further, JUnit has some problems. Perhaps by following Cedric's thinking we can make it simpler, more flexible and more generally applicable.

    You're opposed to this?

    --Kevin
  20. JUnit is poorly designed. One of the most ridiculous annoyances is that (last time I used it) the JUnit test runner had no way to specify the order of tests. yes, I know they are unit tests so order doesn't matter, but sometimes it does! Sun's test tool for compatibility test suites is a much better example of a good test tool. It allows exclusion lists, creating test lists that execute in a particular order, etc. To the extent that these features can be brought forth in a simple tool to replace JUnit, God bless.

    -geoff
  21. Hi geoff, your assertion is wrong.
    From junit javadoc :

    A test runner either expects a static method suite as the entry point to get a test to run or it will extract the suite automatically.
    public static Test suite() {
          suite.addTest(new MathTest("testAdd"));
          suite.addTest(new MathTest("testDivideByZero"));
          return suite;
    }

    meaning that if you provide a suite() method you can describe an ordered suite of test methods.

    JUnit is a very flexible framework, this is a common quality of good design. It's just a matter of understanding the how and the why : there's always tradeoffs in any framework.
    Using annotation (which were not available at the time of JUnit design) will probably simplify test cases in future but so far JUnit is doing very very well (think about the number of extensions you can find right now).
  22. I couldn't imagine one writing tests using this 'Next Generation' thing.
    I think that Junit is good (very good) for what was suppose to be - a testing framework.
    I think that writing tests with annotations would be at least hard to read. I am not afraid of things that make life easier, but to me testing is something that you don't want to mess up with and create some super abstractions.
    :wq
  23. TestNG: Testing, the Next Generation[ Go to top ]

    There's some great ideas in here, Cedric. Let me throw out some criticisms. There are a couple of things that feel rather sloppy:

    1) @ExpectedException attttribute. There are two problems with this. First of all, you have a real lack of specificity by saying the Exception can be thrown anywhere in the test to pass. We really should expect developers to specify the specific method where the exception is thrown. We have already seen a couple of examples of how to do this in this thread.
       Second, you have broken the paradign by moving the assertion out of the assert statement and into the annotation. This inconsistency makes the tests harder to read, IMHO.

    2) If you are going to support multiple beforeTest..., afterTest... methods, then you have to provide a means for specifying the ordering of these methods. It is probably cleaner to only allow one of each (like JUnit does), and allow the single method to specify the ordering of set-up and tear-down tasks.
  24. Good points Corby.
    There's some great ideas in here, Cedric. Let me throw out some criticisms. There are a couple of things that feel rather sloppy:1) @ExpectedException attttribute. There are two problems with this. First of all, you have a real lack of specificity by saying the Exception can be thrown anywhere in the test to pass. We really should expect developers to specify the specific method where the exception is thrown. We have already seen a couple of examples of how to do this in this thread.
    True. I copied ExpectedException from NUnit but just like you, I have mixed feelings about it. In a way, it forces you to write a simpler test method since its sole purpose is to throw an exception. I think this might come in handy when you expect something like RollbackException or ObjectNotFoundException.

    For more complex cases, you should just use a regular test method and put your asserts in the right locations (try, catch, finally).
    Second, you have broken the paradign by moving the assertion out of the assert statement and into the annotation. This inconsistency makes the tests harder to read,
    True.
    IMHO.2) If you are going to support multiple beforeTest..., afterTest... methods, then you have to provide a means for specifying the ordering of these methods. It is probably cleaner to only allow one of each (like JUnit does), and allow the single method to specify the ordering of set-up and tear-down tasks.
    Agreed. I think people will typically only use one method for each (and I further suspect they will call it setUp() and tearDown() :-)), but the multiple method feature just comes for free, so I thought "why not?".

    --
    Cedric
    http://beust.com/weblog
  25. one more new thing on the board - with mostly poor documentation. i know its free open source blah blah blah... most of junior programmers like me dont care anymore.. too many kits released everyday - just causing confusion. what to use and what not to use. and worst is the documentation.
  26. I was thinking this afternoon and I had an epiphany: You should rename it to StrongTest, with the marketing line being "StrongTest is awesome."

    See also: http://www.homestarrunner.com/

    Peace,

    Cameron Purdy
    Tangosol, Inc.
    Coherence: Clustered JCache for Grid Computing!
  27. TestNG looks like a natural progression of unit testing frameworks to me. I've seen unit testing become really popular among developers over the past 2 years. (Lots of them reject Agile Programming but instead adopt a test-first attitude.)

    TestNG's documentation urges developers to reuse unit tests as functional tests to also assist in deploying an application. I write about that kind of reuse in my new book Java Testing and Design: From Unit Tests to Automated Web Tests (Prentice Hall). Details on the book are at http://thebook.pushtotest.com.

    In my view software developers, QA technicians and IT managers work together nicely when they all use a common framework that turns unit tests into functional tests. QA technicians use the functional tests when run multiple titmes concurrently as a scalability/load test. IT managers can reuse the functional tests as a quality of service (QOS) monitor by leaving the tests running over a long period of time. By leveraging the software developer's unit test the whole enterprise delivers better service to the user.

    -Frank Cohen
    PushToTest
    http://www.pushtotest.com