Discussions

News: TestNG: Catch the Testing Fever

  1. TestNG: Catch the Testing Fever (24 messages)

    Javalobby has posted a new introductory article on TestNG by Matthew Schmidt, a relatively new testing framework that competes (and sometimes complements) JUnit. The article discusses the TestNG annotations, Ant integration, producing pretty reports with JUnitReport, and how to use TestNG with JDK 1.4.

    Read the article: TestNG: Catch the Testing Fever

    Threaded Messages (24)

  2. Just looking through the examples provided, it seems that TestNG looked more complicated than a similar JUnit setup. Granted I do not run JUnit tests from the command line directly ever since I used WSAD/Eclipse and Eclipse provides facilities to run tests selectively already.

    As for the naming conventions, once you know setUp, tearDown and test methods beginning with the word test I am not really sure what is too complicated about it. Its the same complexity as learning annotations.
  3. Just looking through the examples provided, it seems that TestNG looked more complicated than a similar JUnit setup. Granted I do not run JUnit tests from the command line directly ever since I used WSAD/Eclipse and Eclipse provides facilities to run tests selectively already.As for the naming conventions, once you know setUp, tearDown and test methods beginning with the word test I am not really sure what is too complicated about it. Its the same complexity as learning annotations.

    Good question. Neither the article nor TestNG's own documentation seem to answer the most basic question:

    What benefits does TestNG bring over JUnit? (apart from the possiblity that you might learn XML).

    In other words, why should people choose TestNG over JUnit?
  4. How is TestNG better than JUnit[ Go to top ]

    Good question. Neither the article nor TestNG's own documentation seem to answer the most basic question:What benefits does TestNG bring over JUnit? (apart from the possiblity that you might learn XML).In other words, why should people choose TestNG over JUnit?
    I purposely didn't make make the TestNG documentation a marketing pamphlet, so I'm sorry if you still have unanswered questions.

    Here are a few highlights:

    - JUnit forces a programming (need to extend a base class) and syntactic (need to name your methods a certain way and they must conform to a certain signature) model upon you. Not POJO-programming friendly.

    - You need to recompile whenever you change the tests you are running. You shouldn't have to, a good testing framework should make the difference between the static and dynamic model of your tests.

    - JUnit doesn't support dependent methods (important as soon as you do more than just unit testing).

    - JUnit doesn't let you pass parameters to your test methods.

    - JUnit instantiates your test class for every single method call.

    - JUnit forces you to use statics to maintain state between test methods.

    ... and much more.

    Take the time to read the documentation, I wouldn't be surprised if at some point, you don't think "Yeah, I could have used this feature in my own tests".

    --
    Cedric
    http://beust.com/weblog
  5. Thanks for the info. I think your article should at least link to this list of benefits (if it is there at least some place obvious). It helps set the background for those that never heard of testng before.
  6. How is TestNG better than JUnit[ Go to top ]

    I personally could use all the features that you've mentioned, Cedric, and I'm sure most people who've used JUnit for a while and encountered some of its warts could too.

    How is the Eclipse plugin coming along? I see the java.net project, but there is 0 activity there as far as I can see.

    cheers..
  7. How is TestNG better than JUnit[ Go to top ]

    I personally could use all the features that you've mentioned, Cedric, and I'm sure most people who've used JUnit for a while and encountered some of its warts could too.How is the Eclipse plugin coming along? I see the java.net project, but there is 0 activity there as far as I can see.cheers..
    Right, Alexandru hasn't checked in his code there yet, but I've seen it, so I promise you it's not vaporware :-)

    --
    Cedric
  8. How is TestNG better than JUnit[ Go to top ]

    Hi Cedric, TestNG looks like a fine testing framework with some incremental improvements over JUnit, but I'm not entirely convinced that it is worth the switch.
    - JUnit forces a programming (need to extend a base class) and syntactic (need to name your methods a certain way and they must conform to a certain signature) model upon you. Not POJO-programming friendly.

    Generally, we avoid using frameworks that force a programming/syntactic model because we want our code to easily port from one framework to the next. Ie. we don't want to be "coupled" to a particular framework. I would argue that this is not much of an issue when it comes to testing frameworks. In general, all test frameworks due pretty much the same thing, which is not very much. They dynamically invoke my test code. While it is true that JUnit tests are tightly coupled to the JUnit framework, it is also true that TestNG tests are coupled to the TestNG framework (they use TestNG specific annotations.). But, does it really matter? How likely is it that I will ever need to port my unit tests? On the other hand, JUnit tests written for JDK1.4 will work on JDK1.5 and vice-versa, but TestNG tests are coupled to the JDK version. Now ask yourself how likely is it that I will need to port an application (and unit tests) to the new JDK?
    - You need to recompile whenever you change the tests you are running. You shouldn't have to, a good testing framework should make the difference between the static and dynamic model of your tests.

    This I agree with. It is valuable to be able to change the set of tests I run without having to edit/re-compile my test code. JUnit does not prohibit this. It just isn't built into the framework. I end up creating extra code in my test suites to allow this behavior, so it is good that TestNG includes the capabillity.
    - JUnit doesn't support dependent methods (important as soon as you do more than just unit testing).

    I believe that unit test frameworks should not encourage creating dependencies betweent test methods. In general, I think the best strategy is to create test methods that can run successfully regardless of order. This requires more work, because each method has to ensure it's own setup and teardown, but it creates a more stable and modular test suite.
    - JUnit doesn't let you pass parameters to your test methods.

    This is true, although I'm not sure what the value of parameterizing test methods would be. Once I have decided upon the test parameters I will use to validate a piece of code, does it matter whether those parameters live in the test method or a configuration file?
    - JUnit instantiates your test class for every single method call.

    This ensures isolation between my tests and discourages dependencies. Again, I feel that it is a good strategy to have completely isolated tests methods.
    - JUnit forces you to use statics to maintain state between test methods....

    I won't beat the dead horse, here. :)


    I really think you've created a nice alternative to JUnit, but I'm not sure that it has any distinct advantages. In general, test frameworks don't do a whole heck of a lot, so I'm not sure if there is much room for improvement on these tools (at least in terms of their abillity to inoke my test code and graphically represent how my tests faired). I believe the real improvements in unit testsing will come with tools surronding these frameworks.
    • Better IDE integration (continuous integration style a la Eclipse)
    • Code generation (automatically generate test cases whenever I create a class or method)
    • Better data setup and verification tools (It is such a pain to have to create and persist my entities to setup the test, then run the test, then go grab my entities, then verify those entities look the way they should)
    Anyway, that's just my $0.02.
  9. How is TestNG better than JUnit[ Go to top ]

    Hi Jacob,

    That's a very fair evaluation of the strengths and weaknesses of TestNG, thanks for taking the time to phrase it. I agree overall with everything you say, so I'll just add a few comments.
    Generally, we avoid using frameworks that force a programming/syntactic model because we want our code to easily port from one framework to the next. Ie. we don't want to be "coupled" to a particular framework.
    The argument I was making wasn't so much with a view to porting but more from the standpoint of intrusiveness. What I have heard from TestNG users is that they enjoyed being able to write real POJO classes. If you looked at the code, it just looks like Java classes written in isolation, no forced hierarchy or forced naming, etc...

    As for the annotations, I'll just point out that "JDK 1.4 annotations" are actually comments, so the coupling is minimal, but then again, I won't dispute your base argument: there's little reason to port from one testing framework to another.
    I believe that unit test frameworks should not encourage creating dependencies betweent test methods. In general, I think the best strategy is to create test methods that can run successfully regardless of order. This requires more work, because each method has to ensure it's own setup and teardown, but it creates a more stable and modular test suite.
    That's the generally admitted knowledge, and I won't argue against it: I agree that in general, you want to build your tests independently from each other.

    Having said that, I have always found that as soon as I do more than unit testing, a small portion of my tests need this feature, and that JUnit was getting in my way for these cases.

    Also, since TestNG handles these dependencies for you, from your standpoint, it's not much different than if your tests were independent: all the necessary methods are invoked in the right order automatically.
    This is true, although I'm not sure what the value of parameterizing test methods would be. Once I have decided upon the test parameters I will use to validate a piece of code, does it matter whether those parameters live in the test method or a configuration file?
    Right. Call it syntactic sugar :-)

    On one hand, you have to implement the look up of the XML/properties/whatever file (probably in your test base class) and invoke it at the beginning of each test method, and with TestNG, these parameters are handed off to you "the Java way"": as parameters.

    Not indispensable, but nice to have.
    In general, test frameworks don't do a whole heck of a lot, so I'm not sure if there is much room for improvement on these tools
    That's a very astute observation, and the fact that so many people are quite happy with JUnit supports your point :-)
        * Better IDE integration (continuous integration style a la Eclipse)
        * Code generation (automatically generate test cases whenever I create a class or method)
        * Better data setup and verification tools (It is such a pain to have to create and persist my entities to setup the test, then run the test, then go grab my entities, then verify those entities look the way they should)
    Absolute agreement here, and that's what we are going to focus on from now on...
    Thanks for your $.02!

    --
    Cedric
  10. How is TestNG better than JUnit[ Go to top ]

    As for the annotations, I'll just point out that "JDK 1.4 annotations" are actually comments, so the coupling is minimal, but then again, I won't dispute your base argument: there's little reason to port from one testing framework to another.

    Yeah, after I thought about this some more, I realized that TestNG really doesn't couple you to the framework (or to the JDK version). You're right, the annotations are just comments and a test framework can selectively interpret the annotations it recognizes. This actually is a bigger improvement than I originally gave it credit for.


    Anyway, take my observations for what they are. At first glance, TestNG does look better than JUnit, but for it to gain critical mass the way JUnit did, it will need to offer something really compelling (ie the three bullets in my previous post).
  11. yet another toy ![ Go to top ]

    Over the years, I am getting a little tired of unit testing frameworks trumpeting their so called benefits! So they can do unit testing of some java code -- big deal !!

    The challenge in in J2EE applications -- especially distributed applications is in ensuring that the system can get the data we expect, ensuring that your SQL joins and unions are returning the rows you expect and not the rows you dont want, ensuring that various services components are implemting their interfaces correctly. Al the while also ensuring that the the data that you insert/retrive matches what the use case specified.

    And being able to do this pre-QA. Thats where I see the real value. Otherwise its all just a toy with pretty colors!
  12. agree with you but...[ Go to top ]

    hi

    i agree with you re. the so called benefits of unit testing. however, having worked in the development of back end services for the last two projects, I can tell you that unit testing along with integration testing allows me to verify exactly that sort of results.. and yes, unit testing with junit, mock testing etc....
  13. agree with you but...[ Go to top ]

    Over the years, I am getting a little tired of unit testing frameworks trumpeting their so called benefits! So they can do unit testing of some java code -- big deal !!The challenge in in J2EE applications -- especially distributed applications is in ensuring that the system can get the data we expect, ensuring that your SQL joins and unions are returning the rows you expect and not the rows you dont want, ensuring that various services components are implemting their interfaces correctly. Al the while also ensuring that the the data that you insert/retrive matches what the use case specified.And being able to do this pre-QA. Thats where I see the real value. Otherwise its all just a toy with pretty colors!
  14. yet another toy ![ Go to top ]

    sorry for the two previous posting...

    Over the years, I am getting a little tired of unit testing frameworks trumpeting their so called benefits! So they can do unit testing of some java code -- big deal !!The challenge in in J2EE applications -- especially distributed applications is in ensuring that the system can get the data we expect, ensuring that your SQL joins and unions are returning the rows you expect and not the rows you dont want, ensuring that various services components are implemting their interfaces correctly. Al the while also ensuring that the the data that you insert/retrive matches what the use case specified.And being able to do this pre-QA. Thats where I see the real value. Otherwise its all just a toy with pretty colors!

    hi

    i agree with you re. the so called benefits of unit testing. however, having worked in the development of back end services for the last two projects, I can tell you that unit testing along with integration testing allows me to verify exactly that sort of results.. and yes, unit testing with junit, mock testing etc....
    cheers
  15. integration with IDEs?[ Go to top ]

    This looks to be a definite improvement over JUnit. Of course one of the advantages of being the incumbent is tools support. Eclipse, IDEA etc all have built in JUnit support which I find very useful.

    Is there any IDE support out there for TestNG at the moment? Any plans?

    regards
    Mike
  16. integration with IDEs?[ Go to top ]

    This looks to be a definite improvement over JUnit. Of course one of the advantages of being the incumbent is tools support. Eclipse, IDEA etc all have built in JUnit support which I find very useful. Is there any IDE support out there for TestNG at the moment? Any plans? regardsMike
    Yes, one of the developers is working on an Eclipse plug-in as we speak. No plans for IntelliJ right now.

    --
    Cedric
  17. Unit-Testing[ Go to top ]

    IMHO unit testing frameworks like TestNG or JUnit were a big step forward by providing a simple and easy to learn scripting mechanism.

    However in real projects test productivity is still very low. Most times we are struggling with the reproducability of test results in environments with persistent data.
    It's simple things that are not covered by those frameworks that require most testing effort, like resetting the environment and the database to some defined state within reasonable time, importing and managing test data etc.

    In reality unit tests fail to provide the required test coverage because they are still too much work to write.
  18. Most times we are struggling with the reproducability of test results in environments with persistent data.It's simple things that are not covered by those frameworks that require most testing effort, like resetting the environment and the database to some defined state within reasonable time, importing and managing test data etc.In reality unit tests fail to provide the required test coverage because they are still too much work to write.

    IMHO, Use JUnit based extensions like DBUnit to achieve reproducability. They are easy to use and do just what you're talking about, and very importantly, are extensible!
  19. Most times we are struggling with the reproducability of test results in environments with persistent data.It's simple things that are not covered by those frameworks that require most testing effort, like resetting the environment and the database to some defined state within reasonable time, importing and managing test data etc.In reality unit tests fail to provide the required test coverage because they are still too much work to write.
    IMHO, Use JUnit based extensions like DBUnit to achieve reproducability. They are easy to use and do just what you're talking about, and very importantly, are extensible!
    Also, making the logic being test not depend the existence of the persistance concern (or any other concern) will help too. At least for unit testing.
  20. Most times we are struggling with the reproducability of test results in environments with persistent data.It's simple things that are not covered by those frameworks that require most testing effort, like resetting the environment and the database to some defined state within reasonable time, importing and managing test data etc.In reality unit tests fail to provide the required test coverage because they are still too much work to write.
    IMHO, Use JUnit based extensions like DBUnit to achieve reproducability. They are easy to use and do just what you're talking about, and very importantly, are extensible!

    Combined that with an in-memory hsqldb database you get isolation. No developer will be running test against the same database 8-)
  21. Test data for TestNG[ Go to top ]

    Hi Cedric,

    I am a novice in the Unit testing and now I am learning JUnit and TestNG.
    It's interesting for me, does TestNG support several values of an input parameter for a test method? Say, I want to check some boundary values in a test method.
    I do not want to rewrite the same test method for each possible boundary value. I want that the test method would run several times for each possible input parameter. Is it possible in TestNG?

    Thanks.
     
    Stan
  22. Test data for TestNG[ Go to top ]

    Hi Cedric,I am a novice in the Unit testing and now I am learning JUnit and TestNG.It's interesting for me, does TestNG support several values of an input parameter for a test method? Say, I want to check some boundary values in a test method.I do not want to rewrite the same test method for each possible boundary value. I want that the test method would run several times for each possible input parameter. Is it possible in TestNG? Thanks. Stan
    Yes, you can use parameters for this.

    For example:

      @Test(parameters = { "age" })
      public void ageIsCorrect(Integer age) {

    and you define this parameter in testng.xml:

    <parameter name="age" value="24" />

    Full details here: http://beust.com/testng/main.html#parameters

    --
    Cedric
  23. Test data for TestNG[ Go to top ]

    Thanks, Cedric. But I meant another thing.
    For example, we have the method:

      @Test(parameters = { "age" })
      public void ageIsCorrect(Integer age) {

    and I want to define this parameter in testng.xml but with several possible values (in other words, I want to check this method with, say, some valid boundary values):

    <parameter name="age" value="24" />
    <parameter name="age" value="1" />
    <parameter name="age" value="100" />

    Now I want to run this method with each of these values. So that the method would be launched several times by TestNG, and TestNG would output the test results for each the boundary value. Is this possible now? Is it possible to not write this test method separately for each of the boundary values?

    Thanks.

    Stan.
  24. Test data for TestNG[ Go to top ]

    Thanks, Cedric. But I meant another thing.For example, we have the method:&nbsp;&nbsp;@Test(parameters = { "age" })&nbsp;&nbsp;public void ageIsCorrect(Integer age) {and I want to define this parameter in testng.xml but with several possible values (in other words, I want to check this method with, say, some valid boundary values):<parameter name="age" value="24" /><parameter name="age" value="1" /><parameter name="age" value="100" />Now I want to run this method with each of these values. So that the method would be launched several times by TestNG, and TestNG would output the test results for each the boundary value. Is this possible now? Is it possible to not write this test method separately for each of the boundary values?Thanks.Stan.

    Yes, you can do that: the <parameter> tag can be used either at the suite or at the test level

    <suite>
      <test name="Test1">
        <parameter name="age" value="24" />
        <...>
      </test>

      <test name="Test2">
        <parameter name="age" value="42" />
        <...>
      </test>
    </suite>

    --
    Cedric
  25. Test data for TestNG[ Go to top ]

    By the way Stanislas, feel free to email me directly if you have more questions... (cedric at beust.com)

    --
    Cedric