|
Sponsored Links
Resources
Enterprise Java
Research Library
Get Java white papers, product information, case studies and webcasts
|
News
News
News
|
Messages: 26
Messages: 26
Messages: 26
Printer friendly
Printer friendly
Printer friendly
Post reply
Post reply
Post reply
XML
XML
XML
|
 |
TestNG: Testing, the Next Generation
Cedric is at it again :) This time he has take a pinch of JUnit, a touch of NUnit, mixed them together with his special sauce and out came TestNG. This new testing framework features: JSR 175 Annotations, flexible test configuration, and a powerful execution model (no more TestSuite).
ExcerptA TestNG test class doesn't need to extend a class or implement an interface. It is a standard Java class annotated with annotations from the com.beust.testng.annotations.* package (only four annotations as I write this document). Read Cedric's TestNG announcement
Learn more at the TestNG home page
|
|
|
Message #119740
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Some good thinking but can this replace JUnit???
This seems like a good academic exercise but I think if we are to replace JUnit there is a lot more thinking to do. I too am frustrated with a few things in JUnit and I think the ideas in TestNG are interesting. Not needing to extend a class is nice. Using JDK1.5's annotations is a great idea, but it will take some time before everyone can switch to JDK. I think there is a typo with the setup/teardown:
public class BaseTest {
@Configuration(beforeTestClass = true)
public void setUp() {
}
@Configuration(beforeTestClass = true)
public void tearDown() {
}
Shouldn't the second method's annotation be false? It looks like setUp & tearDown would be run at the same time. Regarding the groups, I don't like the separation of "Functional" and "Check-in" tests (especially saying checkin tests are a subset of functional tests).. How about: Unit tests (run constantly while in development and before any check-in, integration tests (similar to unit tests but they test several units together), and functional tests (run during a daily build but can be run any time, may require a deployment). Finally, I really like the Assert class in JUnit. I like having Assert.assertNotNull(), Assert.assertNotEqual(), etc. I think it's more readable.
In summary, there are some good ideas here but I think replacing JUnit will require broad support from JUnit users. Lots of work to do here and TestNG has some good ideas to get started.
Michael Mattox
|
|
|
Message #119741
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Some good thinking but can this replace JUnit???
Using JDK1.5's annotations is a great idea, but it will take some time before everyone can switch to JDK. True, but as I said in the documentation, you only need JDK 1.5 to write and run the tests, but you can test Java programs written with any JDK. I think there is a typo with the setup/teardown:public class BaseTest { @Configuration(beforeTestClass = true) public void setUp() { } @Configuration(beforeTestClass = true) public void tearDown() { }Shouldn't the second method's annotation be false? Oops, you are right, typo, except that it should read "afterTestClass = true" for the tearDown() method. I fixed the documentation. Regarding the groups, I don't like the separation of "Functional" and "Check-in" tests (especially saying checkin tests are a subset of functional tests).. How about: Unit tests (run constantly while in development and before any check-in, integration tests (similar to unit tests but they test several units together), and functional tests (run during a daily build but can be run any time, may require a deployment). Well, that's really up to you, TestNG doesn't impose any methodology or terminology. Place your classes and methods into groups that make sense to you and invoke them as you see fit. Finally, I really like the Assert class in JUnit. I like having Assert.assertNotNull(), Assert.assertNotEqual(), etc. I think it's more readable. These are convenient methods but I think the price you have to pay to get them (extending a base class) is a little too expensive. It would certainly be easy to provide such methods in a static helper class.In summary, there are some good ideas here but I think replacing JUnit will require broad support from JUnit users. Lots of work to do here and TestNG has some good ideas to get started.Michael Mattox Thanks, that was my only intention at this early stage: get the community interested and throw a few new ideas around.
I have a lot of ideas for improvement for TestNG, they will hopefully come together in the next few months.
--
Cedric
|
|
|
Message #119742
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
ahhhh, ok...
I've been using JUnit for some time now, but was never sure WHY (and still am not). I knew I wanted/needed unit tests, so hey, why not use the thingy that everyone seems to be using.
Like Michael I also like the JUnit assertion methods. And the fact that each added Test method is executed automatically, also saves me the trouble of extending a main. So I have no beef with JUnit, but I also could be writing my own standalone unit testing classes (I really don't need any progressbar GUI for unit testing).
The same goes for TestNG. A plus is the addon to the "test methods are run automatically", by making that selective, but I could also enable that behaviour with a few if statements at the start of a JUnit test method checking a system property.
So for me this is not a "killer app", but I probably would use TestNG when I switch to JDK1.5.
|
|
|
Message #119743
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
utility class
Finally, I really like the Assert class in JUnit. I like having Assert.assertNotNull(), Assert.assertNotEqual(), etc. I think it's more readable. These are convenient methods but I think the price you have to pay to get them (extending a base class) is a little too expensive. It would certainly be easy to provide such methods in a static helper class.
One could also provide a utility class, e.g. Assertions.assertNotNull(), which does not require extending.
|
|
|
Message #119744
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
ahhhh, ok...
I've been using JUnit for some time now, but was never sure WHY (and still am not). I knew I wanted/needed unit tests, so hey, why not use the thingy that everyone seems to be using. Like Michael I also like the JUnit assertion methods. And the fact that each added Test method is executed automatically, also saves me the trouble of extending a main. So I have no beef with JUnit, but I also could be writing my own standalone unit testing classes (I really don't need any progressbar GUI for unit testing).The same goes for TestNG. A plus is the addon to the "test methods are run automatically", by making that selective, but I could also enable that behaviour with a few if statements at the start of a JUnit test method checking a system property. So for me this is not a "killer app", but I probably would use TestNG when I switch to JDK1.5. The "killer app" part is definitely tongue in cheek, kind of my signature :-)
You are making a good point about the "automatic" part. Right now, you need to add a @TestMethod annotation on top of a method in order to make it part of the test run, but I could actually make this optional by having all the methods in a TestClass part of a default group, say "test", which is run automatically when no include-group is specified.
This way, all the methods in your test class will by default be invoked even if they don't have any annotation.
I wonder if this is something desirable?
--
Cedric
|
|
|
Message #119760
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
All methods in a default test group
I wonder if this is something desirable? Yes, it is!
To run all tests in a test class requires that all public methods without an annotation and all annotated test methods are members in an implicit default group.
One more suggestion: It would be nice if you could select the groups to run by a regexp.
--
Ole
|
|
|
Message #119762
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Some good thinking but can this replace JUnit???
Finally, I really like the Assert class in JUnit. I like having Assert.assertNotNull(), Assert.assertNotEqual(), etc. I think it's more readable. In NUnit these are Assert.notNull(), Assert.notEqual() etc. which in my opinion is very nice and readable.
|
|
|
Message #119765
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
assert is static
Right now, you need to add a @TestMethod annotation on top of a method in order to make it part of the test run, but I could actually make this optional by having all the methods in a TestClass part of a default group, say "test", which is run automatically when no include-group is specified.This way, all the methods in your test class will by default be invoked even if they don't have any annotation.I wonder if this is something desirable?-- Cedric I think this is a good approach. In general, I think if you can provide the same behavior as JUnit as a default, that would enable very easy migration and lessen resistence. The power will be to go beyond JUnit functionality.
BTW the Assert in JUnit *is* static:
http://www.junit.org/junit/javadoc/3.8.1/index.htm
Michael
|
|
|
Message #119769
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
TestNG: Testing, the Next Generation
I like it. Obviously a long way to go--Jon's point about tool support for JUnit is a good one--but I think the use of attributes is very promising, as NUnit has already shown.
Regards,
Rod
|
|
|
Message #119773
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
TestNG: Testing, the Next Generation
I'm not a big admirer of NUnit's style tests, though I like some features of NUnit. Most interesting is ExpectedException Attribute. It is used to specify "expected" exception(s) that should/can be thrown by given test. It could be nice to have this feature in JUnit, it could simplify writing tests a lot for me. I will stay with JUnit for now (as it is rock solid) and will wait until JUnit authors incorporate new features using annotations.
Artur
|
|
|
Message #119780
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
TestNG: Testing, the Next Generation
Most interesting is ExpectedException Attribute. It is used to specify "expected" exception(s) that should/can be thrown by given test. It could be nice to have this feature in JUnit, it could simplify writing tests a lot for me. A while back I wrote a small utility class called ThrowableAssert which provides a set of JUnit-style assert methods:
ThrowableAssert.assertThrows(ThrowableOperation op, Throwable t);
ThrowableAssert.assertThrowsIllegalArgumentException(ThrowableOperation op);
etc...
Where ThrowableOperation is defined as an interface with a single method:
public void run() throws Throwable;
In case of a successful test, the thrown Throwable can be retrieved for further inspection, for example to assert that the expected error offset is provided by a thrown ParseException. It didn't take long to write and has been very useful. My point being that its very easy to extend JUnit to provide additional functionality.
|
|
|
Message #119787
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
TestNG: Testing, the Next Generation
A while back I wrote a small utility class called ThrowableAssert which provides a set of JUnit-style assert methods: ThrowableAssert.assertThrows(ThrowableOperation op, Throwable t); ThrowableAssert.assertThrowsIllegalArgumentException(ThrowableOperation op); etc...Where ThrowableOperation is defined as an interface with a single method: public void run() throws Throwable;In case of a successful test, the thrown Throwable can be retrieved for further inspection, for example to assert that the expected error offset is provided by a thrown ParseException. It didn't take long to write and has been very useful. My point being that its very easy to extend JUnit to provide additional functionality. Isn't it more difficult to create a new "command" class than to just do the following:
try {
doSomethingThatShouldThrow();
fail("Successfully did something wrong!");
} catch (MyException expected) {
}
|
|
|
Message #119793
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
TestNG: Testing, the Next Generation
We found that the problem with that structure is that developers can (and do) make mistakes - the classic being the test never fails: try {
somethingThatShouldThrow();
} catch (MyException expected) {
} With ThrowableAssert, the above test becomes: ThrowableAssert.assertThrows(new ThrowableOperation() {
public void run() throws MyException {
somethingThatShouldThrow();
}
}, MyException.class); Or for supported common exceptions: ThrowableAssert.assertThrowsIllegalStateException(new ThrowableOperation() {
public void run() {
myObj.doB(); // Oops, wrong way round ...
myObj.doA(); // ... should throw an IllegalStateException
}
}); IMHO, its more difficult to track down a bug caused by the first flawed test case than it is to write a test case using ThrowableAssert, which by design, makes it far harder to write bad unit tests - w.r.t exceptions, of course :)
|
|
|
Message #119795
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
TestNG: Testing, the Next Generation
If you don't like it, it does not mean it is not good ( I am reffeing to JUnit).
This 'Next Generation' thing is only used by ms like companies. I would improve what we have.
Junit is a framework used my real programmers.
You are just trying to make java/junit complicated. If you like complicated things, I think MS will be happy to hire you.
I would keep things that work in place. Java succeeded being simple.
Please stop pushing for changes that make sense to you only. I hope you hear the crowd - stop killing java.
|
|
|
Message #119796
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
TestNG: Testing, the Next Generation
IMHO, its more difficult to track down a bug caused by the first flawed test case than it is to write a test case using ThrowableAssert, which by design, makes it far harder to write bad unit tests - w.r.t exceptions, of course :) This seems to be a matter of taste. I see your point but I still prefer my way for these reasons:
- missing fail statements are easy to spot (maybe even automatically?)
- the code is more clear (less verbose)
- tests need to be written, checked and rechecked with great care anyway
How do you focus the exception expectations (funny expression:-) to a small piece of code? Do you pass the current state as constructor parameters to your ThrowableOperation object? That's quite verbose IMHO. (Alternatively you could put everything else in setUp and tearDown but that makes it impossible to use the test class for anything else.)
For me your pattern would make it harder not only to write bad unit tests but unit tests period:)
As tests are often a kind of specification, they need to be as clear as possible so it's easy to check if they are sane.
Anyway, let's just use whatever way that makes us most productive.
Ville Oikarinen
|
|
|
Message #119798
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
What?
This is just bad thinking. People shouldn't innovate?
Cedric is not trying to complicate things; he's trying to make things simpler. If you don't think he has done that, then say why. It seems some people think he's going in an interesting direction.
I certainly hope people continue to experiment and try to improve Java. I like JUnit a lot, but if I can have a framework that does the same things in fewer classes and more flexibly, I'm interested! Further, JUnit has some problems. Perhaps by following Cedric's thinking we can make it simpler, more flexible and more generally applicable.
You're opposed to this?
--Kevin
|
|
|
Message #119801
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
TestNG: Testing, the Next Generation
There's some great ideas in here, Cedric. Let me throw out some criticisms. There are a couple of things that feel rather sloppy:
1) @ExpectedException attttribute. There are two problems with this. First of all, you have a real lack of specificity by saying the Exception can be thrown anywhere in the test to pass. We really should expect developers to specify the specific method where the exception is thrown. We have already seen a couple of examples of how to do this in this thread.
Second, you have broken the paradign by moving the assertion out of the assert statement and into the annotation. This inconsistency makes the tests harder to read, IMHO.
2) If you are going to support multiple beforeTest..., afterTest... methods, then you have to provide a means for specifying the ordering of these methods. It is probably cleaner to only allow one of each (like JUnit does), and allow the single method to specify the ordering of set-up and tear-down tasks.
|
|
|
Message #119803
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
TestNG: Testing, the Next Generation
Good points Corby.There's some great ideas in here, Cedric. Let me throw out some criticisms. There are a couple of things that feel rather sloppy:1) @ExpectedException attttribute. There are two problems with this. First of all, you have a real lack of specificity by saying the Exception can be thrown anywhere in the test to pass. We really should expect developers to specify the specific method where the exception is thrown. We have already seen a couple of examples of how to do this in this thread. True. I copied ExpectedException from NUnit but just like you, I have mixed feelings about it. In a way, it forces you to write a simpler test method since its sole purpose is to throw an exception. I think this might come in handy when you expect something like RollbackException or ObjectNotFoundException.
For more complex cases, you should just use a regular test method and put your asserts in the right locations (try, catch, finally). Second, you have broken the paradign by moving the assertion out of the assert statement and into the annotation. This inconsistency makes the tests harder to read, True. IMHO.2) If you are going to support multiple beforeTest..., afterTest... methods, then you have to provide a means for specifying the ordering of these methods. It is probably cleaner to only allow one of each (like JUnit does), and allow the single method to specify the ordering of set-up and tear-down tasks. Agreed. I think people will typically only use one method for each (and I further suspect they will call it setUp() and tearDown() :-)), but the multiple method feature just comes for free, so I thought "why not?".
--
Cedric
http://beust.com/weblog
|
|
|
Message #119809
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
TestNG: Testing, the Next Generation
JUnit is poorly designed. One of the most ridiculous annoyances is that (last time I used it) the JUnit test runner had no way to specify the order of tests. yes, I know they are unit tests so order doesn't matter, but sometimes it does! Sun's test tool for compatibility test suites is a much better example of a good test tool. It allows exclusion lists, creating test lists that execute in a particular order, etc. To the extent that these features can be brought forth in a simple tool to replace JUnit, God bless.
-geoff
|
|
|
Message #119823
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
TestNG: Testing, the Next Generation
one more new thing on the board - with mostly poor documentation. i know its free open source blah blah blah... most of junior programmers like me dont care anymore.. too many kits released everyday - just causing confusion. what to use and what not to use. and worst is the documentation.
|
|
|
Message #119825
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
TestNG: Testing, the Next Generation
Hi geoff, your assertion is wrong.
From junit javadoc :
A test runner either expects a static method suite as the entry point to get a test to run or it will extract the suite automatically.
public static Test suite() {
suite.addTest(new MathTest("testAdd"));
suite.addTest(new MathTest("testDivideByZero"));
return suite;
}
meaning that if you provide a suite() method you can describe an ordered suite of test methods.
JUnit is a very flexible framework, this is a common quality of good design. It's just a matter of understanding the how and the why : there's always tradeoffs in any framework.
Using annotation (which were not available at the time of JUnit design) will probably simplify test cases in future but so far JUnit is doing very very well (think about the number of extensions you can find right now).
|
|
|
Message #119841
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
TestNG: Testing, the Next Generation
I couldn't imagine one writing tests using this 'Next Generation' thing.
I think that Junit is good (very good) for what was suppose to be - a testing framework.
I think that writing tests with annotations would be at least hard to read. I am not afraid of things that make life easier, but to me testing is something that you don't want to mess up with and create some super abstractions.
:wq
|
|
|
Message #120004
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Assert.assertNotNull(), etc.
I don't think anyone else pointed this out, but if someone did, oops.
You don't have to lose Assert.assertNull(), etc., if you stop inheriting from a base class. You could use the new Java 1.5 feature, static imports (http://www.jcp.org/en/jsr/detail?id=201), to just import those handy methods. That feature is still going to be a part of 1.5, right?
Moses
|
|
|
Message #120266
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Unit test -> functional test -> scalability test -> qos monitor
TestNG looks like a natural progression of unit testing frameworks to me. I've seen unit testing become really popular among developers over the past 2 years. (Lots of them reject Agile Programming but instead adopt a test-first attitude.)
TestNG's documentation urges developers to reuse unit tests as functional tests to also assist in deploying an application. I write about that kind of reuse in my new book Java Testing and Design: From Unit Tests to Automated Web Tests (Prentice Hall). Details on the book are at http://thebook.pushtotest.com.
In my view software developers, QA technicians and IT managers work together nicely when they all use a common framework that turns unit tests into functional tests. QA technicians use the functional tests when run multiple titmes concurrently as a scalability/load test. IT managers can reuse the functional tests as a quality of service (QOS) monitor by leaving the tests running over a long period of time. By leveraging the software developer's unit test the whole enterprise delivers better service to the user.
-Frank Cohen
PushToTest
http://www.pushtotest.com
|
|
|
Message #167040
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
It will be better to combine testNG with junit
I am using junit to do unit test. I have try to use testNG, but I give it up. I like the style of junit and I think junit is simple. But junit can not test several methods or classes in order that is why I try testNG. I hope there is tool based on junit and can provide rules testing serveral methods or classes in order.That will be perfect.
|
|
|
| New content on TheServerSide.comNew content on TheServerSide.comNew content on TheServerSide.com |
|
|
Reza Rahman explores the features of the proposed JSR 299, Contexts and Dependency Injection for Java EE (CDI). When approved, it promises to be a key feature of Java EE 6.
(November 2, Article)
SAML is an XML-based standard for exchanging authentication and authorization data between security domains. The single most important problem that SAML was created to solve is the Web browser Single Sign-On problem. Many organizations are debating whether to stay with version 1.1 or move to 2.0. This article makes observations about both options.
(September 28, Article)
Joe Ottinger takes a look at how people learn, and applies it to the practice of programming. He notes that understanding how people learn is an essential part of working in a programming team.
(September 22, Article)
Stephen Maryka gave us an article about the Asynchronous Web and posed a number of questions that get examined like an approach to delivering Asynchronous Web capabilities through extensions to existing Java EE technologies.
(July 14, Article)
JavaServer Faces Flex goal is to provide users capability in creating standard Flex components, part of flexSDK which is open sourced through MPL license, as normal JSF components. This article by Ji Hoon Kim will provide an overview of creating a simple multilingual JSF page consisting of JSF Flex tags.
(June 29, Article)
In this session Jeff explores the key characteristics of successful SOA projects. He covers some of the patterns, and anti-patterns, tool sets, and strategies that he himself learned the hard way. Last, he provides a strategy and blueprint for achieving a high likelihood of success in your SOA project.
(June 23, Tech Talk)
Ari Zilka, CTO of Terracotta, Inc., talks about the new features in Terracotta 3.1, announced during JavaOne and available now.
(June 15, Tech Talk)
In this Tech Talk, Josh Long explores an integration challenge using Spring Integration and walks through the implementation, employing and expanding on the basic patterns of Enterprise Application Integration to tie together components into a function integration solution, and then demonstrates how Spring Integration helps address the integration requirements.
(June 15, Tech Talk)
In this Tech Talk, David Geary teaches you: The basics of Google Web Toolkit; How to implement Ajax-enabled applications in Java; Internationalization; Hooking into the browser history mechanism; Remote procedure calls.
(June 4, Tech Talk)
Jon Kern discusses the best architecture/technical solutions and ensure that they are repeated by all developers. By tackling the architecture up-front in a serial manner, subsequent parallel development will be much more manageable and predictable.
(May 28, Tech Talk)
This keynote describes the frustrations of modern knowledge workers in their quest to actually get some work done, and solutions for how to guard yourself against all those distractions. Neal Ford talks about environments, coding, acceleration, automation, and avoiding repetition as ways to defeat the misguided attempts to sap your ability to produce good work.
(May 26, Tech Talk)
Gil demonstrates how new, aggressive uses of already abundant compute capacity by common applications offer competitive value for application designers.
(May 21, Tech Talk)
Chris Keene introduces WaveMaker as a new way to automate the ability to generate Hibernate classes in order to more quickly bring OR mapping into an application.
(May 19, Article)
In this session Nati Shalom demonstrates how to take a standard Java EE web application and scale it out or down dynamically without changes to the application code. Seeing as most web applications are over-provisioned to meet infrequent peak loads, this is a dramatic change because it enables growing your application as needed, when needed, without paying for unutilized resources.
(May 19, Tech Talk)
Download the entire book of Jakarta-Struts Live and learn about Struts MVC, Tiles, the Validator, DynaActionForms, plug-ins, internationalization, and more.
(Book PDF Download)
The Application Server Matrix is a detailed listing of J2EE vendors and their application server products, with information on latest version numbers, J2EE spec support and licensing, pricing, platform support, and links to product downloads and reviews.
(Application Server Comparison Matrix)
|
|
