TestNG version 5.3 has been released

Discussions

News: TestNG version 5.3 has been released

  1. TestNG version 5.3 has been released (15 messages)

    TestNG 5.3 has been released. TestNG is a unit test framework which can be used in combination with Java 1.5 annotations or javadoc annotations if you are still using Java 1.4. One of the more interesting new features is the introduction of Annotation Transformers. Annotation Transformers can be used to modify the content of the annotations while running tests. This is useful if the annotations in the source code are correct most of the time, but there are situations where you'd like to override their attribute values. To get a clearer idea about the Annotation Transformers lets take a look at the the following example:public class MyTest { @Test(invocationCount = 2) public void testMethodA() { } @Test(invocationCount = 3) public void testMethodB() { } }This test case contains two methods, in which both @Test annotations have the invocationCount attribute. Now, if you would want to change the number of times a given method is invoked, you normally would have to change the source code. By using an Annotation tranformer you can do this programmatically:public class MyTransformer implements IAnnotationTransformer { public void transform(ITest annotation, Class testClass, Constructor testConstructor, Method testMethod) { if (testMethod != null) { String methodName = testMethod.getName(); if ("testMethodA".equals(methodName) { annotation.setInvocationCount(10); } else if ("testMethodB".equals(methodName) { annotation.setInvocationCount(20); } } } }Now, when the testcase is run the transformer will 'intercept' the call to the two test methods and it will first call the transform method on the MyTransformer class. This method will first check if the method name is equal to "testMethodA"; if this is the case then the invocationCount is changed from two to ten, which would have the effect that the test method is invoked ten times instead of two times. To register the custom transformer it is sufficient to add a command line option:java org.testng.TestNG -listener MyTransformer testng.xmlor programmatically:TestNG ng = new TestNG(); ng.setAnnotationTransformer(new MyTransformer());More information about the Annotation Transformer or other features take a look at the documentation for TestNG. Message was edited by: joeo@enigmastation.com

    Threaded Messages (15)

  2. Multiple Annotation Transformers[ Go to top ]

    Congratulations to TestNG team! It continues to outperform JUnit. Just out of curiosity, what if there is a need to have multiple Annotation Transformers (for each package or group) for example? I could imagine that it would make sense when there is a group of developers writing lots of test cases and they might need to divide their work into logical places/domains/packages with each having its own Annotation Transformers...
  3. Re: Multiple Annotation Transformers[ Go to top ]

    Hi Ruslan, and thanks!
    Congratulations to TestNG team! It continues to outperform JUnit.

    Just out of curiosity, what if there is a need to have multiple Annotation Transformers (for each package or group) for example? I could imagine that it would make sense when there is a group of developers writing lots of test cases and they might need to divide their work into logical places/domains/packages with each having its own Annotation Transformers...
    It would be pretty easy to add, but I didn't think there was much point: an annotation transformer is a simple class that will typically do a bunch of if/equals based on the class/method/annotation names and decide what to do. If you want to split this logic in several classes, it's pretty easy to do even if you only declare only one transformer to TestNG. -- Cedric
  4. Congratulations to TestNG team! It continues to outperform JUnit.

    Just out of curiosity, what if there is a need to have multiple Annotation Transformers (for each package or group) for example? I could imagine that it would make sense when there is a group of developers writing lots of test cases and they might need to divide their work into logical places/domains/packages with each having its own Annotation Transformers...
    I'm not shure how it was intended but to mee it seems that it is just to override some test settings in case you want to perform the test just a bit different then you would normaly do. Like for example to reproduce something that only occurs when te test is run multiple times. I woul not personaly recomend to use the Annotation Tranformers as a normal part of you're test suite. Since other developers would have to look in two different places to see wat the test settings are, it would be overlooked quite easy. Cedric, could you enlighten us on this? Was it intended to be used as a normal part of the test suite or was it intended to be used in a way just to to something different in occasional situations.
  5. Hi Ronald, Yes, Annotation Transformers (AT) are for special cases. They are a good example of a feature that very few people will use, but when you need it, you'll be happy you have it. AT's are especially useful when you need to make a sweeping change to many tests scattered in many different places and you don't have any static information to pick these out. TestNG already gives you plenty of ways to pick test methods in various ways (groups, regexps on the class/test/method names, BeanShell, etc...) but there are cases where even that is not enough, and the cost of going over your code base and making all these changes manually (as suggested elsewhere) is prohibitive. Especially when it's a temporary change that you intend to revert in a couple of weeks. Another example is when the value for the annotation cannot be computed statically. For example, imagine that you want to calculate the timeOut value of your test methods at runtime, because it will depend on various factors (an IP address you are trying to contact, the load of the current machine, etc...). In this case, you would specify a placeholder timeOut() value in the annotation but replace it at runtime with a dynamically computed value. Does this make sense? -- Cedric
  6. Another example is when the value for the annotation cannot be computed statically. For example, imagine that you want to calculate the timeOut value of your test methods at runtime, because it will depend on various factors (an IP address you are trying to contact, the load of the current machine, etc...). In this case, you would specify a placeholder timeOut() value in the annotation but replace it at runtime with a dynamically computed value.

    Does this make sense?

    --
    Cedric
    Thanks for the response Cedric, this is a very nice example indeed, and i think it does clarify it for alot of people.
  7. Congratulation! I'm a fun of TestNG, IMHO the best java test library!
  8. a simple real use case?[ Go to top ]

    congrats and thx for the new release, cedric. i read about the transformers on your weblog the other day and still can't think of a use case where it isn't much simpler to just write another test... cheers, -- eokyere :)
  9. Yes, congratulations Cedric et al! What a great product! Keep up the good work!
  10. I'm still on JUnit...[ Go to top ]

    Congratulations on the release. TestNG is a terrific product, and I think it's on track for replacing JUnit as the best testing framework for Java. (JUnit 4 is a joke.) However, I thought I'd post why I'm still using Junit 3.8.1 in the hope that TestNG will fill the gap for me. The primary reason is that the JUnit 3.8.1 execution framework is predictable, simple and well-designed, making it an easy thing to decorate and extend. I know that TestNG boasts a "powerful execution model," and that may be true, but it's at the cost of simplicity and tractability. In the end, I understand how my tests are going to be bundled and executed because of the single entry point, the suite() method on the TestCase for the entire suite and the runBare method on the TestCase for a single test method. This will be the case regardless of whether the tests are executed from the command line, from ant, or from my IDE. In contrast, TestNG's execution model is complex enough that it's hard for me to reliably predict how a given test method is going to be invoked. What if the test method is invoked with a different testng.xml file? Will it be excluded? Will the parameters be supplied correctly? Will its group be executed in the right order? And it's also hard for me to reliably understand the differences between invoking it from, say, ant as opposed to my IDE. So, for example, say I want all tests that are in a certain group to be executed wrapped within some logic (e.g. JAAS authentication "doAs" or something) but I also want to be sure that if a single test method is invoked outside of the context of the group (e.g. from my IDE) that it will be wrapped also. There's no good way to do that in TestNG because of the complexity of the execution model. I can do that from a method-level with the IHookable interface, but there's no way for me to do that reliably from the level of a test case, the level of a test group, or the level of a test suite. Now, granted, it's not seamless in JUnit; I have to implement the suite() method and override the runBare() method, but at least its reliable, predictable and functional.
  11. Re: I'm still on JUnit...[ Go to top ]

    Hi Ryan, and thanks for your interesting points. In my experience, there are several reasons why people stick with JUnit3 instead of switching to TestNG, among which: 1) They have a huge JUnit code base and a lot of knowledge accumulated in the team over the years, and migration would be too expensive. 2) They are happy with JUnit. In my experience, 1) is predominant and 2) is extremely rare :-) Interestingly, you seem to belong to the second category. One of the main strengths of TestNG compared to JUnit is its processing model, and it's clear to me that it's exactly the feature you don't need. If you're happy with the way JUnit runs your tests, then by all means, there is very little reason for you to switch. If you take a look at the archives of the mailing-list, you will see there are very few messages to the effect of "I don't understand why this method is being run then", which makes me think that despite its apparent complexity, TestNG's execution model is both predictible and intuitive. Of course, you are welcome to call me biased on that one :-) Your point about the testng.xml is a good illustration of what I'm trying to say. My vision of a testing environment has always been that it should be made of two very clearly distinct part: the business logic (the "static" part: your test classes) and how the tests are run (the "dynamic" part). I *want* to be able to change the dynamic part without having to touch anything to the static part. If I want a colleague of mine to reproduce a certain failure, I don't need to ask them to check out a few tests, rename a few methods or change the suite() method: I can just email them the testng.xml. I've always disliked the way JUnit conflates these two aspects, thereby forcing you to alter your Java code (recompiling) whenever you want to run different tests. You, on the other hand, seem to be comfortable with this approach, nothing wrong with that. Another illustation of JUnit's weak running model is that as soon as you want to do something simple and yet not supported, such as run Before/AfterClass initialization, you are forced to get down and dirty and start doing things such as implementing TestSetups, decorators, overriding runBare(), etc... This is not only unfriendly, it makes it hard to share your tests with your team. Anyway, as I said, I totally understand your position and at the moment, it doesn't look like TestNG is a good fit for you. Hopefully, at some time in the future, you will start needing some of the TestNG features and I'll be happy to help you in the transition then :-) -- Cedric
  12. Re: I'm still on JUnit...[ Go to top ]

    You've got some interesting points, and I'm glad to understand your testing paradigm a bit better. Maybe I'm just inured to JUnit's paradigm, but I've never had any problems with sharing my tests with the team. Whenever a bug is discovered, I just write a new test method that exposes it. As bugs are exposed and fixed, you get more test coverage as you write more tests. The most useful features I recognize with TestNG in a collaborative engineering environment include the ability to group tests and run certain groups, declaring dependencies for test methods, and the ability to disable tests so you can check-in a disabled test that exposes a bug without the fear of breaking the build. I also find the ability to distribute tests across multiple machines very attractive. Maybe I'm not understanding the collaborative environment you're describing, but it seems to me that if each member of the team is running tests under different configurations, you lose the security that you changes won't break anybody else's tests; they're running under a different test configuration. In other words, your tests lose some effectiveness as regression tests. You make a good point that there aren't many people, if any, who complain about not understanding why a test method is being run the way it does in TestNG. So for me to assert that the execution model isn't predictable or intuitive might not be completely fair. My statements are based on my experience in trying to use TestNG to do what I need to "logic wrap" a group of tests. Using IHookable for a single method worked fine. But then I wanted to wrap once the execution of a whole group of tests. The only way I could see to do that was to extend the TestRunner somehow and it got all messy and I retreated in cowardice back to the warm (albeit drafty) security of JUnit. And even if I did extend TestRunner, my IDE wouldn't use the runner that I needed. So I don't see myself moving over to TestNG until you provide some mechanism akin to IHookable for test groups and test suites.
  13. Re: I'm still on JUnit...[ Go to top ]

    So I don't see myself moving over to TestNG until you provide some mechanism akin to IHookable for test groups and test suites.
    I'll go out on a limb and claim that your inclination to try to solve this problem by extending the runner or implementing an interface of the extension API is a JUnit stigma, where it's the standard way of proceeding whenever you want to do something that JUnit wasn't designed to do (which happens quite a bit in my experience :-)). But let's be more concrete: can you explain in more details what you need to do exactly? What you want to wrap, and how? I'm curious to understand why you say that TestNG's regular configuration mechanisms (@Before/@After) are not sufficient... -- Cedric
  14. Re: I'm still on JUnit...[ Go to top ]

    Sure. Let's take the example you chose in the documentation for IHookable: "If a test class implements this interface, its run() method will be invoked instead of each @Test method found. The invocation of the test method will then be performed upon invocation of the callBack() method of the IHookCallBack parameter. This is useful to test classes that require JAAS authentication...." Great. Cool. That's exactly what I need at the level of a test method. But let's say the JAAS authentication took up significant resources. (In my case, it's not JAAS authentication, but, like JAAS authentication, I have to wrap my logic in something akin to a doAs method.) Well, if it takes significant resources, when I run all test methods in a suite (or in a group, etc.), I want to do the authentication ONCE and wrap all invocations of the test methods of that suite or group in that logic. This way, I don't have to incur the resource hit every time I execute a test method. Make sense?
  15. Re: I'm still on JUnit...[ Go to top ]

    Great. Cool. That's exactly what I need at the level of a test method.

    But let's say the JAAS authentication took up significant resources. (In my case, it's not JAAS authentication, but, like JAAS authentication, I have to wrap my logic in something akin to a doAs method.) Well, if it takes significant resources, when I run all test methods in a suite (or in a group, etc.), I want to do the authentication ONCE and wrap all invocations of the test methods of that suite or group in that logic. This way, I don't have to incur the resource hit every time I execute a test method.

    Make sense?
    What stops you from storing the result of the authentication in a static field the very first time it's invoked, and then simply reuse that result every time the wrapping method is invoked? -- Cedric
  16. Re: I'm still on JUnit...[ Go to top ]

    Yeah, I actually tried that... or something like that. Like I said, in my case it's not JAAS authentication. The problem in my case is that it's not calculating the *result* that takes up significant resources, it's the doAs method itself--the wrapping logic. I'll be the first to admit that my case is pretty unique. The real example is that my test cases need (at least most of them) to be invoked within the Annotation Processing Environment (set up by Sun's new Java 5 Annotation Processing Tool used for preprocessing source code). Basically, I'm writing an AnnotationProcessorFactory that I need to test. It's big--it's my whole project, so there are a significant number of tests. Well, APT is invoked from the command line with the FQN of my AnnotationProcessorFactory passed as a parameter. It then sets up the environment and calls the process() method of my factory. The factory uses the Mirror API to iterate through all the source-level declarations and types to do its work. So the overhead is with the invocation of APT and the establishment of the APT environment. It's actually not too bad, but intensive enough that it makes a significant difference if it's done for every test method. I actually tried to save off the environment in a static variable, but as it turns out, the declarations and types are lazy-loaded at runtime, so it's useless to try to use the environment outside the APT context. I apologize for making a stink about such a small corner-case. I don't expect you to solve all my problems, Cedric :-). I was just disappointed that I wasn't able to use TestNG, and I thought I'd share my experience. Like I said, I think TestNG is a terrific product, it just didn't fit my needs in this case.