Discussions

News: Test Framework Comparison

  1. Test Framework Comparison (69 messages)

    Currently, JUnit remains the de facto standard for unit testing in Java. Justin Lee has written up a comparison between JUnit, JTiger, and TestNG, focusing on the differences between them in simple cases to illustrate when and where each might be used, and highlighting the strengths and weaknesses of each.

    Read Test Framework Comparison

    Threaded Messages (69)

  2. JUnit[ Go to top ]

    I like a lot the standard that JUnit has defined.
  3. Test Framework Comparison[ Go to top ]

    It is sad that JUnit had been neglected for some time by its creators. Now Kent and Erich seem to wake up, but they had to do so a year ago, when JSE 5 release candidate appeared. As a result we have today a several unit-test frameworks with similar JDK5-oriented functions and I honestly have no idea which better to choose for future projects.
  4. Test Framework Comparison[ Go to top ]

    JUnit will remain the standard framwork for unit tests. JTiger and TestNG will be obsolete and even more niche than now when JUnit 4 is finished.

    It was good that the JUnit 4 developers have delayed the improvements. Now they see the options for improvement more clearly than before because they can learn from the good and the bad things in JTiger and in TestNG and make it better then the two other testing frameworks.

    JUnit is not just a standard, its a masterpiece. You can see it in nearly every detail. Of course there has been room for improvements. But thats why they do JUnit 4 now, the next masterpiece. I really like to work with such a high quality piece of software!

    I prefer @ignore over TestNG's not so readable "enabled = false". I don't like TestNGs "XML way", although its optional. There is too much XML crap out there already. I prefer JUnit's readable @Before over TestNG's not so readable @Configuration(beforeTestMethod = true).

    Grouping of tests is okay. But TestNGs subgroup feature goes too far. I fear that you can easily create a complicated mess by using it. There is another feature of TestNG which I don't like. You can annotate that test method b() is dependent on test method a(). This sounds good initially. But I fear that administrating such dependencies is confusing, too much work and that it will be very error prone. I have no problem with seeing that 100 tests failed, even if only 1 test failed and 99 tests failed only because of the first ones failure. Usually a developer finds quickly the problem by looking at any failed test. So it is not a real value, just a cosmetic thing (but for a too high price).

    Why some people have a problem to prefix a test with the word "test"? Its the most natural thing to call a test method "testBalance", "testCalculator()" or whatever(). Its more expressive than naming it just "balance()" or "calculator()". Same with setUp() and tearDown(). I never mistyped it, so its no real world problem that the names have been fixed.

    I will miss it that the distinction between failure and error will be removed in JUnit 4. If an error is reported, I know that it is a serious problem. If a failure is reported, then it is usually not such a serious bug. On the other hand jTiger distinguishes between Ignored (Annotated), Ignored (Cannot Invoke), Failure (Set Up), Failure, Failure (Tear Down). THATS obsolete.

    A framework is also the art of omitting dangerous or obsolete features. TestNG doesn't seem to master this art in my eyes. It is a feature overkill.

    Now a word to @ExpectException. I had no problem to use the try-catch-block for testing exceptions. I think it is more readable than using @ExpectException. By using a try-catch-block, I am more free to organize the testing in a test method. And I can test as many exceptions as I want in a test method. But by using the annotation I can only test one exception per test method.

    I never need to log anything in a test. In my eyes, logging is obsolete for tests. Either you use loggings or you use tests with assertions, but not both. Who will bother to read all the logging messages? If I see a green bar, I am not interested in reading any logging messages of a test. And if its a red bar, then I want to read the repored failure message and not any loggings.

    I think that JUnit 4 will let you rerun only the failed tests, but I am not sure. Would be nice I think. I hope that they will add "assertNotEquals(...)", because I am missing it.
  5. Test Framework Comparison[ Go to top ]

    I've used TestNG a bit more extensively on another project or two and it's pretty nice. I find I prefer JTiger's annotations to TestNG's for things like ignoring/disabling tests. But only ever so slightly. TestNG is a pretty solid framework though. It's nice to work with though it seems to have issues with abstract classes in the hierarchy. I'll be looking into that soon and seeing if I can't rectify that problem.

    JUnit 4 will defintely remove some of the reasons for moving to one of these others. But there's no release date for JUnit 4 as yet and my queries in that regard went unanswered (though the developers were more than helpful in other areas.) Though, to be honest, when JUnit 4 *does* ship porting your tests to it would be about as much work as porting them to either of these other two. JUnit 4 probably will remain the "standard" but largely through inertia I think.
  6. Test Framework Comparison[ Go to top ]

    I think there are a lot of large real world projects which don't use Java 5 yet. So they couldn't use JUnit 4 now, even if it was finished. For the other projects its worth to wait I think.

    It is usually a bad thing to use a niche product instead of a marketleader, especially if both are free and when there are no big differences beween them.

    I hope that the simlicity of JUnit 4 will not be decreased by "cool new features".

    The best practices for these cool new features are unknown.
  7. Test Framework Comparison[ Go to top ]

    I think there are a lot of large real world projects which don't use Java 5 yet. So they couldn't use JUnit 4 now, even if it was finished.
    This is the reason why TestNG provides both a JDK5 and JDK1.4 version: users were pretty clear on the fact that a lot of them wouldn't be able to switch to JDK5 before a long time. The adoption rate of TestNG increased dramatically after we made the JDK1.4 version available.

    --
    Cedric
    http://testng.org
  8. Test Framework Comparison[ Go to top ]

    It's nice to work with though it seems to have issues with abstract classes in the hierarchy.
    You definitely pique my interest here... Please email me privately and let me know what is going on.

    Great job on the article!

    --
    Cedric
    http://testng.org
  9. Test Framework Comparison[ Go to top ]

    "JUnit is not just a standard, its a masterpiece. You can see it in nearly every detail. Of course there has been room for improvements. But thats why they do JUnit 4 now, the next masterpiece. I really like to work with such a high quality piece of software!"

    A masterpiece?! You're expectations for a masterpiece must be much lower than mine. JUnit is nowhere close to a masterpiece. It's got many problems and that's why you have things like TestNG and JTiger. If JUnit didn't have all these flaws, these other frameworks simply wouldn't exist. Your comments are an insult to the developers of these frameworks. I can't believe you even go on to reward the JUnit folks for sitting around and doing nothing so they can wait and see what they can learn (read: steal) from people with apparently a much greater imagination.
  10. Test Framework Comparison[ Go to top ]

    A masterpiece?! You're expectations for a masterpiece must be much lower than mine. JUnit is nowhere close to a masterpiece. It's got many problems and that's why you have things like TestNG and JTiger. If JUnit didn't have all these flaws, these other frameworks simply wouldn't exist. Your comments are an insult to the developers of these frameworks. I can't believe you even go on to reward the JUnit folks for sitting around and doing nothing so they can wait and see what they can learn (read: steal) from people with apparently a much greater imagination.
    Believe it or not, nearly none of my testing problems have not been due to JUnit 3.8.x.

    The new features of JUnit 4 are nice, I like them. And I still like JUnit 3.8.1. Its a very simple (not too simple) general purpose testing framework.

    I don't want to insult the schismatic ones. ;-)

    The people who try to replace JUnit with their own testing framework have "stolen" much more ideas than JUnit 4 has "stolen" from them. Besides this, ideas can be born at two places at the same time without any "stealing". I don't believe in the idea of intellectual property.
  11. Test Framework Comparison[ Go to top ]

    "The people who try to replace JUnit with their own testing framework have "stolen" much more ideas than JUnit 4 has "stolen" from them."

    Only in the sense that they didn't invent this variety of unit testing in the first place. TestNG is nothing like JUnit at all except for the fact it has asserts (and even those are handled very differently).

    "I don't want to insult the schismatic ones. ;-)"

    lol...that's the first time I've been called a schismatic. I don't deny it though. In truth, I'm simply brutally utilitarian about software. JUnit is lacking in a lot of utility at this time so in my mind it is inferior and less worthy than its competitors. It's that simple for me.
  12. Test Framework Comparison[ Go to top ]

    "lol...that's the first time I've been called a schismatic. I don't deny it though. In truth, I'm simply brutally utilitarian about software. JUnit is lacking in a lot of utility at this time so in my mind it is inferior and less worthy than its competitors. It's that simple for me.
    I didn't meant you.

    As I said, and I repeat it, 99% of my testing problems have not been due to JUnit 3.8.x. I cannot say anything about your experiences, just about mine.
  13. Test Framework Comparison[ Go to top ]

    What about all the extensions available on top of junit. For example DBUNIT to test databases. I think the wealth of these extensions is the biggest insertia point for adoption of the new Frameworks like TestNG. I am sure lot of people would just wait for JUnit 4.

    Also according to the artical, there is no need to rewrite the existing tests and they can use existing frameworks and extensions in JUnit 4 as well.
  14. Test Framework Comparison[ Go to top ]

    What about all the extensions available on top of junit. For example DBUNIT to test databases. I think the wealth of these extensions is the biggest insertia point for adoption of the new Frameworks like TestNG.
    We looke at the most popular JUnit extensions and quickly reached the conclusion that using them in TestNG is so easy that it doesn't even require work on our part.

    Basically, most of these frameworks integrate with JUnit by supplying their test base class that you need to extend inste of JUnit's TestCase. And the only reason for this is usually because this class overrides setUp() and sometimes tearDown().

    All you need to do to reuse these frameworks on TestNG is annotate these methods with @Configuration and you're ready to go (we verified this with DBUnit and HTTPUnit).

    --
    Cedric
    http://testng.org
  15. This may be a bit off-topic, but there is a feature which I personally miss in, so it seems, both of these frameworks.

    In many of the cases I've seen, the testing of some functionality relies on some external data. A simple example would be validation of URLs according to some internal rules (e.g. certain parameters exist in combination of others).

    What one (well, I...) would like to have is a short testing code, combined with some meta-data file with a description of inputs and outputs. Instead, I keep seeing people copying and pasting test methods which do the same thing, but with different inputs. Of course you can build your own input readers, but then you lose the beauty of a framework. The result is that people use painfully small subsets of inputs as possible test cases - specially if the testing code that runs them changes over time.

    Any thoughts about such functionality?
  16. You surely will get answers about it in the official JUnit Yahoo Group: http://groups.yahoo.com/group/junit/

    I'm not sure if it does apply to your question, but maybe this helps you:

    public void testSomething()
    {
      ...
      ExpectedOutput eo = new ExpectedOutput();
      eo.setResult1...
      eo.setResult2...
      eo.setResultN...
      ReplacementTime rt = rtCalculator.replacementTime(article1);
      doTestSomething(rt, eo);
      ...
      eo = ...
      rt = rtCalculator.replacementTime(article2);
      doTestSomething(rt, eo);
    }

    private void doTestSomething(ReplacementTime rt, ExpectedOutput eo)
    {
      // assertments against rt, expecting the results to match eo
      ...
    }

    No need for a meta file I think. J.B. Rainsberger suggested this way of testing for certain situations in his book "JUnit Recipes".
  17. Re: Test Framework Comparison[ Go to top ]

    I've often used this approach. It's very very handy. You could also go with an (abstract) parent class that has the actual test methods ( doTestBlah() ) and just subclass it and pass in data. I've done this on a large system (~4000 classes) and it's quite nice.

    To answer the earlier question, though, TestNG has an @Factory annotation that might provide what you're looking for. See http://www.testng.org/main.html#factories for more details.
  18. Re: Test Framework Comparison[ Go to top ]

    Yes, it looks like TestNG and @Factory is what I was looking for. I'll look into it; thanks.

    The JUnit solution that was suggested will work, but it looks to me like working around the framework instead of having a framework. You still need to repeat the testing logic as code (although a wrapper code). You don't get the strengths of reflection and management - so what's the point?

    I assume that any non-trivial tests (and I do refer to those tests that go beyond the testing of a single class) will still need some mockups and serialize/desrialize of inputs and outputs, and that most of this functionality cannot be generalized. Still, a built-in interface in the framework for them would be nice.

    My solution was indeed to exetend JUnit by havine a subclass of TestCase that would read inputs/outputs from an external xml, and be a factory of TestCases. It works fine, except that again - it felt like fighting against the framework instead of complementing it.
  19. Yes, it looks like TestNG and @Factory is what I was looking for. I'll look into it; thanks.
    Please let me know if you need help.
    My solution was indeed to exetend JUnit by havine a subclass of TestCase that would read inputs/outputs from an external xml, and be a factory of TestCases. It works fine, except that again - it felt like fighting against the framework instead of complementing it.
    Indeed. I find that too many questions on how to achieve a certain thing with JUnit are answered by "Write your own TestDecorator" or "Replace JUnit's TestRunner with your own".

    Hopefully, you will find that TestNG requires you to look a lot less under the hood and simply use the framework as documented...

    --
    Cedric
    http://testng.org
  20. It works fine, except that again - it felt like fighting against the framework instead of complementing it.

    This is precicely the reason I switched to TestNG for authoring tests that go over a scope of dynamic webpages. I'm looking for acceptance/functional testing, and jUnit forced me to override too many methods to "feel" correct. And even with the hack, I couldn't get a report that was correct.

    TestNG made it easy for me to accomplish what I needed to.
  21. Re: Test Framework Comparison[ Go to top ]

    I'm looking for acceptance/functional testing,
    Why don't you take a look at Ward Cunningham's FIT (Framework for Integrated Test)? It was designed for that purpose.

    Now it is easier to learn because there is a book about it: http://www.amazon.com/exec/obidos/tg/detail/-/0321269349/qid=1121790281/sr=1-1/ref=sr_1_1/002-1283728-0940010?v=glance&s=books
  22. Re: Test Framework Comparison[ Go to top ]

    I forgot to write the link to FIT: http://fit.c2.com/
  23. Re: Test Framework Comparison[ Go to top ]

    I'm looking for acceptance/functional testing,
    Why don't you take a look at Ward Cunningham's FIT (Framework for Integrated Test)? It was designed for that purpose.Now it is easier to learn because there is a book about it: http://www.amazon.com/exec/obidos/tg/detail/-/0321269349/qid=1121790281/sr=1-1/ref=sr_1_1/002-1283728-0940010?v=glance&s=books

    It can't do what I need to do. I looked at it...won't work, not dynamic enough.
  24. Re: Test Framework Comparison[ Go to top ]

    I looked at it...won't work, not dynamic enough.
    What do you need to do?
  25. Re: Test Framework Comparison[ Go to top ]

    I looked at it...won't work, not dynamic enough.
    What do you need to do?

    To answer your question. I have two problems with the same data, and two different things I need to test

    I need to test that all web pages loaded in the application contain valid links. This cannot be done via a spider because the URLs are encoded. So I login to the database and generate URLs for a specfic run of data. I then can use HtmlUnit to go out and test each individual page to make sure they have valid links (or additional things like valid page headings, etc.). All I provide is the id of the run I need to test.

    In addition, I want to do the same sort of test on the actual objects that are used to generate the pages.

    The difference with all of this is that, I'm not looking to test the object itself, I'm looking to test the contents of that object to make sure it's valid for the application being written.

    This is not a usual test scenario that one usually thinks of when they think of JUnit. In fact, I've been told on many occasions that it's improper to test the way I test. Yet, when pressed, those folks cannot come up with an actual feasable solution for my issue. I often work with clients who do not know what the boundaries are for their system, and cannot find out. Hence I need to test any assumptions used to parse the incoming binary streams and make sure they are valid throughout the development process.

    In essense, it doesn't really matter what I need FIT to do, since TestNG seems to be working wonderfully for me and took me 2 days to get up and running after I spent a good month reading on the Yahoo Groups JUnit board trying to find ways to handle it. And I found ways to handle it...ParameterizedTestCase was one, but even that needed to be hacked, since it did not solve the problem of running out of memory. Storing 2000 objects in memory was the limit, and I often need to regression test 100,000 or more.

    TestNG didn't have the memory limitation and as such was the winner for my problem. Nothing against JUnit, but it couldn't do what I needed, and I've often found you create more problems trying to cram a square peg into a round hole, especially when you've got a round peg lying around in your toolbox.
  26. Re: Test Framework Comparison[ Go to top ]

    In addition, I want to do the same sort of test on the actual objects that are used to generate the pages.

    The difference with all of this is that, I'm not looking to test the object itself, I'm looking to test the contents of that object to make sure it's valid for the application being written.

    I don't understand that although I tried hard. You seem to already test the resulting HTML pages, which is a kind of acceptance test. Additionally you want to unit test the objects which generated the web pages by testing the contents of them?

    What is the difference to you between "testing the object itself" and "testing the contents of the object"? Do you want to examine them by using reflection or what?

    Why it is not enough only to test the resulting HTML pages? Or why it is not enough to only test the HTML generating objects? Testing is a trade-off because the time is limited and the tests are endless.

    The contents of the objects are simple to test, aren't they? Just call their getters and compare the result with your expectation. Can it be so easy to solve your problem?

    What specific feature of TestNG helps you for solving that problem which I still don't understand? Maybe you can show me some code of your TestNG solution so that I can understand what you mean? Or maybe another reader of this forum can tell me what Mike's problem is?
  27. Re: Test Framework Comparison[ Go to top ]

    I don't understand that although I tried hard.

    That's OK, it is a difficult scenario to explain without violating corporate secret type things.
    You seem to already test the resulting HTML pages, which is a kind of acceptance test.

    The HTML pages are generated by a container object that contains very, ah, flexible objects inside. Rather than using objects that specify exactly what they are by their name, these specify what they are by attributes. (I didn't pay attention in patterns class :P)

    So I need to test that for the given set of objects for the particular set of data, that the HTML is valid. The pages for each user will be very very different. It's not enough to say, oh, mock up a container object with your high and low limits, because 6 years of experience of doing this for my clients has taught me that the limits given will be broken within 6 months because of changes on the client side.

    Essentially, I cannot trust the documentation that says "Here are the limits".
    Additionally you want to unit test the objects which generated the web pages by testing the contents of them? What is the difference to you between "testing the object itself" and "testing the contents of the object"?

    Because the objects are flexible it makes it more challenging. For example, I might have an address container which contains address lines. The program that builds the container parses some sore of binary image file to grab this data.

    So how do I make sure I get the address? what if the piece of the image moves? How can I regression test this parsing rule without eyeballing it?

    I test the object AFTER parsing it. I create my own acceptance test that says, I expect to have say 4 lines in my address.

    What can happen, and has happened, is I find I've gotten 10 lines! Uh oh! So now, I can go back, look at the specific input stream and figure out what needs to be changed, on my end to make it work...the client isn't going to change.
    Testing is a trade-off because the time is limited and the tests are endless.
    Which is exactly why this had to be automated. And exactly why JUnit could not work. If it's memory limit due to object storage is 2000, it's not going to be efficient. I need to leave for the day, set up a script and start testing on today's runs. Sure it's not as efficent
    The contents of the objects are simple to test, aren't they? Just call their getters and compare the result with your expectation. Can it be so easy to solve your problem?

    I hope I've been able to show that it isn't so easy. It's not a Unit test, because the underlying objects i've used have been tested...the getters and setters work. The contents of the object are what I need to test and make sure that they follow business rules for each specific client's input stream.
    What specific feature of TestNG helps you for solving that problem which I still don't understand? Maybe you can show me some code of your TestNG solution so that I can understand what you mean? Or maybe another reader of this forum can tell me what Mike's problem is?

    TestNG allowed me to replace a bunch of hacked JUnit code (where I overloaded 50% of the reflection to properly name my tests), with a @Factory annotated method:

    @Factory( parameters = { "testClass", "runID", "limit", "JDBCURL", "DBUser", "DBPassword", "URLPrefix", "URLSuffix", "CipherAlgorithm", "CipherKey", "voucherString"} )
        public Object[] factory(
                String testClass,
                String runID,
                String limit,
                String JDBCURL,
                String DBUser,
                String DBPassword,
                String URLPrefix,
                String URLSuffix,
                String cipherAlgorithm,
                String cipherKey,
                String voucherString) {
            List<Object> fixtureList = new ArrayList();
            List objList = getObjList(runID, JDBCURL, DBUser, DBPassword, URLPrefix, URLSuffix, cipherAlgorithm, cipherKey, voucherString);
            int intLimit = 0;
            try {
                intLimit = Integer.parseInt(limit);
            } catch (Exception e) {
                intLimit = 100;
            }
            try {
                Class theTestClass = Class.forName(testClass);
                Class[] args= { String.class, String.class };
                Constructor theTestConstructor = theTestClass.getConstructor(args);
                Utils.log("TestFactory2", 2, "Adding test instances");
                for (int j=0; j < objList.size() && j < intLimit; j++) {
                    fixtureList.add(theTestConstructor.newInstance(new Object[]{(String) objList.get(j), (String) objList.get(j)}));
                }
            } catch (Exception e) {
                e.printStackTrace();
            }
            Utils.log("TestFactory2", 1, "Returning "+ fixtureList.size() +" test objects from factory method and beginning tests.");
            return fixtureList.toArray(new Object[fixtureList.size()]);
        }
        

    I hope that's readable. This is for the HTML test.

    Each test object takes a URL and an ID. They then use a setUp with @Configuration(beforeTestMethod=true) to resolve the issue of OutOfMemoryError that JUnit provided. Since it only loads each container before each testMethod. Takes a few extra hits to the J2EE container, but that's not a big concern.

    Then each @Test() method just uses the container object to test whatever it needs to test.

    The other advantage of this, is that it can also be used to troubleshoot production issues without having to pull data down to an alpha environment. For example, if a problem occurs, with a specific type of end user, I can quickly setup a test to determine how many customers are impacted by the actual data in the database, rather than (yet again) have the client overstate the problem as the end of the world.

    In essence, the flexibility of TestNG solves my unique problems. And I have a feeling JTiger would also solve these same types of issues...I just ran into TestNG first and started trying it out. And it's a really nice thing to have in my toolbox for the situations where JUnit just isn't the correct solution.
  28. Re: Test Framework Comparison[ Go to top ]

    TestNG allowed me to replace a bunch of hacked JUnit code (where I overloaded 50% of the reflection to properly name my tests), with a @Factory annotated method
    That's some seriously cool stuff, Mike!

    And a great example of a framework providing flexible features (parameters and factories) that get used by developers in ways that I had not anticipated...

    --
    Cedric
  29. Re: Test Framework Comparison[ Go to top ]

    Hi Mike,

    right now my life is a little bit complicated so that I cannot think about the problem with full power.

    I will try to provide a JUnit solution as soon as possible.

    Bye,

    Hans
  30. Re: Test Framework Comparison[ Go to top ]

    Hi Mike,right now my life is a little bit complicated so that I cannot think about the problem with full power.I will try to provide a JUnit solution as soon as possible.Bye,Hans

    I don't need a JUnit solution. My current one works just fine. Thanks though.

    If it ain't broke, don't fix it. :)
  31. TestNG IntelliJ plug-in[ Go to top ]

    By the way, I'm happy to announce the first release of the TestNG IntelliJ plug-in.

    --
    Cedric
    http://testng.org
  32. Hi Efri,

    there are a few extensions of JUnit on Sourceforge that try to fill the topic of providing multiple input/output data for the same testing code block.

    ddtunit.sourceforege.net
    jtestcase.sourceforge.net
    jetif.sourceforge.net

    Have a look.

    best regards

    Jörg Gellien
  33. Test Framework Comparison[ Go to top ]

    I work on systems with millions of lines of code and equally large architectures. I started years ago using JUnit and the like and have been very disappointed in their ability to test anything but low level class tests. It seems difficult or impossible to test things like:

       * When the result of a test is an external message
       * When the result of a test is a DB update
       * When the result of a test is an operator message
       * When the client of the test is several layers of
         indirection removed from the code being tested
       * When the requirement being tested spans JVMs, machines,
         programming languages, etc.
       * When the system you're testing has large amounts of
         legacy code that aren't suitable for NUnit testing.

    If you work hard enough and have flexible factory based implementations that can be easily stubbed out, then you can use JUnit or something like it for automated testing. However, it comes with a lot of expense and in the end doesn't test the system in a realistic manner.

    How do you write a JUnit test that says "process a group of related messages that are sent in real time over MQ and another proprietary protocol that gets passed around 10 different servers on 3 different machines, for both Java, C++, Fortran, and C, and verify that 4 products were produced for 10 different customers"?


    Because I'm a fan of automated testing, I've adapted the JUnit mindset to a proprietary testing framework that accounts for all of these problems. I've decoupled the assertion of expected results from the assertion of actual results so that two different processes can independenly make these assertions into a common DB table. Because all share a common DB interface, it is open to all languages for updates and report generation. The definition and scope of the automated test is wide open.

    If anyone else has struggled with this and has any insight, I'd love to hear about it. I can't believe I'm alone in the universe.
  34. Test Framework Comparison[ Go to top ]

    I work on systems with millions of lines of code and equally large architectures. I started years ago using JUnit and the like and have been very disappointed in their ability to test anything but low level class tests.
    Well, JUnit is a unit-testing framework, you're not supposed to test anything but the internal workings of a single class. If you ever end up testing more than one class, JUnit is probably no longer the right tool.

    Having said that, the problems you list (which are quite real) go way beyond what a testing framework cover, especially when it comes to trying to test legacy code that not only was not designed for testing, but might actually be written in a language other than Java...

    --
    Cedric
    http://testng.org
  35. Test Framework Comparison[ Go to top ]

    Well, JUnit is a unit-testing framework, you're not supposed to test anything but the internal workings of a single class. If you ever end up testing more than one class, JUnit is probably no longer the right tool.

    Yeah, but JUnit also seems to work as a component-level testing framework on many projects too. Especially if the component has been set up for easy testing: Mock objects available for dependencies.

    I found it helpful to have a separate package (suite) of tests for each "component" that could be considered "black-box" component tests.

    As long as you don't need to go through too many hoops, JUnit will also work well for other kinds of tests. Plus you get the benefits of having only one testing framework to master.

    As always, it depends on the project, right?
  36. Test Framework Comparison[ Go to top ]

    I work on systems with millions of lines of code and equally large architectures. I started years ago using JUnit and the like and have been very disappointed in their ability to test anything but low level class tests. It seems difficult or impossible to test things like: ...
    Maybe with legacy code which was NOT developed in a test driven way, there is a problem of testing it. As Rainsberger wrote, it is usually a design problem, not a testing problem if something can't be tested or can't be easily tested.

    Private methods never should be tested. The time wasted for it and the benefit you get from it is in no healthy proportion. Only test public methods. It amazes me that such a smart man like Dave Thomas suggests to make the private methods public in order to be able to test them (see tech talk). Don't do it.

    Back to your hard to test code. I suggest you improve the design of the classes to test little by little in order to be able to test it easily.

    Its one of the most important things in softare business that automated testing becomes easy. Thats why there are things like Spring and Dependency Injection. Favor them whenever you can over not so testable technical foundations or designs. If the software to develop is not easily testable and your development environment doesn't provide fast feedback (like continuous integration) it is likely to result in a death march project (a little bit off topic now...)
  37. Test Framework Comparison[ Go to top ]

    Private methods never should be tested. The time wasted for it and the benefit you get from it is in no healthy proportion. Only test public methods.
    I disagree. Sometimes, testing private methods allow you to catch failures much faster.

    Imagine that you are writing a Swing table widget that lets you order your columns in several different ways. Your sorting algorithm is private and one day, you introduce a bug in it.

    If all you do is test the public methods, the widget will simply start showing wrong results and your tests won't tell you anything more. For all you know, there might be a bug in your Swing code.

    If you unit test your (private) sorting algorithm, you will catch the error right away instead of wasting it "higher up the stack".

    This kind of scenario has happened to me countless times, to the point that my rule of thumb is more: "if it can break, test it".
    It amazes me that such a smart man like Dave Thomas suggests to make the private methods public in order to be able to test them (see tech talk). Don't do it
    If it can break, do it. Who care if you make a few methods public, seriously? The quality of your code is certainly worth the slight loss in encapsulation you will suffer.

    If it really bothers you, add a comment to this method or put it in an "internal" package or make it a static private inner class, but don't turn away from testing just for dogmatic reasons.
    Back to your hard to test code. I suggest you improve the design of the classes to test little by little in order to be able to test it easily
    Interestingly, this advice actually contradicts your point: if you want to test little by little, you will most likely start by testing private methods, since they are the founding block of your code. And eventually, you will "move up the stack" and start testing public methods.

    --
    Cedric
    http://testng.org
  38. Test Framework Comparison[ Go to top ]

    If it can break, do it. Who care if you make a few methods public, seriously? The quality of your code is certainly worth the slight loss in encapsulation you will suffer.

    If it really bothers you, add a comment to this method or put it in an "internal" package or make it a static private inner class, but don't turn away from testing just for dogmatic reasons.

    I disagree with this. I do a lot of API programing, and being able to properly encapsulate libraries is very important. Violating my object design is not an option. In general you can write smart test cases that use your public methods in ways that allows you to properly test you private methods. Sometimes you can also subclass the class you want to test, and create public methods that are only used to test private methods. I do this sometimes when I feel it is important to test directly against the private methods.

    Cedric, do you honestly feel that it is OK to make everything public just for ease of testing?
  39. Test Framework Comparison[ Go to top ]

    I disagree with this. I do a lot of API programing, and being able to properly encapsulate libraries is very important. Violating my object design is not an option. In general you can write smart test cases that use your public methods in ways that allows you to properly test you private methods. Sometimes you can also subclass the class you want to test, and create public methods that are only used to test private methods. I do this sometimes when I feel it is important to test directly against the private methods.
    Fair points. It's a dangerous line to walk, but I dislike "never test private methods" just as much as "violating my object design is not an option": these are extreme positions that do nothing but preventing you from using your judgment and your experience for each situation.

    Everything becomes an option at some point :-)

    Another point you hint at is that these decisions are different depending on whether you are testing an application or a library.
    Cedric, do you honestly feel that it is OK to make everything public just for ease of testing?
    That's a delicate topic. I have a few random thoughts right now, give me a few days and I'll post a more thorough blog entry (I already just posted one: http://beust.com/weblog/archives/000303.html).

    --
    Cedric
    http://testng.org
  40. Test Framework Comparison[ Go to top ]

    It's a dangerous line to walk, but I dislike "never test private methods" just as much as "violating my object design is not an option": these are extreme positions that do nothing but preventing you from using your judgment and your experience for each situation.

    Very true, I totally agree with you. There are no 'hard fast rules' that trump using good judgment and experience to apply to a specific situation.
  41. Test Framework Comparison[ Go to top ]

    Hi Cedric,

    I understand your objections about not testing private methods, but I have good reasons for it. It would be off topic here to discuss it.

    With "test little by little" I meant that he should refactor the legacy code first, before he tests public methods. My English is not so good.

    Don't get me wrong. You and others did a good job of bringing in ideas to improve JUnit. But please don't try to replace it. No schism please.

    It is so good to have just a single standard testing framework to learn and master and not a babylonic mass of home-grown testing-frameworks. Reuse and extension is the Java way, replacement of standards by "improving" them is the Microsoft way.

    Why not try to "plug in" your extra features into JUnit instead of replacing it? I don't know if the JUnit design allows it. If it doesn't allow it, why don't you ask the JUnit developers to redesign it so that everyone can extend it instead of trying to replace it?

    I suggest that you and other unit test framework developers add your extra features on top of JUnit 4 and not replace it with an own framework.
  42. Test Framework Comparison[ Go to top ]

    Hi Cedric,I understand your objections about not testing private methods, but I have good reasons for it.
    That's sad, where would you want to discuss them? Do you have a blog?
     It would be off topic here to discuss it.With "test little by little" I meant that he should refactor the legacy code first
    I think this is already self-contradictory: it is legacy code, refactoring it is not necessarily an option, nor is it even possible sometimes.
    You and others did a good job of bringing in ideas to improve JUnit.
    Thanks for acknowledging it.

     But please don't try to replace it. No schism please.It is so good to have just a single standard testing framework to learn and master and not a babylonic mass of home-grown testing-frameworks.
    As I already said on this forum before: I don't understand this attitude.

    What's wrong with having several implementations based on different ideas and let users decide?
    Why not try to "plug in" your extra features into JUnit instead of replacing it?
    I tried to do that, of course, but it was clear to me it wouldn't work, mostly because parts of the implementation of JUnit are based on design decisions that I precisely wanted to avoid in the first place (such as reinstantiating the test class before each method invocation).

    Besides, JUnit is focused on *unit* testing (testing the internals of a single class) while TestNG covers end-to-end testing, so while the two frameworks have some overlap, they address different problems.

    --
    Cedric
    http://testng.org
  43. Standards DO NOT LAST forever![ Go to top ]

    Standards are defined. Standards are used. Standards are dropped. JUnit is a "standard" test framework right now but that means nothing and DOES NOT prevent people to develop and/or use other frameworks that might become future standards.
    but DO NOT FORGET!...nothing lasts forever.... especially when people get arrogant and think they have defined THE standard that everybody should unconditionally use forever. Thats when the falling of that standard begins....
  44. Test Framework Comparison[ Go to top ]

    That's sad, where would you want to discuss them? Do you have a blog?
    I wrote somthing on why not testing private methods in your blog.
    I think this is already self-contradictory: it is legacy code, refactoring it is not necessarily an option, nor is it even possible sometimes.
    There are different kinds of legacy code. So it depends.
    What's wrong with having several implementations based on different ideas and let users decide?
    What's wrong with standards? Whats wrong extending standards at their "extension points" instead of trying to replace them?

    I have no objection that such a innovative thing like FIT doesn't extend JUnit. It is really something different. It doesn't compete with it at all.

    But TestNG and jTiger will basically be the same thing in green for most developers compared to JUnit 4. The compete with JUnit.

    If you and the writer of jTiger don't cause a testing framework schism, I have no objection if a project teams chooses either TestNG or jTiger instead of JUnit.

    But if it creates a schism (it will not happen, because JUnit 4 will prevent it), then it is a bad thing for the Java platform (like SWT and logging frameworks). People in the Javaworld suddenly need more knowledge to accomplish the same thing. If I find another job, maybe I have to learn TestNG. My knowledge of JUnit will help me, but it would still take time to learn it and to master it like JUnit. In another project maybe I have to learn jTiger, because they use it there. The effort for learning can be minimized by having standards in the Java world. One of such a standard is JUnit. At least all other testing framworks should extend it, not replace it if possible. Standards are good, but should improve themselves so that home-grown things have no chance. Hibernate would never exist and never be successful if Entity Beans wouldn't be a nightmare. Don't get me wrong. The first thing one should do is to suggest the improvements to the developers of the standard. If it takes much to long for them to implement it or if they even refuse to implement it, then it is justified to create something new which competes against it.

    I don't want to argue about it with you. This is just my opinion.
  45. Test Framework Comparison[ Go to top ]

     It doesn't compete with it at all.But TestNG and jTiger will basically be the same thing in green for most developers compared to JUnit 4.
    You seem to forget that a year ago, there was no JUnit 4 (and there is still none today). I started TestNG a year ago with two goals: 1) experiment with new ideas (annotations, dependent methods, test groups, flexible runtime, etc...) and 2) try to shake out the testing community which, frankly, had been asleep for four years.

    One year later, JUnit 4 is in the works, JTiger is out and everybody is talking about testing and thinking of new and innovative ways to apply the new concepts introduced by TestNG into their daily lives. I am very happy to see that, and even happier (and surprised) to see that TestNG has acquired a life of its own.

    I really don't see what's wrong about this situation.
    But if it creates a schism (it will not happen, because JUnit 4 will prevent it)
    Then you have nothing to worry about, do you? Just let us experiment with cool new features and let the JUnit 4 authors pick whatever they like from it and ignore the rest.
     One of such a standard is JUnit. At least all other testing framworks should extend it, not replace it if possible.
    You can clearly see the influence of JUnit on TestNG and JTiger. You can also see how simple the concepts are (methods tagged as tests, configuration methods that wrap around tests, and that's it), so you shouldn't worry about reusing your JUnit knowledge with any of the Next-Generation testing frameworks.

    What you might have to learn is how to use the features these frameworks offer that are not present in JUnit, but that's just standard evolution.
    Hibernate would never exist and never be successful if Entity Beans wouldn't be a nightmare.
    Well, Hibernate started because Gavin thought the existing solutions were not good enough. I started TestNG for the very same reason... So why don't we wait for a little bit to see if the reasons that led to the creation of TestNG were justified?

    Let a thousand flowers bloom...

    --
    Cedric
    http://testng.org
  46. Test Framework Comparison[ Go to top ]

    If you and the writer of jTiger don't cause a testing framework schism, I have no objection if a project teams chooses either TestNG or jTiger instead of JUnit.But if it creates a schism (it will not happen, because JUnit 4 will prevent it), then it is a bad thing for the Java platform (like SWT and logging frameworks). People in the Javaworld suddenly need more knowledge to accomplish the same thing. If I find another job, maybe I have to learn TestNG. My knowledge of JUnit will help me, but it would still take time to learn it and to master it like JUnit.

    Wow, it took me a day to convert a testing tool based off of JUnit to TestNG. And another day to add the functionality I couldn't add in JUnit.

    You seem scared with your comments. Sounds much like some of the COBOL programmers downstairs who are frightened that parts of the system they maintain are going to be replaced by C# on .NET. Except, I get why they're scared, going from COBOL to OO is challenging. Going from JUnit to TestNG isn't.

    That said, I am curious about JUnit 4 and will be eager to see if the tool I constructed could be easily replicated in the next version without having to over-hack TestCase.
  47. Test Framework Comparison[ Go to top ]

    You seem scared with your comments. Sounds much like some of the COBOL programmers downstairs who are frightened that parts of the system they maintain are going to be replaced by C# on .NET. Except, I get why they're scared, going from COBOL to OO is challenging. Going from JUnit to TestNG isn't.
    You didn't understand my arguments and didn't answer to them. Instead you write some polemic stuff.
  48. Test Framework Comparison[ Go to top ]

    You seem scared with your comments. Sounds much like some of the COBOL programmers downstairs who are frightened that parts of the system they maintain are going to be replaced by C# on .NET. Except, I get why they're scared, going from COBOL to OO is challenging. Going from JUnit to TestNG isn't.
    You didn't understand my arguments and didn't answer to them. Instead you write some polemic stuff.

    I heard your arguments, but I just flat out disagree with them. Nothing to argue, since we won't agree.
  49. Test Framework Comparison[ Go to top ]

    Besides, JUnit is focused on *unit* testing (testing the internals of a single class) while TestNG covers end-to-end testing, so while the two frameworks have some overlap, they address different problems.
    I forgot to comment about that.

    As you surely know, the JUnit expert J.B. Rainsberger prefers to call them "programmer tests" instead of "unit tests".

    Being a programmer, I also do things like integration testing. It can be hard to detect bugs when just testing "units" and using mocks too often. Nevertheless I try to write most tests so that they are "unit tests".

    A unit can be anything I think. Is there a official definition? If not, it can be a class, a package or even a component.
  50. Test Framework Comparison[ Go to top ]

    With "test little by little" I meant that he should refactor the legacy code first, before he tests public methods.

    By definition, refactoring requires that tests back you up before you start changing code.
    I don't know if the JUnit design allows it. If it doesn't allow it, why don't you ask the JUnit developers to redesign it so that everyone can extend it instead of trying to replace it?

    I get the distinct impression that you've a) never looked at JUnit's Javadocs let alone its code (yuck!) and b) never tried to do anything more complex than a testXxx() method. There's obviously nothing nothing wrong with that, but arguing against solutions to problems you don't have and don't know anything about makes you sound ignorant.
  51. Test Framework Comparison[ Go to top ]

    By definition, refactoring requires that tests back you up before you start changing code.
    Then a test should be done the "ugly" way first. Afterwards refactoring the legacy code by improving design and testability. Then run test again. If still green, refactor the test in order to improve it. Hope that sounds good now.
    ... but arguing against solutions to problems you don't have and don't know anything about makes you sound ignorant.
    My concern is to keep unit testing simple by not introducing features into a general unit testing framework which only 1% of all developers demand but which might complicate writing tests for the other 99%.
  52. Test Framework Comparison[ Go to top ]

    My concern is to keep unit testing simple by not introducing features into a general unit testing framework which only 1% of all developers demand but which might complicate writing tests for the other 99%.
    That's a legitimate concern, but I fail to see how it applies to TestNG or JTiger.

    In both these frameworks, if all you want is reuse your knowledge of JUnit, it's pretty much all you have to do: add a couple of annotations, asserts and you're done.

    --
    Cedric
  53. Test Framework Comparison[ Go to top ]

    Hi Cedric,

    you try to replace JUnit and I will try to replace Ant. Lets see who is first. ;-)
  54. Test Framework Comparison[ Go to top ]

    Hi Cedric,you try to replace JUnit and I will try to replace Ant. Lets see who is first. ;-)
    Funny you should say that, I have actually given some thought to it :-)

    But ant is pretty good already, the only problem I have is that over time, my build files are looking more and more like entire programs for which I need advanced OO functionalities (variables with various visibilities, overriding, etc...).

    And yes, I'm aware of the various initiatives with Groovy and friends.

    --
    Cedric
  55. Test Framework Comparison[ Go to top ]

    My idea is not to use a scripting language. And of course no XML, because thats the greatest thing which I dislike concerning Ant. A few months ago I read that even the Ant inventor himself feels that XML was not the best choice.

    So I thought about using plain old Java instead of XML. The rough idea is that Java build files are called by a small build framework instead of XML files by the Ant framework.

    Of course this has the disadvantage that it would not be so universal like Ant with its XML stuff. On the other hand it would be easier for a Java developer to learn, write, read, test and debug. And there could be implementations for all other languages, like for NUnit or log4x.
  56. Test Framework Comparison[ Go to top ]

    Private methods never should be tested.

    I disagree too. Take the following example: you have a CRM application and you have requirements to code a "getCustomerWithContactInfo" service. What will you do? Maybe create a new EJB method and in its body put the code to fetch the customer, fetch its contact info and then merge the two data structures and return it to the caller. Your tests will check that the returned customer info is correct and that the contact info is correct too. Ideally you would have one or more tests for each aspect.

    10 day later you get a requirement to write new a service, let's say "getCustomerWithPurchaseHistory". You'd have to go through the same process again: code the part that fetches the customer, then the code that merges the purchase history, then merge the data and return it. But since you are a good engineer, you'll realize you already have a piece of code that fetches the customer, and you'll want to reuse it. If you haven't already done so, you'll probably perform an "extract method" refactoring and create a getCustomer method - which will probably be "not public", because none needs to fetch just the customer - and you will use this new method as a brick for implementing your second service.

    Now, your method is not public, should you test it or not? if you say no - go the dogmatic way, as Cédric puts it - you'll have to think what defects your extracted method may introduce and test for it on each public service that will ever call it. This kinda kills your test case maintainability, since each time you'll fix a bug in your getCustomer method you'll have to add test cases on each public method that uses it. And each time you add a new service that uses getCustomer, you'll have to duplicate all the getCustomer tests again. Because you only test public methods :-). It's just huge waste of time.

    On the other hand, if you say yes, as part of your "extract method" refactoring, you could take the test cases that test the customer part of your first service and refactor it to test the extracted method. This should be fairly easy, since you should be able to reuse the fixtures that set up the Customer part of your data configuration. As a result, if new bugs appear, you only have to write tests for your "non public" method, not for each service that uses it. And if you write a new service that uses your getCustomer method, you know that the customer part is already tested so you don't have to worry about it any more. It's much more productive and your test case base is much more maintainable.
    Back to your hard to test code. I suggest you improve the design of the classes to test little by little in order to be able to test it easily.

    Hans, sometimes, this is just not possible, because in enterprise computing you often have to integrate ancient systems - such as IBM mainframes that are so old that not even IBM consultants know how to boot it any more :-) - or you may have to deal with a recent (sub)systems written by specialized third parties, and who just don't care about your testable code ideals. They can afford not to care about it because the domain logic they write code for is so specialized, very few other people can do it. So as an integrator, all you can do is deal with such subsystems the best you can. Most often you wouldn't test that sub-system's functionality, but the way it integrates with your application.
    it is usually a design problem, not a testing problem if something can't be tested or can't be easily tested

    This so funny. I suppose it's true for green field academic projects. Get on complicated distributed computing integration project for a while, where deployment and economic constraints don't always let you take the design decision you'd like, you'll see things are not so simple as they're depicted in books and blogs. Then you'll wish to have a test framework that tries to address your problems.


    Regards,
    Emil Kirschner (http://testare.dev.java.net)
  57. Test Framework Comparison[ Go to top ]

    I disagree too. Take the following example: ...
    Hi Emil,

    I read fast your text because its so much. Maybe I have wrote it more extreme than I meant it.

    Sometimes I make private methods public too for a test. But only if it doesn't worsen the design. I don't make something public just in order to test it blindly.

    There are usually lots of other ways to test a method indirectly if making it public would worsen the design.
    I suppose it's true for green field academic projects. Get on complicated distributed computing integration project for a while, where deployment and economic constraints don't always let you take the design decision you'd like, you'll see things are not so simple as they're depicted in books and blogs. Then you'll wish to have a test framework that tries to address your problems.

    I work on a complicated distributed computing integration project myself. I'm not sure if you expect too much of a framework. The real creative work of testing has to be done by ones own grey cells. I have written a base class for JUnit in which we have all the things we need.
  58. Test Framework Comparison[ Go to top ]

    If anyone else has struggled with this and has any insight, I'd love to hear about it. I can't believe I'm alone in the universe

    Hey Mark,

    I know what you mean, I encounter the same problems. I tried to address some of these issues in TESTARE (http://testare.dev.java.net). It's got support for remote test environment introspection through test probes to let you check if remote data environments have been modified as expected (this works for databases accessible through data sources but can also work for more exotic back ends such as AS/400 data queues, CISC services or other resources accessible through j2ee connectors). When you write a test case or a test case decorator you can specify an execution scenario and the test runner will automatically dispatch end execute each artifact on the appropriate environment. Would be really cool if you could have a look - a white paper describing current features is available here. IF you feel like it you either send your impression directly to me or post on the project's mailing lists.

    Regards,
    Emil Kirschner.
  59. JTiger[ Go to top ]

    Hello everyone, I am the author of the JTiger Unit Test Framework. Having read some of the replies on this thread, I feel compelled to reply also.

    Some of the concerns about using a non 'standard' framework seem a little unfounded to me, since JTiger (and I think TestNG) can handle JUnit test cases quite simply, and as a result, can utilise the many existing JUnit extensions. The process for executing JUnit test cases under JTiger is documented here: http://www.jtiger.org/userdoc/userdoc.html#6

    JTiger was developed out of frustration of many lacking features that I believe a test framework should provide. Being a follower of adaptations of the Test Driven Development and Design By Contract approach, I use my test framework for expressing the requirements of my software, rather than 'testing' my software. This may sound like the same thing, but from a different perspective, and to a point, I agree. Whatever the case, JUnit falls far short of what I expect from a test framework, and how I intend to use one. The most prominent features, at least according to its user base, is the JTiger assertion package (org.jtiger.assertion), which covers many of the common 'assertion types' that are used in unit testing (e.g. Object equals/hashCode method contracts) and its reporting. Granted, JUnit has a lot of this functionality by way of third party extensions.

    JUnit 4 appears strikingly similar to JTiger. As an employee of the same company as Erich Gamma, JTiger went passed his desk before it was released to the public for Intellectual Property approval. Assuming that JUnit 4 does turn out to have only superficial differences to JTiger, I'm quite confident that there is still much room for improvement for a future JTiger 3. I sense a bit of competition in this arena, however, I'm not the competitive type; JTiger exists 'for my own benefit, and anyone else who may want to benefit', and as a result, JUnit 4 will probably take most developers who are looking for a new test framework.

    If you're up for some discussion on JTiger, feel free to pop in to the IRC channel irc://irc.freenode.net/JTiger

    Excellent article Justin!!
    Received well by many of my colleagues.
  60. JTiger[ Go to top ]

    Some of the concerns about using a non 'standard' framework seem a little unfounded to me, since JTiger (and I think TestNG) can handle JUnit test cases quite simply, and as a result, can utilise the many existing JUnit extensions.
    I think it doesn't handly JUnit 4 test cases yet. ;-)

    The article is okay. Just for contrast, the quality of this (AOP) comparison is much better: http://www-128.ibm.com/developerworks/java/library/j-aopwork1/ and http://www-128.ibm.com/developerworks/library/j-aopwork2/

    A little warning: I haven't used JTiger to TestNG, just read about its features.
  61. Testing private methods...[ Go to top ]

    I just make my private methods package-private and put my unit tests in the same package. The methods are just as secure and don't show up in the public API (or the Javadocs). Actually, I've gotten to the point where I just use the "default" access (package-private) instead of making everything explicitly private. It's less typing and clutter anyway.
  62. Testing private methods...[ Go to top ]

    That's probably not a bad idea. I tend to promote to protected but that's because it's weird to me to see "naked" methods like that without an access modifier.
  63. Testing private methods...[ Go to top ]

    That's probably not a bad idea. I tend to promote to protected but that's because it's weird to me to see "naked" methods like that without an access modifier.

    The problem with that is protected members are part of the "public" API, i.e. they are exposed to users who extend your class and show up in the Javadocs.
  64. IDE Integration[ Go to top ]

    Does NG or Tiger integrate with any IDE's out there?
  65. IDE Integration[ Go to top ]

    JTiger is about to support Eclipse.
    You can track progress at http://forum.jtiger.org/posts/list/44.page

    There has also been talk of a plugin Intellij IDEA from some of the JTiger users. If anyone gets serious about it, I'll post to the forum.
  66. Vaporware and Trash[ Go to top ]

    Tony, don't tell us what you are talking about, tell us what you have done in terms of integration! Your product seems to be just one more trivial implementation of reflection. I read some of your comments on your forum and frankly I think you have a dangerous lack of understanding of funcamental testing issues. It's too bad that throwing together some reflection code and setting up a web page entitles some of the worst software developers to instant recognition, or so they think.
  67. IDE Integration[ Go to top ]

    Does NG or Tiger integrate with any IDE's out there?
    TestNG has an Eclipse plug-in:

    http://testng.org/main.html#eclipse

    --
    Cedric
  68. IDE Integration[ Go to top ]

    Does NG or Tiger integrate with any IDE's out there?
    Oh and Mark started making some good progress with the IDEA plug-in as well:

    http://data.talios.com/testngj-starting-to-look-pretty.png


    --
    Cedric
    http://testng.org
  69. Is there any Test Automation framework available to automate PL/SQL APIs? Test cases(PL/SQL API, input parameter,etc) should be described in XML format. The framework can be written in PL/SQL or Java, but should take .xml file as input for test cases execution.

    Thanks in advance.
  70. Here is just a link on PL SQL testing framework:
    http://oracle.oreilly.com/utplsql/

    I didn't test it myself jet, but it seems pretty complete

    best regards

    Jörg Gellien