News: Regression Tests of Code Performance in Javolution

  1. Tests often focus on only one aspect: Validation. But although a code modification might not break your application; it may very well impact the performance significantly (for the better or worse). External elements (JVM, O/S, memory available, runtime options) are also likely to affect performance. It is therefore important to not only be able to assess the performance but also to detect (regression tests) when any change you make in your code or runtime environment breaks your timing assumptions. To this end, Javolution (BSD licence) offers a new testing framework supporting performance and performance regression tests. Developers in need of a simple framework to assess the performance of their code are welcomed to try it. This framework is currently used for the Javolution's micro-benchmark, where it has already detected inefficiencies in few of the collection operations (Java and Javolution alike). From the package javadoc:
    In a normal situation, the developer creates a TestSuite which is basically a collection of TestCases logically grouped together. Then by running within an appropriate TestContext, the developer can focus on any particular aspect of interest (behavior, performance, memory usage, et cetera.) For example:// Default tests execution, simple validation and logging of the results. new MyTestSuite().run(); // Specialized context measuring execution time // (default average time, with minimum time in parenthesis). TimeContext.enter(); try { new MyTestSuite().run(); } finally { TimeContext.exit(); } // Regression tests // (no output, AssertionException raised if any test fails). TestContext.enter(TestContext.REGRESSION); // Or TimeContext.REGRESSION for performance regression test. try { new MyTestSuite().run(); } finally { TestContext.exit(); }Logging/tests contexts do not have to output the results in a text form. Implementations may store results in databases, spreadsheets or show them graphically. For example:// Logs output to console. LogContext.enter(LogContext.CONSOLE); try { new MyTestSuite().run(); } finally { LogContext.exit(); }
    What do you think?

    Threaded Messages (4)

  2. a fantastical idea ...[ Go to top ]

    I love javolution. It works very well, is easy to use, and is well documented. I can only say that for a handful of java projects. I think the idea of obtaining metrics other than pass/fail is fantastic. I'm wondering why the popular frameworks haven't thought of it. The big question I have is why another testing framework? I have so much invested in Testng, that it would be painful to move away from it. Why not have a "Testng" or "Junit" plugin that sits atop these existing frameworks? This way you're not reinventing the wheel, but are adding definite value.
  3. Re: a fantastical idea ...[ Go to top ]

    ...I'm wondering why the popular frameworks haven't thought of it...
    May be I can clarify you why such a small amount of other frameworks were concentrating on this issue: From the definition of Unit test (wee Wikipedia for example) "By definition, it only tests the functionality of the units themselves. Therefore, it may not catch integration errors, performance problems or other system-wide issues." We never tried this kind of things for any our unit tests because we really don't care about how performant EasyMock Mocks are :) There are however some other types of JUnit tests that we (automated acceptance tests, e.g. HTTPUnit) have and there we would be most probably more interesting to see, but integration costs of such framework may overweight the benifits. For example, do I need to setup a database to integrate performance testing in my continious integration envoronment, how to see results, should the continous integration build failed in case of performance drop? Too many questions.
  4. Re: a fantastical idea ...[ Go to top ]

    We never tried this kind of things for any our unit tests because we really don't care about how performant EasyMock Mocks are :)
    Yes, but no one I know uses a testing framework exclusively for "unit" tests. Unit tests are the fundamental components of a test system, but I don't see them as the entire test system in and of themselves. You can't speculate that a component works just because the individual pieces work. There has to be some sort of integration test that verifies the pieces work together as predicted. In my experience, this is where misconfigurations, not code bugs, are found. Just my $0.02.
  5. JBoss Profiler[ Go to top ]

    I know Clebert Suconic has done a bunch of work around this area too. Maybe you guys should collaborate? ping me. -- Bill Burke JBoss, a division of Red Hat http://bill.burkecentral.com