CSIRO Releases ECPerf kit for JBoss Application Server

Discussions

News: CSIRO Releases ECPerf kit for JBoss Application Server

  1. CSIRO has got ECPerf running on JBoss, but due to the well-known ECPerf/JBoss licensing problems the results can't be made public. Instead, they've produced an ECPerf JBoss kit so people can easily try it out for themselves.

    There's a link to it from:

    http://www.cmis.csiro.au/adsat/mte.htm

    The kit includes a README with ECPerf deployment, run, and configuration/optimisation hints.

    Good luck, and please let us know how you go,

    Paul Brebner
    Software Architecture and Component Technologies
    CSIRO Australia

    Threaded Messages (20)

  2. What are the "well-known ECPerf/JBoss licensing problems"? I thought the only issue was that any app server that wanted to run ECPerf has to be J2EE certified. Wasn't it announced at JavaOne that SUN was going to allow JBoss to become J2EE certified without paying the license fee?

    Regardless, it's nice to see support for ECperf on JBoss. It will be very interesting when the officials results are published.
  3. When will BES5 results be available?
  4. Probably very soon.

    And they are extremely interesting!

    Jeff Gosper (Manager - Middleware Technology Evaluation, CSIRO
  5. From the CSIRO website:

    "We've also been doing experiments with ECPerf, another industry standard J2EE benchmark. We recently got ECPerf 1.0 update 2 running on JBoss. We'd like to reveal the BBops/min for JBoss on our test hardware, but due to the strict publication rules for ECPerf we can't do this for JBoss results as:

    - Tested products must have passed the Sun licensed CTS test suite.
    - Products must have 24x7 support.
    - Permission from any vendors that are performance critical must be obtained.
    - The results must be submitted to and approved by the ECPerf review committee (which will reject JBoss results due to lack of CTS).
    - Approved results are then published on http://ecperf.theserverside.com/ecperf"
  6. That's crazy. What are Sun going to do, shut down CSIRO's site?

    Surely as long as it's made clear that JBOss hasn't passed the certification, Sun has no grounds to complain?
  7. Ironically, the reasons why JBoss has conflicts with ECperf are exactly the same as why you don’t want to bet your enterprise core business to an open source free shareware app server

    - Tested products must have passed the Sun licensed CTS test suite
    - Products must have 24x7 support
    - Permission from any vendors that are performance critical must be obtained

    Of course, you want your infrastructure to be J2EE certified - that’s the whole idea of J2EE and standards compliance. It gives you compatibility, portability, support, knowledge base, and so on.

    Of course, you want a reliable 24x7 support from a trusted vendor, not from some third party bunch of contractors or news groups. After all, by saving $10K on a license you are risking your whole business.

    And, you want to be sure that the configuration you are using is approved, supported, and tested by all other participating components’ vendors, like databases, o/s, and such.

    For these reasons, how many fortune 500 companies are running on JBoss, or other free-ware? They could have saved millions in licenses considering how many CPUs they usually run.

    By the way, as recent ECperf tests have shown, the price of app server is small compared the the whole cost of h/w, other s/w, disk space, networking, support, and other expenses.

    However, I agree - JBoss is a great educational tool, and a good entry point to the whole J2EE technology. It's not the place to stay, though.

    George Northon.
  8. Such a nasty attack could only come from a defensive vendor employee.

    1) I don't think anybody disputes that JBoss is J2EE compatible from a technical standpoint. It also appears from the outcome of the recent Apache dispute with Sun, that Sun is moving toward certification for J2EE and Open Source.

    2) As for "24-7 support from a trusted vendor" what this actually means is that you are guaranteed to get call center employees who don't know much (otherwise they wouldnt' be working in that job) on the other end of the phone. I have dealt with my share of call centers from "trusted vendors" where I had to wait months for issues to be addressed by somebody knowledgeable. Contrary to your assertion, JBoss Group offers support directly, using their core developers, which is something no one else does. I've heard they deliver excellent quality.

    3)How many Fortune 500 companies use open-source? If you were not an ignorant and biased vendor salesman, you would realize that 100% of Fortune 500 companies use Open Source. The Internet standards--mailing, http, dns are all open source standards. Linux and Apache are already widely used in big corporate installations, so I would expect similar success for JBoss.
  9. The intention of ECPerf + CTS seems to be a good one in general, but in practice it doesn't always work.

    In testing out various J2EE AppServers over the last two years we've found that being J2EE compatible or not is largely irrelevant - we've often had problems getting our test code working on J2EE compatible products.

    With ECPerf we've also found:

    1 ECPerf 1.0 update 2 had a bug in the code: It has worked on most "J2EE compatible" products, but not one - it turns out that the code (and probably CTS) is at fault, not the product it didn't work on.

    2 ECPerf on JBoss at first didn't work correctly. JBoss has at least one bug which makes it non-J2EE compliant. The information in the README file we provide seems to get it to work but with low injection rates only.

    So, the assertion that JBoss is J2EE compatible is hard to judge.

    In terms of using JBoss in high-end Fortune 500 deployments, the ECPerf results published indicate that hardware is the major cost factor (not including development, deployment and administration costs of course, which are probably even larger but not accounted for :-). I don't think it would be a suprise if I told you that JBoss isn't the fastest AppServer around - version 1.1 of our J2EE AppServer report indicates as much (free summary at http://www.cmis.csiro.au/adsat/mte_reports.htm).

    Is JBoss cost-effective if have to throw say 2-4 times the hardware resources at the AppServer cf other products?

    We've had some discussions with JBoss people about the relatively low performance and scalability issues - other vendors have taken our observations seriously and enhanced their offerings, but so far we've yet to see any real improvements in JBoss performance/scalability.

    Hopefully standard benchmarks like ECPerf will help both the users and developers of JBoss discover the real (and relative!) performance and make improvements.

    Paul.
  10. In reading the CSIRO report, decisions like not using "isModified" in JBoss tuning and claiming that "isModified" is a doubtful design practice, which provides no performance improvement, demonstrate ignorance. This is one of the deepest optimizations you can do on JBoss. Can you confirm that anyone _qualified_ from JBoss even looked at your configuration, let alone signed-off on it?

    I've seen these hardware arguments before. Even taking into account the dubious benchmark, how many applications out there really call for even 1000 invocations per second, which I've found JBoss to handle easily even on mediocre hardware.
  11. isModified etc[ Go to top ]

    Hi again Pierre,

    This comment really suprises me - isn't the whole point of J2EE the provide code portability??!:

    "In reading the CSIRO report, decisions like not using "isModified" in JBoss tuning and claiming that "isModified" is a doubtful design practice, which provides no performance improvement, demonstrate ignorance. This is one of the deepest optimizations you can do on JBoss. "

    If this really is the "deepest" optimization JBoss provides then it is has problems. Anway, I thought JBoss has tuned updates? If this is done well then the container should be clever enough to supress ejbStores automatically.

    Tricks like this may help with some applications, but won't help with ECPerf which doesn't allow code changes.

    For our own application, StockOnline, we do extensive experimentation to optimise performance on each product. We did this last year for version 1 of our comparison report and found that many of the optimisations suggested by JBoss people (yes, we actually did engage in extensive interactions with JBoss developers) didn't produce the impact expected, and in some cases slowed things down.

    The full quote should actually be:

    "JBoss supports an isModified method in addition to tuned updates for CMP. There was very little difference in performance between tuned updates and an isModified method; in some cases, tuned updates outperformed isModified. Since isModified is a doubtful design practice and provides no performance improvement, it was not used."

    More detail at:

    http://www.cmis.csiro.au/adsat/jboss.htm

    So, no, our assertion that "using isModified provides no performance improvement" doesn't demonstrate ignorance, just derived from the observed facts (on our h/w, using our test application etc). This is an empirical observation, not a statement of belief.

    The claim of 1000 invocations per seconds for JBoss is very believable. However, it depends on the application, read/write mix, and client load etc to be meaningful. Under more realistic test and load conditions I think you'll find it difficult to get even a peak performance of this, and this is saying nothing about scalability.

    The perception that 1000 transactions (I assume that what you're talking about?) a second is high is revealing. Our experience is with mid to high end enterprise applications, and with web based systems where the load is peaky and unpredicatable. In such situations you want an AppServer that scales well.

    Regards,

    Paul.
  12. isModified etc[ Go to top ]

    You should not be in the benchmark business if you really believe, through observation or theory, that "isModified" is a doubtful design practice with no performance improvement. This completely bypasses db and store entirely, and, no, tuned updates don't do that. Not in the CMP 1.1 engine of JBoss. This is beginner knowledge about JBoss. You should buy the JBoss documentation and actually read it.

    You say you were in discussions with JBoss folks. I find it hard to believe they would sign off on a statement like the one above, and your configuration and results. Can you then confirm that anybody at JBoss signed off on your publishings of performance numbers on JBoss? Are you authorized by JBoss to publish these numbers? A simple yes or no will do.

    It is my understanding, the permission clause is designed by Sun to prevent incompetent benchmarks from being published, and even publishing some of the benchmark findings on app server without permission appears to violate that clause.

    So, in your global app server production experience, tell us how many CPU installations (in % of overall market) really run 1000 customer transactions per second?
  13. isModified etc[ Go to top ]

    Pierre,

    I think I've covered most of this before...

    Taken in context, our assertion that "isModified is a doubtful design practice with no performance improvement" really means:

    1 Code that uses isModified isn't portable - end of story.
    2 there is no performance improvement over tuned updates (see the full quote above)

    Regarding the semantics of tuned updates, what do you believe it does? We can only go by what the documentation says, what other products have done in the past with this option, etc. For example:

    tuned-updates: when this option is turned on (off by default) JAWS will only update in the database the fields of your bean that have actually changed. http://www.jboss.org/online-manual/HTML/ch06s03.html

    For our application (not ECPerf which is what this thread started with :-), typical results from our original report (v1) on dual cpu server (option B) were:

    tuned-update: 129TPS
    isModified: 126TPS
    none: 111TPS

    So, not a huge difference, but tuned-updates are faster than isModified.

    Again you seem confused about ECPerf vs our benchmark, StockOnline. We havn't published ECPerf results for anything - not even JBoss!

    However, we have published StockOnline results for JBoss. We got configuration and optimisation advice from JBoss forums (however, some were subsequently deleted :-), and sent the results to JBoss forums and senior developers for comment.

    An email dated 16 August 2001 from someone significant in the JBoss community (you can probably guess who) said:
     
    "thanks, good luck with your project".

    Our report actually affirms that all the products evaluated are probably adequate for non-demanding deployments - however, that doesn't mean that it isn't critical to try and distinguish between them and determine what the limits of their capabilities are (and how they change from one version to another).

    If JBoss developers really want to do a service to the low end users of J2EE, it would be worth keeping track of how well the measured performance and scalability of the JBoss product is doing compared with other products. Initially JBoss people were happy with our benchmarking results and their relative ranking to other products, as JBoss is free and others weren't. However, the gap may be widening - providing our ECPerf kit was meant to give JBoss developers and users a chance to do some objective comparisons themselves - and use the results to provide improvements where required.

    Regards,

    Paul
    CSIRO


  14. CSIRO,

    So did you guys use JBoss because every other vendor would sue you for publishing ECPerf results on their app server without first obtaining their permission?

    At what point did you get the JBoss developers' permission to publish these results? From what I read, you guys do a poor job of optimizing JBoss and I am surprised they would sign off on this work.

    <quote>
    - Tested products must have passed the Sun licensed CTS test suite.
    - Products must have 24x7 support.
    - Permission from any vendors that are performance critical must be obtained.
    - The results must be submitted to and approved by the ECPerf review committee (which will reject JBoss results due to lack of CTS).
    - Approved results are then published on http://ecperf.theserverside.com/ecperf"
    </quote>
  15. Pierre,

    There seems to be some confusion here - we havn't and can't publish ECPerf results for JBoss - or any other product for that matter. Only the ECPerf review committee can do this - all we've done is get ECPerf going ourselves on JBoss and made some information available so others can try it.

    Regarding optimization of app on JBoss, as a group we probably nkow more about optimizing EJB applications across a range of products than some of the vendors themselves.

    The README file for the ECPerf JBoss kit has some hints for optimising ECperf on JBoss, but you'll note that we point people to the JBoss EJB container configuration guide rather than claim to know it all ourselves. The optimal configurations are very likely to depend on more than just the application from our experience (number of cpus, JVM, memory, jdbc, etc all have an impact).

    Regards,

    Paul.


  16. common guys, run ECPERF and post your results and hardware
    config here !!!
  17. IANAL, but it seems to me that as soon as people start posting their full disclosure reports on this discussion thread, TheServerSide will have to worry about being sued by Sun.

    I would love to see these numbers, but I think a better way to proceed would be for us to design an equivalent benchmark (and maybe a better one, with the benefit of hindsight) just for non-certified implementations, like JBoss and Jonas.

    In any case, JBoss can be expected to win the 'smallest codebase in the industry' award, but not one for the maximum throughput. To begin with, I don't think it has TCP/IP multiplexing. Does it?
  18. An "open source" J2EE benchmark could be interesting, although there are already alternatives to ECPerf (e.g. our own Stock-Online which we've used for a variety of technologies including J2EE).

    The crucial thing about ECPerf from the perspective of Sun, the vendors and possibly users, however, is that ECPerf is a J2EE benchmark. That is, J2EE according to CTS, not just by some vague conformity to the standards. The argument goes that if there is an ECPerf result for a product then you know that another J2EE application will run on it. If the product hasn't passed CTS then it is possible that the product really isn't fully J2EE. It is also possible that vendors could produce an "ECPerf version" of their product that runs ECPerf faster than a compliant J2EE version (unlikely, but stranger things have probably happened), so CTS is required to enforce J2EEness. (Mind you, CTS doesn't guarantee lots of other useful things such as "will the J2EE application actually run?", and is the product easy to use, deploy ECPerf to etc :-)

    Maybe a series of mini-benchmarks might be the idea? (Not micro-benchmarks, but one step up!). Each could test some sensible subset of J2EE, prove that a product actually does implement the subset of J2EE being benchmarked, and allow comparison of different J2EE features for performance/scalability - something ECPerf isn't designed for at present.

    Regards,

    Paul
  19. The crucial thing about ECPerf from the perspective of Sun,

    > the vendors and possibly users, however, is that ECPerf is a J2EE benchmark.
    I'd like to add to this that ECperf is not a full J2EE benchmark yet. The current version is an EJB benchmark. SPECjAppServer2002 will be more like a J2EE benchmark.

    Regards,
    Pieter.
  20. Hello,

    I'm a bit surprised that the people from CSIRO get flamed about being "jboss beginners" or about "not having had a sign off from jboss developers". The CSIRO people made ECperf run on jboss and were so nice to save all of us the hassle of repeating that amount of work. So, everybody can now get ECperf numbers on their own jboss system. How about saying "thanks"?

    And btw, since when do I need to have a "sign off" from anybody if I want to run a speed test or whatever on GPL'ed software? It may be good practice to ask the developers about tuning, but it doesn't render a "black-box-no-tuning-from-the-developers" test useless.

    Whether jboss is fast or not should not be a matter of religion, but of testing. And my personal opinion is it is quite fair from CSIRO to say what they measured, even if the results are not so stellar as the "big talk" on jboss.org would make you believe.

    Cheers,
       Henrik Klagges
       TNGtech


  21. Thanks for the "thanks"![ Go to top ]

    Dear Henrik,

    Thanks for the "thanks"! Nice to see someone else stating the obvious for a change :-)

    Thanks again,

    Paul Brebner
    (Feeling warm and fuzzy at last)