Geronimo 1.1.1 Performance Profile Available

Discussions

News: Geronimo 1.1.1 Performance Profile Available

  1. Geronimo 1.1.1 Performance Profile Available (15 messages)

    There have been many requests about the performance of Apache Geronimo. In order to answer the question about where Geronimo stands and not enter into the quagmire of performance politics a quick report has been assembled to address this question. The report contains the actual performance data of Apache Geronimo running a performance benchmark called DayTrader. Results from several primitives are available as well as some information about the benchmark's runtime mode which simulates a stock trading application. There are no direct comparisons to other Application Servers but rather a derived metric based on the author's sampling of several other servers running the same application. The good news as well as the bad news is presented. At the end of the day Apache Geronimo does really well on Web tier work and an acceptable job in the EJB tier. More work is necessary in the EJB tier but for most people it is ready to do some heavy lifting today. As mentioned, this is a quick analysis, not a formal spec benchmark:
    This report characterizes performance using a benchmark sample. Further evaluation is required for suitability of any Application Server in its unique deployment scenario. Don't go buying $30 million worth of hardware and blame me because you didn't do any further testing.
    Without repeating the entire report, here's the summary:
    For those that prefer the bottom line, here it is. The performance of Apache Geronimo has been characterized using the Apache DayTrader Benchmark Sample. The comparison target of Apache Geronimo is a derived metric based on testing and observation of several application servers (both Open Source and Commercial). The Performance Target (PT) is not necessarily indicative of any specific server’s performance, but represents the best possible performance of the servers tested. In some cases the Performance Target was inflated based on my expectations. The benchmark application used for testing is flexible enough to compare performance of the Web, EJB and an application that represents a stock trading simulation. The Web tests show that Apache Geronimo is very competitive against alternatives in the marketplace. In almost all cases Apache Geronimo either was neck in neck with the other containers or exceeded their best number. Also, scalability of the Web tier to almost 100% while servicing 100 simultaneous clients shows there are no inherent bottlenecks that artificially limit its scalability. Note that the 100 simultaneous clients were zero think-time and were to generate a load. A separate set of tests will be conducted on multi-user scalability. The EJB tier shows that where pass-by-value semantics are required Apache Geronimo has area for improvement. Apache Geronimo lacks two significant features (a Stateless Session bean cache and the ability to specify SELECT FOR UPDATE on EJB queries) that do not allow for good performance under a heavy workload. In essence, where multiple concurrent updates to tables represented by CMP Entity beans -911 deadlocks may be inevitable. The Apache DayTrader Application highlights this problem. These issues are being addressed in the upcoming Apache Geronimo 1.2 release. Overall Apache Geronimo is very competitive for Web based workloads. Also, workloads that do not require pass-by-value semantics with EJB workloads also perform acceptably. Workloads using Container Managed Persistence may encounter deadlocks under high load.
    [Editor's note: Nice work, Matt! This is a great start.]

    Threaded Messages (15)

  2. The report would be much more useful if the DayTrader application was made available at the same time of publication and and a comprehensive execution profile was described highlighting the key component interactions across layers. The following link is not found: http://geronimo.apache.org/daytrader Before I could derive some value from the test I would need to see much more in terms of the performance profile breakdown. Showing a 10% delta in a test where the database constitutents a large proportion of the overall execution time would be misleading. A quick scan of the report did not reveal whether this was indeed discounted from reported figures. What about distributions? What is the confidence level in the reporting of averages if any are reported? If you send me a link to the setup files and application I would be interested in getting a performance engineer perform an in-depth analysis of the underlying execution runtime profile. As it stands at this moment the report will be much more valuable to the Apache software performance engineering team in tracking costs over future releases. Of course this assumes that the test environment can be placed under proper change management control (hard to achieve in practice). Kind regards, William Louth JXInsight Product Architect CTO, JInspired "Java EE tuning, testing, tracing, and monitoring with JXInsight" http://www.jinspired.com
  3. Web Site Coming[ Go to top ]

    William, I agree with your comments. I'm working on the site today and did note in the report there may be a lag. I'll have some rudimentary information up today as well as getting the profile. I have some database profiles that I'll be adding in and building out the information about the benchmark. Thanks for taking the time to read and comment.
  4. The code is available here: http://svn.apache.org/repos/asf/geronimo/daytrader/trunk/
  5. Code Availability[ Go to top ]

    Actually...its http://svn.apache.org/repos/asf/geronimo/daytrader/branches/1.1 which is is the process of being released. Packaging is changing slightly (version numbers, etc.) but the programatic content is frozen. Cheers
  6. Not surprising[ Go to top ]

    I'd imagine that saying that the EJB tier is 'acceptable' is about the nicest thing you could say. From what I know, geronimo's ejb code is based off openEJB, which has, historically, bean the worst container you could possibly find. You'd have to look pretty hard to find it too, only to be filled with various degrees of regret/rage once you achieve that dubious goal. It's not surprising that G's performance will always be sub-par. The frankenstein approach of software building is a great recipe for bad performance. Sure, you'll have lots of pretty diagrams, great looking dependency graphs, and loose coupling. All of which are fairly irrelevant to users who want a coherent appserver that they can treat as a black box.
  7. Re: Not surprising[ Go to top ]

    So what would you suggest in its place, Hani?
  8. Re: Not surprising[ Go to top ]

    The frankenstein approach of software building is a great recipe for bad performance. Sure, you'll have lots of pretty diagrams, great looking dependency graphs, and loose coupling. All of which are fairly irrelevant to users who want a coherent appserver that they can treat as a black box.
    Which makes it all the more remarkable that the Geronimo developers would be forthright about the performance of the server. This is the proper way to do open source: bending over backwards to say not only what works, but what doesn't. http://wwwcdf.pd.infn.it/~loreti/science.html
    The easiest way to explain this idea is to contrast it, for example, with advertising. Last night I heard that Wesson oil doesn't soak through food. Well, that's true. It's not dishonest; but the thing I'm talking about is not just a matter of not being dishonest; it's a matter of scientific integrity, which is another level. The fact that should be added to that advertising statement is that no oils soak through food, if operated at a certain temperature. If operated at another temperature, they all will -- including Wesson oil. So it's the implication which has been conveyed, not the fact, which is true, and the difference is what we have to deal with.
    Guglielmo
  9. Well Hani I can't disagree with your comment about working through the performance issues. You are right it is OpenEJB is the container and it does present some performance challenges. The tests I made that didn't require pass-by-value were better but I haven't re-tested with pass-by-reference. As far as the Frankenstein approach it does make development harder and I expect that's why JBoss and Sun are pulling into their turtle shells and pulling in more components into a unified position. There are days where that option is appealing :) Yes, most people only care about how the server works and could care less about Maven and GBeans. They care about how do I create apps that can be deployed on the server and not about the server internals. I thought you'd be impressed that the report didn't say anything about Gbeans or Maven :)
  10. acceptable?[ Go to top ]

    According to the results, telling "acceptable performance" for EJB is a gross over statement. it seemed to me simply sucked. At the end, web performance is actually Tomcat performance too. Plus, there is no comparison of other app servers, no Sun or BEA JDK's.. No EJB3.. i dont find it logical the explanation about not using other servers. There is no problem as long as same hardware, application and scenarios are used. it is a fancy looking but disappointing test.
  11. shell[ Go to top ]

    As far as the Frankenstein approach it does make development harder and I expect that's why JBoss and Sun are pulling into their turtle shells and pulling in more components into a unified position.
    Well, Both Sun Glassfish and Jboss are open source. Especially Glassfish is CDDL licensed, would not be a problem right? so where is the turtle shell? seems more like Geronimo's head is in the sand.
  12. shell...[ Go to top ]

    ... I expect that's why JBoss and Sun are pulling into their turtle shells ...


    Well, Both Sun Glassfish and Jboss are open source. Especially Glassfish is CDDL licensed, would not be a problem right? so where is the turtle shell? seems more like Geronimo's head is in the sand.
    First off, I think my analogy was not appropriate. To clarify, Glassfish, JBoss and Apache Geronimo are Open Source. The trend I see unfolding is that Glassfish and JBoss are moving to a more defensive strategy where they control the source code in their project. From what I understand, and I could be wrong, Sun and JBoss both are moving away from relying on other open source projects like Tomcat and doing more internal integration. This makes a lot of sense because it allows them to make their source code available as an open source offering and yet allows them to control the development process more closely. Apache is very different (I didn't say better or worse) in that it is more focused on the community rather than the technology. I'll be the first to admit that from a delivery standpoint it makes it more challenging to deliver on a schedule and since you are federating multiple Open Source projects you don't own all the code. Apache Geronimo has good working relationships with most of the projects that make up the server but they are independent projects. Its important to separate Code Availability and Community. You can do Open Source with both models. It will be interesting to see how the models develop over time. As far as the head in the sand comment...better the sand than some other location...I'm clearly not flexible :)
  13. Re: shell...[ Go to top ]

    The trend I see unfolding is that Glassfish and JBoss are moving to a more defensive strategy where they control the source code in their project. From what I understand, and I could be wrong, Sun and JBoss both are moving away from relying on other open source projects like Tomcat and doing more internal integration. This makes a lot of sense because it allows them to make their source code available as an open source offering and yet allows them to control the development process more closely.

    Apache is very different (I didn't say better or worse) in that it is more focused on the community rather than the technology.
    I wonder if the difference is that IBM supports Geronimo because they want to sell WebSphere licenses, server hardware, and who knows what else, whereas JBoss needs to make keep their open code as closed as possible so that they can be the exclusive support resource for JBoss. IBM probably doesn't care if a host of start up companies provide support for Geronimo. It's probably peanuts.
  14. Re: Not surprising[ Go to top ]

    Hani, the 'Frankenstein approach' you call out is very much in keeping with the history of OSS (i.e. reuse as much as possible, until it makes sense to re-implement or reuse something else). Trying to do everything in-house or taking control of other communities, is, if you'll oblige me, a very commercial vendor like thing to do ;-) Everything in a Linux distro isn't developed under one roof...so we can find at least one example of the 'Frankenstein approach' working okay.
  15. Not anonymous: Geronimo vs JBoss[ Go to top ]

    It is a very nice benchmark but I'm a bit disapointed. I prefer to see the results and the name of the competition. A real Geronimo vs JBoss (vs Glassfish) tournament!
  16. Performance Target[ Go to top ]

    Ahmet, James, The "Performance Target (PT)" number represents, if I read correctly, the best performance from commercial & OSS alternatives for the given test case. So, if JBoss performs best at Test A1, then, the PT target for A1 is defined by the JBoss score. If WebSphere performs best for Test A2, then the PT target for A2 would be defined by the WebSphere score and so on. This sounds like a good evaluation for the Geronimo team to determine which areas they need to work on to get to best-in-class performance. Cheers, Savio