Discussions

News: Performance and Scalability of EJB Applications Updated

  1. Rice University has released an update on its article on the combined effect of application implementation method, container design, and efficiency of communication layers on the performance scalability.
    This update adds EB 1.1 BMP vs CMP, EJB 2.0 local interfaces (using JBoss 3.0 and JOnAS 2.5), and JDK 1.4 performance measurements.

    Full article available at http://www.cs.rice.edu/CS/Systems/DynaServer/perf_scalability_ejb.pdf
     
    Web site with full experiment reports is http://www.cs.rice.edu/CS/Systems/DynaServer/

    Discussion about the previous version of the article is available here: http://theserverside.com/home/thread.jsp?thread_id=12698

    Threaded Messages (36)

  2. I couldn't get to the PDF, I get an authorization error.
  3. I also get an authorization error.
  4. Ditto - no matter which of their site links I try, access is always denied.
  5. When I try to access the PDF, I get:

    "You have been denied access to the document you requested. If you believe you have incorrectly been given this error, please contact the administrator at the address given below."
  6. I had the same problem with the pdf link - they may be in the process of publishing this article on OOPLA02. You can see the results from this link:

    http://www.cs.rice.edu/CS/Systems/DynaServer/RUBiS/Results/index.html

  7. The numbers of the test are misleading. JoNas is allowed 100 db connections in connection pool, JBoss only 50, hence you see a drop off at peek user connections for JBoss earlier than for JoNas.

    We have contacted the RICE people on this. Dain Sundstrom (dain at jboss dot org) has more info and I hope he comments on this more.

    Regards,

    Bill Burke
    JBossGroup
  8. Do you have any reference to say that JOnAS has 100 connections and JBoss just has 50 db connections?
    I am currently using JBoss in my project.
                                                    Hans
  9. Hans,

    I'm pretty sure it has to do with default settings. I'll get Dain to respond in full detail.
  10. Google's cache of the pdf is here:
    http://216.239.51.100/search?q=cache:jCOmaFzuYesC:www.cs.rice.edu/CS/Systems/DynaServer/perf_scalability_ejb.pdf+&hl=en&ie=UTF-8
  11. The google cache is a pdf-to-html format and it screws up the images and diagrams.

    I found a copy of the actual pdf file at:

    www.carfield.com.hk/document/java/articles/perf_scalability_ejb.pdf


    regards,
  12. The google cache is a pdf-to-html format and it screws up .the images and diagrams.

    >
    >I found a copy of the actual pdf file at:
    >
    >www.carfield.com.hk/document/java/articles/perf_scalability_ejb.pdf


    Sorry - this link is to the old article - not the updated one version using jdk 1.4. the google link is old too.
    regards,
     
  13. This link http://www.cs.rice.edu/~willy/papers/oopsla02.pdf works and probably contains similar results given that the PDF was created a couple of days ago.
  14. The originally posted link is back up now.
  15. The idea behind RUBis is very nice and I applaude the initiative. Being able to test EJB design patterns can bring very rich results. But, it's a pity that the actual implementation of RUBis breaks many OOAD and n-tier development rules of thumb. For instance, presentation is handled at the business tier: HTML transits between layers, not data. Also, the design is unfortunately quite poor: many reusability opportunities were missed. I even think those design flaws impact the results of the benchmarks and break the very good work on the performance metrics, which may (or may not) alter the conclusions drawn. Security is not handled properly using the J2EE standard way. Instead a user has to authenticate everytime she wants to access a functionality using a custom authentication mechanism (that is rewritten several times, not reused). HTTPSession objects or stateful session beans were not used to keep session data when needed (it could be a design choice but...). I'm not really sure anybody would be able to really use RUBis but I understand it was not the original point.

    However, I really like the idea of automatically generating contents for the database with parameters such as the percentage of master-detail relationships (items and bids). And the level of details in the results is amazing!

    Oh, and yes! I, for one, read the comments about the items that are exactly what we need! :))

    Keep on the good work! It is genuinely being appreciated (despite the design issues of course).

                    Yann
  16. This test suite and analysis is progressing nicely. The results are good and its especially nice to see the improvements given with local interfaces in EJB 2.0.

    Some things I'd like to see in future versions:

    - Full comparisons on JDK1.4 using EJB 2.0 / J2EE 1.3 complient server versions

    - Comparisons between container managed relationships and relationships managed in business code

    - Comparisons of finder methods generated using EJB-QL as apposed to the application servers native finder code generation (I hope these would be identical!)

    - Running the servlet layer in a container running on the same application server as the EJBs to remove the massive communications overhead of servlets calling the session facades

    - Some figures for message driven bean performance and scalability

    - How performance improves or degrades in the presence of clustering
  17. Oh, also forgot:

    - Using XDoclet to generate home/remote/local/pk code in order to give a fairer comparison of how many lines of code the developer would actually have to write (as opposed to how many they could write if they didn't use the tools avialable to them)

  18. The main thing to take from this is: EJB 2.0 CMP with local interfaces provides 1/3 the performance of straight servlets. This is an order of magnitude improvement over EJB1.1 CMP or EJB 2.0 BMP! And, pretty darn good compared to straight servlets.

    The second most important thing to note is the complexity of implementing a cmp ejb in terms of classes and deployment descriptors. Speaking from experience, the only way to successfully manage the explosion of deployment descriptors and interface files is using a tool like xdoclet or ejbgen.

    Looking forward to some future papers that replace entity beans with JDO!!!!
  19. <QUOTE>
    This is an order of magnitude improvement over EJB1.1 CMP or EJB 2.0 BMP!
    <QUOTE>

    The reason for this is the Rice study is run on an extremely stupid setup over several VMs. If you run within one JVM with optimized invocations (which is pretty much available on every decent EJB container) you don't see this difference (the invocation is "local" anyway).

    Conclusions are tricky. Reading the conclusions of this study are, hmmm, shall we say... not very flattering to the people who wrote it.

    /T
  20. From page 6:

    "Session beans offer performance comparable to the Java servlets only implementation. All other EJB-based implementation methods fare considerably worse. Session fa├žade beans and EJB 2.0 local interfaces are more than a factor of 2 slower than the session beans implementations."

    This seems to confirm the anecdotal evidence that has been floating around the J2EE world for a very long time. Servlets offer the best performance, and from the looks of this paper, the scalability generally touted as the real benefit of EJB seem to be fairly mythical as well.

    <polemic>
    EJB is a bloated technology, pushed by vendors, consultants, dilletantes and heartless manipulators, a technology based on fashion, style, contempt, and everything that is rotten in the IT world.
    </polemic>


  21. Hi

    Does the servlet only implementation handle the transaction?

    If servlet + session bean offers nothing but slower, I will not use any EJB any more. Cos the configuration of servlet is so much easier.

    From Edmund
  22. <QUOTE>
    If servlet + session bean offers nothing but slower, I will not use any EJB any more.
    <QUOTE>

    Have fun hand coding your transaction logic.

    <QUOTE>
     Cos the configuration of servlet is so much easier.
    <QUOTE>

    Yes, and EJBs are just soooooooo hard.

    /T
  23. Have fun hand coding your transaction logic


    Context ic = new InitialContext();
    UserTransaction ut = (UserTransaction) ic.lookup("java:comp/UserTransaction");
    ut.begin();
      // do some transactional stuff with XA-capable JDBC DataSources etc
    } catch (...) {
      ut.rollback();
    }

    Not too hard, and applicable in any plain Java class running in a J2EE environment, not just in servlets. Of course you need JTA support in this J2EE environment, but not necessarily EJB support. You can have that in web-centric containers like Resin 2.1, JRun 4, or Tomcat 4.1 + Tyrex 1.0, for example.

    EJB is about -declarative- transactions and -declarative- security, and about distributed objects in general, of course. If you don't need the distributed access, then that leaves the distributed transactions, basically. If you don't need either, you don't need EJBs IMHO. Note that I am mainly talking about Session Beans here.

    >Yes, and EJBs are just soooooooo hard.

    Well, not really, but I consider them overhead if you don't -need- any of its benefits. And most important of all, EJB usage ties you to an EJB container. I want to have an environment of -appropriate- complexity, so I like to be able to stick to Resin for non-EJB web apps, for example.

    Juergen
  24. <QUOTE>
    And most important of all, EJB usage ties you to an EJB container.
    <QUOTE>

    And servlet usage ties you to a servlet container. What's your fucking point?

    /T
  25. And servlet usage ties you to a servlet container. What's your fucking point?


    Let me repeat it: I wan't to have an environment of APPROPRIATE COMPLEXITY, so I want to be able to use the admirably simple Resin for non-EJB web apps, for example. And there is a greater number of robust servlet containers than servlet+EJB containers with robust EJB implementation.

    Web-only resp. web-centric containers tend to be easier to understand and set up, both for development and production. Deployment and debugging is far easier and less error-prone. Compare Resin or Tomcat to Orion or JBoss, for example. I admit that it's not too hard with the latter two, but still more complex and time-consuming.

    Juergen

    PS: Note that there wasn't a single "f**k" in my answer...
  26. <QUOTE>
    <polemic>
    EJB is a bloated technology, pushed by vendors, consultants, dilletantes and heartless manipulators, a technology based on fashion, style, contempt, and everything that is rotten in the IT world.
    </polemic>
    <QUOTE>

    You see only what you want to see. Hope your polemic is based on more than just this "study".

    /T
  27. The discussion here is interesting, and seems to break into the two perpetual camps:

    1. EJB's are [pick one or more; stupid, silly, too heavy, too much work, to little return] because [pick one or more; I said so, I read so, I think so, Somebody else said so].

    1. Servlets are [pick one or more; better, silly, too much work, to difficult to manage] because [pick one or more; I said so, somebody else said so, I read so].

    The fact of the matter is that you can build a solution with any of the patterns discussed in the article. Remember that, since thats the beauty of the *entire* environment. You can pick your design based upon your individual needs and requirements.

    So.. is an EJB silly or a bad idea to use? If you are building a guest book on a page.. probably. If you are building an application with distributed access to the business logic and a great deal of transactional control? Perhaps not. Sure you could build it with RMI.. CORBA.. stateless beans.. etc.. but you should know enough to make a real call for your own requirements.

    So, I think the performance analysis was great.. not because it shows that running through an EJB stack is slower. Anybody who thinks about it should realize that, just based upon the amount of work necessary to invoke a EJB method as opposed to a local class method. But because you can look and compare based upon your application requirements (load, scalability, business logic encapsulation, etc.) which solution and/or pattern makes the most sense.

    Another point is that this was looking at two specific application servers. It would be very interesting to see this test extended to some commercial servers as well, to get a feel for their optimizations as well. I would be curious to see how other jsp/servlet engines compare.. and how other EJB containers compare. This test/benchmark set the playing field, so lets get the other players out on the field as well and see what happens.(1)

    (1) Of course, since other products may play games with restrictions on publishing benchmarks, I realize that this may be difficult to do. It would atleast be nice to see other JSP/Servlet engines such as Jetty, Caucho, etc dropped in to the evaluation.
  28. One of my major gripes with the ENTIRE industry is the continual promotion of technologies that are more or less designed to generate value for those with a vested interest (vendors, consultants, service providers) and not a great deal of value for the people who actually use the technology.

    EJB is a great example of this type of technology. My own prediction is that in another 2 years it will be dead as a dodo, like Corba. I know, I know, there are uses for both EJB and Corba, but remember that both technologies have really been pushed to much broader applications.

    I know this shows my bias, but look to .Net to see what is happening to the technology. It really is a leap forward for all of this stuff, and it is worth keeping in mind that Microsoft did half of this stuff long before Java. COM and MTS was around before EJB was a specification. Microsoft has moved away somehwat from the COM/EJB model with .Net, based largely on the experiences with MTS, DCOM and particularly COM+. You can bag the company, but remember that some very very smart people work for them, including Szyperski, who wrote the book on components (literally).

    Things like Web Services are interesting because they wrap Stateless Session beans (if you are going to have a bean, make it stateless) and may provide the possiblity very different model of component distribution than generally envisaged in our RPC-centric universe.

    Anecdotally, if you look around the forums here and elsewhere, scan the various newsletters and reports that come out, there is a fairly broad consensus that EJB are not worth the effort. Interestingly, pretty much all of the big vendors will not let you publish results like the Rice analysis, so we can't actually have an emprically based discussion of the issues.

  29. I know this shows my bias, but look to .Net to

    >see what is happening to the technology. It
    >really is a leap forward for all of this
    >stuff, and it is worth keeping in mind
    >that Microsoft did half of this stuff long
    >before Java.

    Yeah, Microsoft did it wrong, which is why people adopted J2EE and for the enterprise, J2EE using EJB.

    The Microsoft solution locks you into XP and Microsoft proprietary solutions. With their current offerings, Microsoft still did not get it right.

    J2EE gives you the freedom to run on Linux, Solaris, etc.


    >EJB is a great example of this type of
    >technology. My own prediction is that in
    >another 2 years it will be dead as a dodo,
    >like Corba.

    Actually, .Net is pretty much already dead while J2EE and EJB is thriving and growing by leaps and bounds. See the following quote from news.com:

    <quote>
    Although developers and analysts have given .Net high marks for its technical design, some Microsoft customers have called the company's marketing plan confusing. Microsoft largely rebranded existing products under the .Net label but added little new technology. Gates on Wednesday acknowledged that shortcoming.

    ...

    In assessing Microsoft's progress toward its initial goals for .Net, Gates said the company has done well in rallying the industry around Web services and in delivering development tools. But he gave Microsoft a grade of "C" for developing what he termed "building-block services" and for making the software-as-a-service concept a reality.

    http://news.com.com/2100-1001-946087.html
    </quote>

    Looks like even Bill Gates has doubts about .Net



  30. In Figure 7, at the high end of user loading (over about 1400 clients), the JOnAS 2.4.4 - Jeremie performance with session beans is slightly better than the Servlets-only implementation.

    This is the only data which shows session beans performing better than servlets only.

    Is this just an anomoly? Any interpretations?

    Don
  31. Sorry for the problems with the link, I had no email access
    and couldn't fix the problem.
    A quick reply to 2 questions :
    1) Running Tomcat and the EJB container in the same JVM give worse results than in separate JVMs under heavy load. This is due to scalability problems of the JVM with a large number of threads. Running tomcat and the EJB container in separate JVMs on the same machine gives better peak performance than in the same JVM
    2) EJB w/ SB has better performance than servlets after 1400 clients mainly because of the connection pooling in the servlets (that we do by hand) that is less efficient than the one performed by the EJB container.

    I will not have regular email access before Sept. 1 but you can post or email me questions, I'll answer them when I come back.

    Emmanuel
  32. I'm curious about the garbage collector statistics for this experiment.

    1. How was GC data gathered?
    2. Where is the raw GC data in the test results? I can't find that data anywhere except as summarized in a few graphs in the final report.

    In my own load testing experiments, I have seen the garbage collector play a much more significant role than what I think I'm reading in this report. For example, I have seen significant performance improvements (up to 400% better transaction throughput in a single-stack JBoss/Tomcat solution on Windows 2000) by simply tuning the HotSpot garbage collector with these options:

    -XX:NewRatio=2 -XX:MaxNewSize=64m

    (See http://java.sun.com/docs/hotspot/gc/index.html for more details.)

    The default values for these parameters (especially on Intel platforms) are not appropriate for server applications even if you use the -server option. Since this report doesn't discuss these options, I assume that the defaults were used. Based on what I've seen in the past I would expect the default parameters to cause the GC to play a fairly significant role in a load test like this one.

    Hopefully you see where I'm going with this. I wonder if GC tuning could help close the gap between the servlet-only and the Entity Bean patterns. I would expect the Entity Bean patterns to be much more demanding on the GC since they involve more classes, lines of code and layers within the container. So GC tuning would probably improve Entity Bean results more than servlet-only results.

    Can anyone test the hypothesis that GC tuning would close the gap between the performance of servlet-only and Entity Bean patterns?

    Piper
  33. About the GC question, we get the time from the sun.misc.GC$Daemon.run from the profiling snapshot. As we have nodes with 1GB of memory and the footprint of our application is very low (see experiment reports), the GC has little influence in our case.

    Emmanuel
  34. Conculsion is wrong.

    Conculsion should be: EJBs are slow! much slower than servlets.
    If you use EJB, you will not be able to scale to the level of servlets.
    Also, there are no comercial sites that use EJB (Amazon, EBay, etc.)
    see baseBeans.com/bad.jsp.


    Therefore.. use JavaBean with DAO, RowSet, etc.

    One alternative to EJB is (source code):
    http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/basicportal/src_05d/basicPortal/src/org/apache/commons/DAO/


    At the very lease, your DAO should hide if you are using EJB, JDO, RowSet, etc. so you can replace, if it does not scale.

    v.
  35. Conculsion should be: EJBs are slow! much slower

    >than servlets.
    >If you use EJB, you will not be able to scale
    >to the level of servlets.
    >Also, there are no comercial sites that use
    >EJB (Amazon, EBay, etc.)

    I am not sure the the author of the above comment knows what he is talking about.
     
    According to the following link, EBay uses EJB for transaction management.

    http://www.ejbsig.com/archive/news_archive/newu_09_06_a.html

  36. The are few points left outside the study:
    1) Maintnace. Major part of product life-cycle. I guess no one doubts that mixing presentation logic with business logic is not such a good idea. Changes and modifications become icreasingly difficult. Reusabily of existing components is dramatically deminished.
    Given that most serious systems undergo major upgdades during its lifetime it becomes obious that approaches such as "servlets only" are very problematic (look at ASP projects).
    2) JSP. The study was using servlets for presentation. From my experience, JSP is much easier way to develop UI. Rich HTML can and should be separated from dynamic content, not to mention business logic.
    3) The two issues discussed above leave us with need to separate business logic to a distinct layer. Very good candidate for that layer is Session Beans. The performance is almost the same and we benefit from Conteiner provided services.
    4) Entity beans: in many places it's said the fine grained access to EB from servlets is one of major problems. How about Data Transfer Object (DTO) described in EJB Design Patters?
    5) Reading and browising is very different from updating. Use Reader Design Pattern (also described in EJB Design Patterns).
    6) Each test case of the study were done with only one technique. I think the most offective design always incorporates many solutions, where each one is optimal for it's sub task.
    While the study provides very important insights into EJB performance and scalability, its results should not be considered as verdict on EJB effectiveness, but rather a data to be uses to combine different aspects of EJB in a successfull EJB application.
    7) EB. The study mentions the EB offers a row-level access to DB. In simple scenarion it's true, but Designs such as Fat Key in Finder methods should overcome this difficulty. Besides, it's very good for "Fat Objects" - big objects with dependent objects and lots of attributes. We can realy benefit from caching of such object espessialy if they don't have transaction affinity.
    8) Not all projects are as simple (small tables, small objects). In my previous company (Semiconductor manufacturing automatization) we spent incredible amount of time and efforts to build framework that provided just same services as EJB (transactions, security, pooling, caching) (It was in C++).

    Overall i think EJB is very usefull and serious way of writing middleware logic and when used in appropriate way can deliver excelent results.
  37. If you analyze the where the Session Beans + JDBC are spending most of their time it is in communication (70-85%).

    According to the performance charts Session Beans + JDBC is pretty close in performance to the servlets only solution but the Servlets only solution is deployed on a single box. I think the performance of Session Beans + JDBC and the other EJB based architectures would significatly improve by co-locating the servlets and EJBs on the same box. Having the servelts on a separate box from the EJBs gives the servlets only architecture an unfair advantage.