Discussions

News: Hibernate team posted Hibernate vs Baoo JPA benchmarks

  1. Well, a short while ago the project was announced on TheServerside.com. In parallel an article was published on HighScalability.com. Both sites created a good attention that even on the first few hours Batoo website went down due to high traffic. That being amusing as well as troublesome experience for a new project, the article on HighScalability.com has become the host Hibernate team posted benchmarks that provided evidence in favor of their new competition.


    As expected, Hibernate Team was so much surprised to see a product as in the same league as their product comes along with a bold promise - Over 15 times faster then Hibernate.Starting with current Hibernate Lead Steve Ebersole they quickly ran the benchmarks and saw some numbers that suggested him Batoo was only twice as fast then Hibernate. Content with what he sees, he failed to note that the number he was looking into was including the times spent by the database, and failed to look at the numbers where Hibernate and Batoo JPA developers make a difference. When the calculations are made, he has seen the truth that what he actually posted was in line with Batoo JPA's statement - Over 15 times faster at JPA layer and twice as fast including the database. Having been informed of the true interpretation of the numbers he published, he started to base his argument on insults rather then facts.


    That's where Steve Ebersole stepped out and fellow Hibernate Developer Anthony Patricio came along with assertions on the benchmark being too much batch oriented - a fair argument. Then Batoo modified the benchmark and  re-ran it. The numbers did not change much, still over 15 times faster... Then they advocated that the three settings were needed to 'tune' Hibernate to do basic CRUD operations while Batoo JPA needed no tuning at all be it in several entities persisted / updated in a single transaction or single parent entity persisted in a single transaction.


    Then Anthony put the blame on using an in memory database. While for an experienced JPA developer changing a datasource in spec complaint persistence.xml is a piece of cake, he refused to run the tests as is. Meanwhile Batoo JPA released a JSF application 'HelloJSF', along with ready to use mysql-persistence.xml that used a 'proper database'. Again looking at HelloJSF connected to JBoss ExampleDS, he again refused to run the application which has JMeter test along and proves that the very same JSF application with presentation -> business -> persistence -> RDBMS layers, Batoo JPA clocked twice as many requests with plain switch from default JBoss Hibernate JPA. The funny thing is a datasource cannot be provided within an application as it is a JTA environment. He again could configure the ExampleDS to point at the database of his choice. He simultaneously asserted that the Batoo JPA's great speed is thanks to using a ByteCode instrumentation. Explaining that the Batoo JPA's speed is archived by line by line code written clean, with performance in mind, and applying the best practices along with ByteCode instrumentation, he seemed to back on that idea as it was never brought up afterwards. But in between the lines he was offended by Batoo JPA is being that fast the Hibernate was badly coded - so seems like he was more of defending his name rather then Hibernate.


    He then claimed that Batoo JPA was lacking the L2 Cache. A small argument on the cache issue, he then agreed that "about second level cache, it's a false friend but many people will use it just to boost some benchmark results". He kept on arguing that using an in-memory database is just as wrong. Obviously he wanted to add a bit more complexity so that the high resources spent by JDBC serialization would overshadow heavy weight Hibernate middleware.


    Then again to convince the community, Batoo JPA released a Spring Application where the datasource could be coded into Spring application - HelloRest. This application demonstrated a REST service that interfaced a CRUD and Query service that is hardwired to external MySQL database. Even in README, the project suggested that a remote MySQL should be set up for the sake of best similarity to a real world example. A Comprehensive JMeter test case was provided, so that we did not count on Batoo counters. Again Hibernate team to date, has not run the tests.


    Instead, Anthony took a different approach and wanted to take the argument to homeland, issued a blog article Decrypting another JPA benchmark. In that article, he refused to base his arguments on the built-in profiler which indeed utilizes the Sun JVM built in profiling support that the VisualVM profiler also uses. But this profiler profiles the benchmark down to line level and with 10ns snapshots. He also refused to employ a respected profiler, nor industry standard benchmarking tool - JMeter. Rather he counted on 'vmstat' to count the precious CPU cycles. That alone explains the bias and performance perception of the Hibernate Team. Even the argument was based on response time rather then throughput + response time. He tought taking the argument to his own blog would pay off and he censored my comments. Bad call!


    He posted 5 tests that he has done, but what was unfortunate that he taken the DB times into the calculation and failed to calculate the ratios. The calculated ratios were:


    Format:  (Hibernate Seconds x CPU Load) / (Batoo Seconds x CPU Load)

    Shoot 1: (13 * 200) / (9 * 210) = 137.6%
    Shoot 2: (44 * 64) / (36 * 55) = 142.2%
    Shoot 3: (301 * 29) / ( 292 * 22) =135.9%
    Shoot 4: (990 * 6) / (1075 * 5) = 110.5%Shoot 5: Numbers are not provided properly.

    A second look into Shoot 4 revealed that rounding actually mattered a lot since the numbers are based on poor vmstat output. With the calculation below, obviously the Shoot 4 performance ration is somewhere between 110.5 ~  132.5%.


    Say Hibernate CPU 6 is rounded down from 6.49 and Batoo CPU is rounded up from 4.51Shoot 4: (990 * 6.49) / (1075 * 4.51) = 132.5%

    He issued "Is this is interesting ? Yes, Should we respect that ? Yes, Is this useful ? No, sorry this is not solving real performance issues at all." But his postings revealed otherwise. Hibernate alone spent half an RDBMS would consume. Uhh too much for a middleware right? We thought so a year ago.


    He then concluded his findings "I don’t care these CPU optimizations I prefer FEATURES!". A quick look at the features revealed another important aspect. Hibernate had more then one third of the issues still open! Obviously Hibernate team was so much caught up with providing features that do not exist in JPA spec, (last time I checked, Hibernate was represented on JPA JCP Board, you smell an attempt to vendor lock-in like I do) but even all those features not only brought Hibernate performance down to its knees, but also left Hibernate users with ~3000 open issues. That being more then 1 / 3 of every issue submitted to Hibernate was never addressed. Again on that remark Anthony stormed with accusation of dishonesty and providing false numbers.


    Upon provision of evidence from Hibernate JIRA, he is yet to comment on the issue. While he concluded his postings to features are what matters, he put the blame on extra features, which of course false - the ~2500 of ~3000 open issues were in the Hibernate  Core components.

    While strongly advocating Hibernate performance was good enough, a few performance related issues popped up  Hibernate JIRA post to Batoo JPA's debut.

    We simply challenged Hibernate "Please prepare the test you like, I challenge Hibernate in every setup, in every application. Batoo JPA can beat Hibernate by far". This we will yet to see and no hesitation at all.


    Now if you are ready to give Batoo JPA and boost your application to the performance it deserves, we invite you to take it out and test drive.

    Regards,

    Hasan Ceylan

    Threaded Messages (15)

  2. Hear, hear!

    Hibernate has a really crappy architecture from trying to generalize Hibernate to do EVERYTHING. Seriously, who's using Hibernate for XML mapping? We've even found examples in Hibernate where one bug compensates for another bug!

  3. I never heard of Batoo JPA before, so this post and http://highscalability.com/blog/2012/10/9/batoo-jpa-the-new-jpa-implementation-that-runs-over-15-times.html are very interesting to me. I have the same question with Nicklas Karlsson (I saw this name somewhere, maybe in Seam) which is if Batoo JPA passes the JPA 2.0 TCK. I guess it doesn't yet (based on what I read "in the list of the things"). I love to see Batoo JPA be a little bit more mature in documentations and the TCK thing.

    I'll look at your project and tests on github when I get home (this network blocks github :( ).

  4. TCK is in the TODO list.[ Go to top ]

    Dear Thai,

    You definitely have a point in lack of Documentation. It was just not prority in the TODO list and we decided to go ahead and release it it is 100% spec based.

    TCK, yes next week we will be applying for the TCK approval. Not to say that Batoo JPA has no bugs - but check the issue managent systems of the three JPA implementations who all passed the TCK. If you do not pass it indicates a compatibilty, but if you pass it never means the project satisfies all the spec requirements.

    Besides, I would love you (and other community members) to give us hints on as 'Documentation' what would be the prority list to document...?

    Best Regards,

    Hasan Ceylan 

  5. Real Life Impact[ Go to top ]

    Hello,

    first, let me say that I really appreciate the fact that there is another JPA implementation out there, and I do get the feeling you are serious about your business, albeit a little (too) enthusiastic. Competition is clearly a good thing, and it is unlikely anyone will get hurt in this (Hibernate being the least likely IMO).

    It has to be noted, however, as Anthony Patricio pointed out in the HighScalability discussion, that "For now, you have zero impact on real life setup". In any application I have seen so far, the time spent inside the JPA stack was a tiny fraction of the overall processing time, so the optimization potential is very small. What would be interesting is if you could claim that the SQL you generated was so optimized (individually for all conceivable databases) that it performed much faster on the database layer, because that is one component that matters much more. Of course, the most prominent source of performance issues remains bad design of the application itself, or of the database model.

    So my warning goes to programmers that think that just by replacing the JPA implementation underneath a badly designed application they will get any noticeable performance gain. As the old saying goes: "There is no silver bullet" (or "no free lunch", whichever you prefer).

    regards + good luck,

    chris

  6. Real Life Impact[ Go to top ]

    The problem with just comparing JPA implementations is that many times, you have to look at the bigger picture:

    1. What do you have to deal with when just using the Standard JPA doesnt suffice?

    2. What do you replace the following with?

    - Hibernate Search

    - Hibernate Envers

    - Hibernate Tools

    - Hibernate OGM

  7. Give it some time...[ Go to top ]

    Dear Mark,

    Hibernate Search -> I would definitely use Solr / Lucene.

    Hibernate Envers -> Wait, We'll cover this soon - again with great performance. All the infrastructure is already exists in Batoo JPA, we need about 4 - 5K lines to do that.

    Hibernate Tools -> As already pointed aout. JPA is evil in wrong hands. Tools generated domain model could lead to serious performance problems. If you are not after performance use the JPA implementation you like or Hibernate. Also who said that HB Tools gnerated model cannot be used with other JPA implementations?

    Hibernate OGM -> That's also on the todo list. I am not even saying years... We'll be also investigating Spanner from Google and what it could and with at what cost.

    IMHO, one of the mistakes that Hibernate did was it tried to save the world. Take 'Search'. it should never be persistence layer's problem. Yet Hibernate was so much enthusiastic with solving every possible problem on the face of the world - it sacrifised other's who do not need bells and whistles well being.

    Other thing is, Batoo JPA does not need extra learning curve like Hibernate does.

    Please keep an eye on the project.

    Regards,

    Hasan Ceylan

  8. Give it some time...[ Go to top ]

    >>I would definitely use Solr / Lucene.

    But that means, basically, creating your own "Hibernate Search".  Just like people who say "I dont use an ORM" and then up "rolling their own".

    >>it should never be persistence layer's problem. 

    It needs to change when your pesistance layer changes.  It IS persistence, in a sense. More along the lines of polyglot pesistence in that you are storing the same data, or parts of the data, in a another technology so that it can be accessed differently. 

    You might know this, but before Hibernate Search, there was Compass. http://compass-project.org/

    Anyway, as an application developer, i should not be spending time getting my entities into and querying Solr/Lucene/etc

    >>Hibernate Tools

    I was not refering to the generation tools. Definitely not.  I was more refering to the Query and Mapping editors. Those are a must when it comes to building and testing. I also make extensive use of Squirrel Hibernate plugin for adhoc queries in addition to testing.

     

  9. English[ Go to top ]

    Hasan, many of us are international users here and very few of us don't make any mistakes in English (I know I certainly make a lot of mistakes).

    But that said, perhaps you could try to improve your English a bit? Especially if you are not just a casual user but are trying to promote your product to an internatonal audience.

  10. English[ Go to top ]

    Dear Augustientje, thank you very much for your observation.

    I will be more careful on my language skills - perhaps I should have a native speaker review my articles before posting. Having said that, we also plan to hire a native English speaker. My apologies for the noise.

     

  11. Dear Christian,

    Definitely you and Anthony has a point. Batoo JPA will never rescue a bad JPA model design nor missing DB indexes, tables structure, etc. Of course you should start with a very good design, as pointed out many times, and in Anthony's blog article's later section, bad desing will kill you.

    On the other hand I disagree with you on the amount of the processing at the JPA stack by Hibernate ( EclipseLink, OpenJPA and others). I think you agree that I can create good domain models and DBs right? But for me I never had performance issues in non of my projects at the DB Level. If you separate the BI copy out, then given the technology in modern databases, DBs are just fine. But on the other hand always my weakest link was J(2)EE stack. And namely two bullet points - 1) Persistence Layer (HTML generation) 2) Persistence Layer.

    I suggest take your app, create a JMeter scenario and throw a good DB Server in the back end and profile your application with Sampling method using Visual VM. Look at the stack traces and create two segments - 1) All the way down to EntityManager 2) Within the Entity Manager 3) Downwards from JDBC.

    The important thing is to note that when the thread is waitin on socket at the JDBC level the CPU consumption is virtually zero! The context is switched and the worker thread is sleeping on IO. So that's where the delusion lies. The total time to respond the request != Total CPU time.

    If you think I have point in there I can assist you on an online test together and then we know what we are talking about.

    I suggest you check out the HelloRest application and run the JMeter test within. Then you'll see what I mean by "Yes Performance is important even with th good designs. So that's where Batoo JPA will help you".

    Best Regards,

    Hasan

  12. TSS = Vendor blog?[ Go to top ]

    Is TSS devolving to a vendor blog?  I can see covering this kerfluffle from an impartial point of view, but the viewpoint of this article is not impartial.

  13. Batoo is proud to announce the fastest JPA implementation on the face of the earth 

     

    Hmm, you claim that just comparing hibernate performances ?

    Well, in the best case it shows a certain lack of knowledge of the players in JPA arena.

    However, the choice of the contestant demonstrates you like to win easy....

     


  14. @Guido,

    Our benchmark case also runs against EclipseLink. Benchmark shows that EclipseLink performs about %30 faster than Hibernate. On top of that with the latest tweaks, Batoo JPA is now 30x faster than Hibernate and 20x faster then EclipseLink. Remember these are JPA Layer comparison.

    You may actually run the benchmark for yourself. Only that EclipseLink is not provided in Central Maven Repo, thus you need to download and deploy it to your local repo and uncomment the dependency in benchmark pom.xml and remove the @Ignore on doeclipselink().

  15. EclipseLink Performance[ Go to top ]

    On behalf of the EclipseLink (www.eclipse.org/eclipselink) and Dali (JPA tooling) development teams and their communities we welcome Batoo to the Java persistence world. We look forward to you joining the community of compliant JPA implementations and hopefully participating in the ongoing evolution of this important standard within the JCP.

    As far as the performance claims, as others have pointed out, this benchmark and its claims lack in credibility as the features you chose to use, the model complexity, the deployment architecture, and the measurement approach are all considered suspect and self serving. The Batoo benchmark as it is right now is very trivial in its use of JPA as compared to what we see in our customers. A good JPA benchmark will need to at least address concurrency protection with locking, L2 caching, rich domain models, and more complex querying and transaction scenarios.

    None-the-less we ran the Batoo benchmark with the default EclipseLink settings on MySQL and measured the total time to complete the operations, which for a real application is all that matters. EclipseLink was faster across the board and this was WITHOUT using any of the numerous performance tuning options available. It is worth noting that we also attempted to run the benchmark on an Oracle database, however Batoo blew up, and did not complete the run.  EclipseLink ran the benchmark on the same DB just fine.

    The SpecJ (http://www.spec.org/) benchmark is one of the mechanisms we use to drive performance innovation into EclipseLink as well as to provide a public result governed by an independent entity. We are very proud of the critical role EclipseLink plays in Oracle's world-record SpecJ results.

    As you will discover, persistence may not be glamorous, but it is a critical component of almost every enterprise application. It’s also really hard. We have been delivering persistence solutions since the mid-90’s and continue to be amazed at how this space continues to evolve and grow with new demands for functionality, scalability, and integrations.  And while O-R mapping is challenging in itself we've also added capabilities in areas such as multi-tenancy, XML-binding, and JSON-binding to meet the needs of real world enterprise applications.

     

    Doug Clarke

    EclipseLink Project co-Lead

  16. EclipseLink Performance[ Go to top ]

    Dear Doug,

    I have just seen the comment, I apologize for the late reply.

    Thank you for spending time to evaluate Batoo JPA and the warm welcome. About JPA compliance we are in contact with Oracle we hope to be certified soon.

    On benchmarks, you claim that it lacks the credibility in a number of aspects. We do appreciate that the benchmark was created by Batoo and we agree that it is self serving but in contrast to what the comment suggests, we use it to assure that over each development iteration the performance is preserved.

    As always we have asserted, we are confident. I am sure EclipseLink has a benchmark suite. Please share it and we both agree on numbers. Please make your benchmark suite available and we benchmark not only Batoo JPA and EclipseLink, others on it as well. As for the L2 cache, our assertion is that Batoo JPA is so fast that you will not need to resort to the L2 Cache. As mention in the discussion on highscalability.com highscalability.com, L2 Cache performs good in benchmarks, but in cluster environments it creates the overhead of cache management, this comment does not apply to read only systems. While at it, Batoo JPA has a L2 Cache implementation that is currently work-in-progress. We will release it in a few weeks.

    I was surprised that EclipseLink, when the benchmark was run on MySQL came faster. A quick run with 100 iterations produced the result below:

    BATOO 4134 180 3954 Criteria Test

    BATOO 8164 459 7704 Find Test

    BATOO 4227 251 3976 Jpql Test

    BATOO 2528 148 2379 Persist Test

    BATOO 2180 61 2119 Remove Test

    BATOO 1997 62 1934 Update Test

    ELINK 8796 4026 4769 Criteria Test

    ELINK 5955 1833 4121 Find Test

    ELINK 5492 1534 3958 Jpql Test

    ELINK 5634 1205 4428 Persist Test

    ELINK 2071 295 1775 Remove Test

    ELINK 2802 468 2334 Update Test

    BATOO 23230 1161 22066 Total

    ELINK 30750 9361 21385 Total

    So Batoo JPA stands faster by far.

    On JSpec Benchmarks, we prefer the benchmarks to be free, open, accessible, modifiable by everyone. If you are interested in, let us get together and create one.

    After all, what matters is the real world applications. We soon will start publishing the enterprises converted to Batoo JPA to get a better performance.