JPA comparison: Hibernate, Toplink, OpenJPA, Eclipselink

Home

News: JPA comparison: Hibernate, Toplink, OpenJPA, Eclipselink

  1. I wrote a relatively simple program which executes some queries and inserts in a MySQL database through JPA. Four fixed-time tests were done with exactly the same code, just changing the JPA implementation library and the persistence.xml. I monitored the resources used by the JVM and counted the inserts and queries executed during the tests. Finally, I show here my conclusions and also the results of these tests, so that you can draw your own. I consider the differences found among the different implementations truly relevant. For the tests performed for this article, nothing except JPA was used. No web pages, no web or application server. Just java threads, JPA and MySQL. JPA implementations comparison: Hibernate, Toplink Essentials, Openjpa, Eclipselink

    Threaded Messages (34)

  2. I wrote a relatively simple program which executes some queries and inserts in a MySQL database through JPA. Four fixed-time tests were done with exactly the same code, just changing the JPA implementation library and the persistence.xml.
    I read your post. The "fixed time test" part of it is very weird, as you are not measuring: 1) How fast each one gets stuff done (e.g. length of an iteration); 2) How efficient each one is (e.g. number of queries and various inserts / updates / deletes per iteration); 3) How concurrent each one is (e.g. effect on throughput as threads are added); 4) How much memory each one uses (e.g. amount of memory per unit of work). While it's great to see work being done to attempt to quantify the strengths of these three projects (i.e. I don't mean to sound negative), I did have a hard time drawing any solid conclusions from your article. One other thing to consider: the JVM memory stats that you used are very unreliable (imprecise). I've started using some of the mbeans in 1.5 and later, and also some of the tools that come with jRockit for looking at stuff like that. Peace, Cameron Purdy Oracle Coherence: Data Grid for Java, .NET and C++
  3. Hi Cameron. Thanks for your comments. Let me answer to some of them. 1.- When you say "get stuff done", I suppose that you mean an iteration mean. You can calculate that dividing the total amount of inserts by 30 minutes (or 1800 seconds), which was the pre-fixed running time for each of the tests, and dividing the number of queries by 5 (number of queries per iteration) and by 30 minutes. 2.- The number of inserts and queries was fixed for each iteration. That is one insert per iteration for the inserts thread and 5 queries per iteration for the queries thread. 3.- I agree that that data could have been interesting, but actually I was more interested in the overall performance after 30 minutes working and the total number of inserts and queries executed after that time. 4.- I am not sure what you mean here. By "unit of work" I suppose you can only mean "thread", and I would say that it is not possible to know how much memory a thread is using, given that all user threads shared the same memory allocation. I am going to rewrite the "description of code" section to make it a bit clearer. About the conclusions, I am sorry to hear that you had a hard time drawing any solid conclusions from my article, but I am not surprised here because I did have that hard time too drawing mine. I would have liked to have had a beautiful and clear results that would lead anyone (and me) to the perfect JPA implementation. Instead of that, I had some weird results from which nobody could really draw any clear winner implementation. So I decided that I would try to draw some objective conclusions with which everyone could agree, even when they would not lead anyone to one implementation or another. So sorry if you were looking for that. About jRockit, all I can tell you is that I tried to use open source or free tools. jRockit belonged to Bea and I supposed that today it belongs to Oracle. So I do not really know if it would have been possible to use it for these tests. Thanks for the tip anyway
  4. "About jRockit, all I can tell you is that I tried to use open source or free tools." Why handicap your analysis? I thought the point of the article was to actually perform an in-depth comparative. You could have published the code but used appropriate tooling during your own testing to get the real insight into the different runtime behaviors and data access patterns. It would be nice to see whether it was all confined to the database due to particular concurrency tx/query mixes in one or more of the products tested. William
  5. Wow, Cameron it looks like you finally came around to the fact that it is the activity and its directly attributed resource consumption that is important in any analysis. Now if you could only upgrade your choice of arsenal there would be no stopping you. I am still surprised that even after years of introducing resource transaction path analysis as a means for performance management that it still does not appear in any benchmark comparisons. William
  6. One other thing to consider: the JVM memory stats that you used are very unreliable (imprecise). I've started using some of the mbeans in 1.5 and later,
    Cameron, those sad looking graphics are in fact Sun's jconsole connected to the VM pulling information through the Java 5 JMX Platform Management Interface. So in fact he did use the same memory stats you advocate. It would be much more interesting to see both the memory footprint as well as the object allocation costs (byte sizes, counts) per unit of work for capacity management when the concurrency is increased to realistic (enterprise) levels. Another important point hinted at is the statistical distribution of times. William
  7. Why weren't any JDO implementations included?
  8. Why weren't any JDO implementations included?
    Because it was a comparison of only JPA implementations. Comparing JPA and JDO was out of the scope of the article.
  9. It'd be nice to be contacted about these things in advance on forums etc, OpenJPA's defaults are not tuned to go fast. The defaults are an existing condition as a Doctor would say. We've been reluctant to change the existing default behavior for fear of breaking/upsetting existing people using it. Some options added to the persistence.xml would make a huge difference in this comparison. Enabling statement caching and other features probably on by default on the others. It would be nice if the article actually listed the performance related features enabled in all the products. Actually, a list of available performance features in it self would be useful. The methodology is a little suspect as Cameron indicated. A 30 minute run isn't that precise. Better would be a 20 minute warmup followed by a 30 minute run. JITs are famous for taking their time to actually start to go fast.
  10. OpenJPA's defaults are not tuned to go fast.
    FWIW, this is also true of Hibernate. The default settings are extremely conservative.
    It would be nice if the article actually listed the performance related features enabled in all the products. Actually, a list of available performance features in it self would be useful.
    Yes, a matrix of options and whether they were enabled in the test would have made the article much more interesting.
  11. I think the real problem with this comparison is that it doesn't take into account the "enhancement" or byte-code weaving that several of the JPA providers utilize. For example, OpenJPA uses an enhancement process on the Entities for a "production" environment. This enhancement can be done statically before execution, or it can be done dynamically via a javaagent (JSE) or via the Container's classloader (Java EE). Without this enhancement process, OpenJPA falls back to a sub-classing approach which is not meant for production use. Sub-classing is used for the initial, easy out-of-the-box experience. That's it. To run a performance comparison without using the enhancement process is suspect. It's not "real". Given that, I downloaded the code and ran a quick test on my Thinkpad (Windows XP, Intel Core 2 CPU, T7600 @ 2.33GHz, 3 Gig of Ram) using the javaagent to dynamically enhance the classes for OpenJPA. First off, I noticed that about half way through the test, I blew MySQL out of the water with a socket IO error and that killed one of the insert threads. (MySQL pegged my system at 96+% CPU during the whole test.) So, with just a single insert thread for half of the test and running on an "office" Thinkpad, I still got 3811 Inserts with 2990 Queries. A huge increase in Inserts and a slight drop in Queries. Bottom line, this type of comparison can generate a lot of conversation (per the appends here and on the original blog post), but they shouldn't be used to make any type of decisions. There are too many variables.
  12. I really want to get the misinformation out of this thread/post right now as I think the conclusions that someone draws from reading this blog and study as well as the set of comments here can actually lead them down the wrong path when making decisions about the performance of JPA in general and specific JPA providers. My issues are below and I am happy for conversation or rebuttals to them: 1) You're not actually testing JPA performance, you're testing your laptops disk sub-system performance: The throughput data pulled out of the benchmark in question is suspect at best given the environment used and, in my view, is incorrect. Having done nothing but performance work and java development for the past 10 years I would never think of benchmarking on the hardware chosen here because its not realistic and the methodology isn't representative of a real environment. Some are probably saying "Yeah it is" at the moment but it's simple to see that it's not since the database alone for the insert test will be the bottleneck not the JPA runtime. I challenge anyone to find me a JPA runtime that cannot saturate a 4200RPM internal laptop drive doing inserts. The IO bandwidth on the disk is significantly less than the data anyone of these runtimes can produce given the processor at hand in even this laptop. Beyond that by testing everything on the same box you're testing mySQL's ability to deal with the inserts and its processing as well as the JPA runtimes, the OS's ability to handle IO as well as context switching between the mySQL process/JVM/kernel, etc so to call this a JPA benchmark is misleading at best. 2) You don't disclose your testing methodology fully: Are you cleaning the database after each test so each test starts with the same amount of rows in the database, does each test start with the system in the same state, are you monitoring other processes going on in the background, do you have the correct indexes on the database create, etc? Performance testing needs to be apples to apples to be accurate but the only thing that is disclosed is the test scenario and code which is hardly enough given how system resource intensive your test case is and how much the variables listed above can influence the results of the test. Why is this important...what happens if you run the OpenJPA test and fill up the MySQL log just to the point of where it needs to spill to disk on a checkpoint then you kick off the Hibernate test. He'll pay the cost for the checkpoint and have lower performance numbers. Just an example of what can happen but I don't see anywhere that you say this doesn't happen. 3) Out of the box configuration testing is incorrect: Gavin and Billy are right here in that most software products ship out of the box with a very basic configurations (I doubt your enhancing your classes) to help enable developers to get things up and running quickly on their laptops and get folks started cracking at coding not setting production performance records. Any good admin worth their salt knows that development and production performance levels are significantly different because in most cases you aren't running the same hardware in production that you are in development and the software can be much more optimally tuned. Developers wants quick startup times and low memory utilization, production users for the most case want low latency and high throughput. As in almost everything in the world you can't maximize both and anyone that says they can are full of it. If you want a fair comparison test the products in the correct mode for what your trying to prove. If its production performance which is what you appear to be claiming configure the providers for that environment. Net is that while I am happy to see folks doing their own performance testing out there you have to do it correctly. After lots of hours testing all these providers on hundreds test cases and lots of different machines and JVM combinations I can say definitively the conclusions reached here aren't typical at all. Sorry for the long post but I felt it was necessary.
  13. Hi John. Thanks for your comments. Let me answer you question by question. 1.- Let's assume that you are right, that the hard disk I/O is a bottleneck. Then how do you explain the fact that each of the implementations performed completely different? Shouldn't they all have performed really bad? Why so many differences when all that it was changed from test to test was the JPA library? 2.- Yes, I cleaned the database (the new inserted records) before starting a new test. I have just included that detail in the article. No, I did not monitor other processes on the background because of these reasons: there was enough memory for the java process and the database and the CPU was below 5% in both of the machines without java and the database working. No, there were no indexes. I left the database as it installed by default. But this was so for all of the tests, so all of them were executed using the same scenario. 3.- I know that a lot of detailed configuration can be done to improve the performance of each of the implementations. The thing is that I intentionally wanted to test them out of the box. Why? Because if I had changed the configuration parameters, there would have always been some expert that could have told me how to improve this or that implementation. And so the tests and the article would have been never-ending and probably never considered fair. So, as you can see, there is not such thing like the perfect test. What you consider that it should be done to make it fairer or better, others would consider it unfair or worse. I thank you any way for your opinion on the article and for your comments/critics/proposals.
  14. Why weren't any JDO implementations included?

    Because it was a comparison of only JPA implementations. Comparing JPA and JDO was out of the scope of the article.
    There are some extremely high performance JDO implementations (eg., DataNuclues, formerly JPOX) that have in previous performance tests shown that a little bit of byte code enhancement can mean an entirely more efficient, cleaner and much, much faster internal architecture. Once you don't have to rely on a proxy pattern to implement your persistence mechanism a whole world of performance improvements come your way. That's likely why Hibernate traditionally never quite hit the speeds attained by the popular JDO implemenations in the past - especially on deep or complex object graphs. Now that some of these JDO implementations also implement JPA it really opens up the playing field. The popular DataNucleus JDO implementation is now also a fully fledged JPA implementation so including DataNucleus in this test would not be comparing JPA to JDO. It would clearly be comparing a very high performance JPA implemenation with the set you have compared with. Please make sure you enable connection pooling though - in some comparisons in the past JPOX/DataNucleus was shown to be 'on par' with Hibernate in terms of performance - the amazing part of this was that JPOX was tested without connection pooling enabled and Hibernate was tested with connection pooling on. With connection pooling on JPOX/DataNucleus gets around a 60% boost!
  15. Ibatis option?[ Go to top ]

    I jumped into your code to add in another comparison... with Ibatis. It is the same yet different of course. I had a few 'issues' on the methodology, so I am not going to say this is an apples to apples comparison, so to speak. 1.) The table design I inferred from the JPA annotations. Did you have this generate tables, or do you have DDL? 2.) I did not see that the title class was mapped to a table? 3.) The query never loads a class instance, so I only saw the need to populate the Employees and Salaries tables with data for my test. 4.) The insert only adds data for the Employees class, so I am not inserting to the rest of the classes. Thus, these numbers may be a bit 'better' due to the fact the JPA instance is automatically posting to many tables, though I don't see where it is getting data set for department, or such. My testing basically ran the same time limits you had and using iBatis, MySql, and my crappy 4 year old laptop with 1.5 gig of ram and a single core processor on windows XP I was able to insert more than 122,000 rows into the employee table and executed 33385 queries. It took me less than an hour to generate the basic iBatis code needed, including a configuration within Spring to bootstrap the system. Probably that was a bit slow, since I spent time trying to parse through what is used and unused, and generate tables by hand. So, I am curious about the methodology in a bit more detail to try to make a true comparison. My test (and I did not perform the JPA one yet, cause I have no implementation setup) ran a factor of 10 more than the results you have. It could be due to hardware, though I hope not? Before I spent the time on the JPA code though, I wanted to get information on what data you 'preset' into the system. Admittedly, I only added 1000 rows into the system to start the system, so if you truly are adding many more than that, it would certainly slow down querying. Very nice start to the discussion though. It is slightly different approach you have taken than measuring a single transaction, though I like the mix of items as it leads to a more realistic profile of randomly hammering the database.
  16. Re: Ibatis option?[ Go to top ]

    Issue by issue: 1.- I created the entities from the employees example database. Please check the reference on the article. 2.- The table "Titles" does exist on the database, but given that I did not use it on the test, I did not pay too much attention to it. 3.- You are right. Actually only the Employees and Salaries tables are used for the tests. The Employees table has 302,847 records. The Salaries table has 2,844,513 records. So there you have the answer why your results have nothing to do. You can not compare executing queries against a 1000 records to executing the same queries against a 2,844,513 records table. You are really comparing apples with pears here. Besides, I did not do any optimization for any of the implementations, as I said before. I am not really sure that you did the same for iBatis. In case you did not, we would be talking about absolute different things.
  17. Re: Ibatis option?[ Go to top ]

    Thanks for looking at this and answering the questions. I agree, the extended period of time taken in queries would certainly make the results non-similar. I am hoping this weekend to make time to try the JPA version on my machine with the same starting point for the data. Then we can have a fair comparison. I do think the test would be more representative if you do a cascading insert. I was relieved not to have to do this and have to write the extra ibatis code, but I think that is part of the advantage and could be a performance question within the inserts inside the framework. I do want to say, I think your test is a good representation of real world usage. I am very pleased to have a benchmark, even if not 100% perfect to look at the JPA solutions as well as iBatis.
  18. There is not an implementation that clearly has the best performance. Some had a very good CPU or memory performance and some did it very well when inserting or querying. But none of them was outstanding as a whole.
    What is meant by very good CPU performance? This cannot be taken on its own as low CPU could imply a bottleneck. Take a look at this analysis specifically the second chart. http://williamlouth.wordpress.com/2009/01/16/profiling-sampling-versus-execution-part-2/ High CPU could be a good trade off in a demanding environment if it reduces contention at the database tier whilst there is lots of processing capacity in the middle tier. The last sentence has me confused. What were you expectations?
  19. Result Meaning ?[ Go to top ]

    Let's take for example the best insert performance: 9607 inserts in 1800 seconds. We're talking about 5 inserts/sec for the winner. Does it make any sense to look any further with that kind of performance result or am I missing anything here? But looking at the insert thread code: tx.begin employe = getEmployeById() insert( copy(employe) ) tx.commit Are you really sure you're testing the insert capability of that framework? Cheers, _marc
  20. Re: Result Meaning ?[ Go to top ]


    Are you really sure you're testing the insert capability of that framework?
    Why not?
  21. Re: Result Meaning ?[ Go to top ]

    I believe it's because of the transaction being created and commited for a single insert, instead of inserting multiple persons (100, 1000 ...) for each transaction. Transaction operations are usually costly. Moreover, the batch insertion is not possible in this scenario.
  22. Re: Result Meaning ?[ Go to top ]

    I believe it's because of the transaction being created and commited for a single insert, instead of inserting multiple persons (100, 1000 ...) for each transaction. Transaction operations are usually costly. Moreover, the batch insertion is not possible in this scenario.
    I am (and was when I did the tests and wrote the article) aware that opening and closing a transaction for each insert knowing that I am going to insert thousands of records is not optimal at all. What I really had in mind was a web application with several, let's say, GWT remote services, or Struts actions, or servlets in short. In that case, for each service you do have to open a transaction, make the persist, commit and close the transaction. And there can be a huge amount of servlet calls (inserts) at the same time. But the application is on-line, not batch-processing. In that scenario, you have to program as I have done it for the tests. And the JPA implementation should be efficient enough to deal with that load and with the continuous transaction open and close.
  23. Re: Result Meaning ?[ Go to top ]

    Santiago, the test is making 18 concurrent SELECT on the employees table and 2 concurrent select/insert transactions on that same table. That's a lot of "potential" contentions on the server side depending on its setup. Are we testing the ORM capabilities or the DB server? I would say that all the JVM are simply waiting which is explaining such a low (abnormal?) CPU usage. My feeling is that we're comparing Formula 1 car performances using that cars on small busy countryside roads; yes some are faster than others but at around 60km/hour for the best one who cares? Don't get me wrong, I'm appreciating that you have a setup for 4 JPA implemenations and proposing such a daunting test ;-) But I guess you should include a "direct JDBC" implementation that will act as a reference telling us what we can expect from the DB server setup to put in perspective the ORM results first. Then, if the results look good then we can have a look to the comparison. And scaling with the number of concurrent connections should be a parameter of the test. Then we can improve the DB server setup and the ORM setup to bring them to their limit and see which one is better in different test cases. And as somebody said use the ORM in more interesting situations : cascade, orphan, batch processing, update, select-for-update, lazy/eager fetch, etc... Regarding the testcase - correct me if I'm wrong: You should add the query from the insert thread in the query result ;-) The employees @OneToMany relations have no cascade specified and their fetch are LAZY; are we inserting an employee with empty relations? The querying thread seems to make only "select count(...) where ... " queries. Is that really what you want? Cheers, _marc
  24. If the memory footprint is indeed part of this test then ideally we need to make a distinction between fixed and variable memory footprint costs which means this test needs to be run with different thread worker counts. I would not be so worried if one framework had an additional fixed memory footprint size of 20-50 MB if the variable cost was kept lower than others.
  25. Having used ORM for several years with different projects and different tools, that benchmark feels pointless to me. Testing for a fixed time doesn't really tell me about the quality of the product. Here is a list of better things to test. If I do a select for update on a grandchild, how does the tool handle the lock? Does it create a large query with outer joins to the grand parent? How does each product handle eager loading? Does it use a join query or is it multiple queries? How does each product handle batch and write behind inserts/updates? Does does using JPA API affect the developers ability to turn on/off optional settings in each product? How does each product handle cascading insert/update of an object graph? How long does it take to save object graphs of different size? peter
  26. Hello all!!! The JPA Vendor comparasion is really nice and give us a shape about the big 4 JPA Implementation default behavior, but I want to introduce a little issue that's rounding my head since I began to work with JPA. I made a post in the hibernate user forums a few days ago, and may be if you take a look could help me to stay away the buzz that don't let me sleep well ;-) This's the post http://forum.hibernate.org/viewtopic.php?t=993709
  27. Just a suggestion[ Go to top ]

    (Inspired by http://shootout.alioth.debian.org) Just tossing the question up in the air here, but how about a similar "shootout" site of performance comparisons for frameworks? That is, a set of simple and well-defined problems be provided within each framework domain and gurus from all camps are free to contribute (and improve upon) solutions based on their favorite framework of choice. This way hardcore Hibernate/Toplink/whatever advocates could put their best skills to work in competition with each other, which would over a number of tests yield some insight into what each framework is really capable of.
  28. Re: Just a suggestion[ Go to top ]

    I think it's called jspec and most vendors are already in that. 70-80% of that benchmark is OR mapping. It's as good a test of JPA implementations as any other.
  29. Re: Just a suggestion[ Go to top ]

    I think it's called jspec
    Where ?
  30. captcha hacker[ Go to top ]

    back to the drawing board...
  31. Some numbers.[ Go to top ]

    If it helps. I did some recent performance testing on a Sun T1000 1.0Ghz 6 core 8G ram and HP Intel 8 CPU 16G RAM MS SQL 2000 DB. Simple web service (Metro), accept request, JDBC insert to DB, respond with identity id. Roughly 3000 request/responses a second. So technically any of the JPA implementations should be able to achieve those numbers or close to them depending on how much they do under the hood. Used Soap UI for the testing.
  32. Since you didnt include DataNucleus in your comparison, here it is (using out-of-box configuration, derby as database): DataNucleus run #1: Inserts: 31600 Queries: 26635 DataNucleus run #2: Inserts: 49150 Queries: 15705 Hibernate run #1: Inserts: 15990 Queries: 15875 Hibernate run #2: Inserts: 16050 Queries: 17320 total number of operations DataNucleus: 123.090 Hibernate : 65.235 DataNucleus is using javaagent for runtime enhancement, which reduces drastically DataNucleus performance. Still, DataNucleus proves being twice faster than Hibernate.
  33. Wow, the developer of DataNucleus does a totally fair and unbiased benchmark and discovers his product is *twice* as fast as the competition. Who would have imagined? Tee hee.
  34. Wow, the developer of DataNucleus does a totally fair and unbiased benchmark and discovers his product is *twice* as fast as the competition. Who would have imagined?


    Tee hee.
    Under the conditions (derby+out-of-box config) and the test proposed here, yes DataNucleus was twice faster. But who would care or advocate this test framework would be any credible? If the criteria is performance, just drop ORM and switch to plain JDBC.
  35. Wow, the developer of DataNucleus does a totally fair and unbiased benchmark and discovers his product is *twice* as fast as the competition. Who would have imagined?


    Tee hee.
    Who would have imagined? I'd guess it's any Java developer with more than a couple of years experience who is able to understand the intuitive performance benefits of a byte code enhanced persistence implementation over one that relies on reflection and a proxy pattern. It's not that hard to imagine really. The byte code enhancement boogie man is out of the cupboard and no one is scared of it anymore. Now developers who insist on using a JPA solution can take advantage of the performance benefits that have been traditionally reserved for JDO users.