Oracle Makes New TORPEDO Submission; Claims Top Spot

Home

News: Oracle Makes New TORPEDO Submission; Claims Top Spot

  1. Oracle Makes New TORPEDO Submission; Claims Top Spot (25 messages)

    Oracle has made a new TORPEDO submission claiming the best results in the process. The Non-Verified submission was made using Oracle TopLink (POJO) 10.0.3 Developer Preview. This submission eclipses the previous best results held by Hibernate and Kodo.

    Oracle's submission totals 21 hits. It is a single server submission that includes updated source code for the TopLink POJO API to do 1-many joins.

    To see all of the results: Click Here

    The TORPEDO announcement TSS discussion.

    Threaded Messages (25)

  2. place2Bids with 1 insert statement?[ Go to top ]

    I don't know TORPEDO that well, but I don't understand how Oracle is doing the 'place2Bids' with only one insert...
  3. place2Bids with 1 insert statement?[ Go to top ]

    I don't know TORPEDO that well, but I don't understand how Oracle is doing the 'place2Bids' with only one insert...
    TopLink is using batched writing, which does exactly as described, it groups together a series of insert/update statements and sends them over in a single database call. The SQL tracing utility used by Torpedo will only show the first insert statement in the batch.

    Dennis Leung
    Oracle Corp.
  4. place2Bids with 1 insert statement?[ Go to top ]

    TopLink is using batched writing, which does exactly as described, it groups together a series of insert/update statements and sends them over in a single database call. The SQL tracing utility used by Torpedo will only show the first insert statement in the batch.Dennis LeungOracle Corp.
    It's worth noting that Kodo supports batch writing (in fact, it uses a modified topological sort to maximize batch size while maintaining foreign key constraints), and I'd be surprised if Hibernate didn't as well. If these frameworks were tested on Oracle like TOPLink's submission, they'd get the same hit count as TOPLink. Instead, they were tested on the TORPEDO reference implementation's Hypersonic database, and Hypersonic doesn't support batching; hence the hit count is one higher.
    Using 1-many joins reduces the database hits but brings back extra data from the database. In some situations it is not a big deal. However when the "m" in the 1-m is large and/or the table size is large, you are better off doing batch reading, which brings back precisely the amount of data the app requires but each relationship level requires a separate database hit.

    When the object model is a bit richer, and you have 1-many relationships that in turn have 1-many relationships, then the extra data read in can quickly grow exponentially.
    I fully agree with Dennis on this. There are many times when one-many joins are not appropriate, and what he calls batch reads (Kodo calls them parallel selects) will result in much better performance. Like him, I look forward to more future versions of TORPEDO with more sophisticated object models and scalability tests.

    Abe White
    SolarMetric
  5. place2Bids with 1 insert statement?[ Go to top ]

    I guess batched writing is common for those OR mappers and it shouldn't influence the results for this test.

    If you look at the results, you can see that the 'place2Bids' statements Oracle uses are:
    12589033 |0|0|statement||SELECT ...
    12589033 |0|0|statement||SELECT ...
    12589033 |0|0|statement||SELECT ...
    12589033 |0|0|statement||INSERT INTO BID (AMOUNT, BID_ID, VERSION, MAX_AMOUNT, BUYER, AUCTION) VALUES ('500.1', '27', '', '650.75', 'Mark Smith', '2')
    12589033 |0|0|commit||

    Could you please explain me how the last statement, the insert, inserts 2 records???

    Hibernate for instance is doing this with:
    -1133534024 |0|0|statement||insert into Bid (amount, max_amount, auction, buyer, bid_id) values (650000.0, 750000.0, '1', 'Mark Smith', '26')
    -1133534024 |0|0|statement||insert into Bid (amount, max_amount, auction, buyer, bid_id) values (500.1, 650.75, '2', 'Mark Smith', '27')

    This is why Oracle uses 1 statement less than Hibernate, Kodo or a better tuned WebLogic: they just don't do the same thing!
  6. place2Bids with 1 insert statement?[ Go to top ]

    Hi Peter,
    Could you please explain me how the last statement, the insert, inserts 2 records???

    Hibernate for instance is doing this with:
    -1133534024 |0|0|statement||insert into Bid (amount, max_amount, auction, buyer, bid_id) values (650000.0, 750000.0, '1', 'Mark Smith', '26')
    -1133534024 |0|0|statement||insert into Bid (amount, max_amount, auction, buyer, bid_id) values (500.1, 650.75, '2', 'Mark Smith', '27')

    This is why Oracle uses 1 statement less than Hibernate, Kodo or a better tuned WebLogic: they just don't do the same thing!
    The problem is that Torpedo is using P6Spy, which has a bug that doesn't show the entire SQL. Since it is a batch, it has more than one "INSERT" in a single JDBC operation.

    Peace,

    Cameron Purdy
    Tangosol, Inc.
    Coherence: Shared Memories for J2EE Clusters
  7. place2Bids with 1 insert statement?[ Go to top ]

    Hi Peter,
    Could you please explain me how the last statement, the insert, inserts 2 records???Hibernate for instance is doing this with:-1133534024 |0|0|statement||insert into Bid (amount, max_amount, auction, buyer, bid_id) values (650000.0, 750000.0, '1', 'Mark Smith', '26')-1133534024 |0|0|statement||insert into Bid (amount, max_amount, auction, buyer, bid_id) values (500.1, 650.75, '2', 'Mark Smith', '27')This is why Oracle uses 1 statement less than Hibernate, Kodo or a better tuned WebLogic: they just don't do the same thing!
    The problem is that Torpedo is using P6Spy, which has a bug that doesn't show the entire SQL. Since it is a batch, it has more than one "INSERT" in a single JDBC operation.Peace,Cameron PurdyTangosol, Inc.Coherence: Shared Memories for J2EE Clusters
    Thanks Cameron, this explains my confusion. It also means that WebLogic could gain 4 operations by enabling batched updates. I don't know about Kodo nor Hibernate, but I can't imagine they don't batch, so they could come down to 21 as well.

    On the other hand I think the individual statements inside the batch should be counted, because generating eg. one insert is nicer, more performant than generating eg. one insert followed by 5 updates on the same record to do the same business logic.
  8. Oracle has provided another TopLink Torpedo submission that uses 1-many joining and results in 21 db hits.

    This is the lowest result posted to date, but it would not always universally be the best configuration we recommend in all customer situations. As previously pointed out, there are other factors to consider to determine ORM effectiveness than just database hits.

    TopLink has 3 results posted.

    97 hits - untuned, out of the box default configs
    30 hits - does not use 1-many joins, uses "batched-reading"
    21 hits - uses 1-many joins

    The Hibernate, Kodo and WebLogic tuned results also use 1-many joins.

    Using 1-many joins reduces the database hits but brings back extra data from the database. In some situations it is not a big deal. However when the "m" in the 1-m is large and/or the table size is large, you are better off doing batch reading, which brings back precisely the amount of data the app requires but each relationship level requires a separate database hit.

    When the object model is a bit richer, and you have 1-many relationships that in turn have 1-many relationships, then the extra data read in can quickly grow exponentially.

    What is best in real-life situations? As usual it depends, but for most enterprise systems that have a significant amount of data, using batched reading is preferable.

    The optimal solution is for developers to have control over these options and when they use joining or batch-reading is application use-case dependent.

    There are many dimensions in measuring the effectiveness of a persistence solution, database hits is just one of them and when examined in isolation, can be misleading. The current version of Torpedo does demonstrate ORM products are capable of generating and managing sophisticated SQL statements. We are optimistic that future versions of Torpedo will grow to provide insight into some of the other dimensions important to consider when looking at ORM solutions.

    Dennis Leung
    Oracle Corp.
  9. ...that uses 1-many joining and results in 21 db hits.This is the lowest result posted to date, but it would not always universally be the best configuration we recommend in all customer situations. As previously pointed out, there are other factors to consider to determine ORM effectiveness than just database hits.... Using 1-many joins reduces the database hits but brings back extra data from the database. In some situations it is not a big deal. However when the "m" in the 1-m is large and/or the table size is large, you are better off doing batch reading, which brings back precisely the amount of data the app requires but each relationship level requires a separate database hit.When the object model is a bit richer, and you have 1-many relationships that in turn have 1-many relationships, then the extra data read in can quickly grow exponentially.What is best in real-life situations? As usual it depends, ....The optimal solution is for developers to have control over these options and when they use joining or batch-reading is application use-case dependent....The current version of Torpedo does demonstrate ORM products are capable of generating and managing sophisticated SQL statements...
    I had to read the initial post again, because I only skimmed it. But all the above comments are great.

    i.e. sometimes use a select of only the columns you need, and use complex where clauses. Sometimes use multiple tables in one select. Sometimes use single table full column selects. It all depends on how you use the data.

    Or to put it another way, there is no single approach to loading data from a database which works best for all use cases/scenarios.

    What I think is wierd is that the object aggregations built into all the O/R tools seem to think you should use the same object model for all situations. e.g. for bulk read for display on a web page use the same objects which you are using when doing a single insert. But, to do this you jump through hoops, and their perception is that because the tables underneath are the same, then the objects on top are the same. Or thats how I hear the object graphers when then rattle on about optimising reads.

    Anyway, all the above quoted comments were very sensible. My problem is I have been on a large toplink project and seen the technology abused dreadfully. Not toplinks fault, but it left me feeling ill.

    Jonathan
    Emeraldjb - Sparking green DAO code generation.
  10. <blockquoteor the table size is large, you are better off doing batch reading, which brings back precisely the amount of data the app requires but each relationship level requires a separate database hit.When the object model is a bit richer, and you have 1-many relationships that in turn have 1-many relationships, then the extra data read in can quickly grow exponentiallyOops, forgot something. This bit is wrong. What you do is have a single select for each level of aggregation. Then you have an interpretation phase which places the results into the correct parent object. I had to do this for a large reporting system (trade warehouse web site) and it was blistering (fast).

    i.e. level 1
    select t1.pkey, t1.whatever from t1 where xxx
    select t1.pkey, t2.pkey, t2.whatever from t1, t2 where t2.parentId=t1.pkey
    select t1.pkey, t2.pkey,, t3.pkey, t3.whatever from t1, t2, t3 where t2.parentId=t1.pkey and t3.parentId=t2.pkey

    etc. (column names above obviously would be better)

    So each level of aggregation/composition results in only a single additional select, no exponential in there at all. The post select phase was a little messy, using hashmaps of pkeys to stick the composites/aggregates into the correct parents. But it was V fast.

    Cheers,
    Jonathan
    Emeraldjb- yes a code generator, thats green.
  11. TORPEDO is a scud.[ Go to top ]

    Why doesn't Torpedo take other things into account, like database compability + cost... two real places where TopLink may not fare so well against other solutions.

    -jp
  12. TORPEDO is a scud.[ Go to top ]

    Now I'm a Hibernate fan for a lot of reasons, but even I know that cost isn't the overriding factor in a lot of projects - performance, support, branding, special licensing arrangements, internal knowledge level, corporate standards, etc. all come into play. How's TORPEDO supposed to factor all of that in?

    TORPEDO isn't supposed to make your mind up for you covering a full suite of concerns and weighted scoring criteria - it's just a performance testing rig, that's all.
  13. TORPEDO is a scud.[ Go to top ]

    Why doesn't Torpedo take other things into account, like database compability + cost... two real places where TopLink may not fare so well against other solutions.-jp
    TopLink does, always has and always will support multiple databases. It's a heterogenous world out there and our customer base uses a wide variety of databases including Oracle, DB2, SQL Server and Sybase. In additional common JDBC-level support, features specific to each of the databases are exposed through TopLink.

    Database compatibility would be a great thing to add to Torpedo as TopLink would fair very well. We'd be pleased if it even went beyond relational databases as TopLink supports EIS systems and XML data sources as well.

    I will proactively add the following point as it is also a common misconception, TopLink is compatible with any J2EE application server.

    In fact, the first set of untuned, out of the box results we submitted was on WebLogic 8.1 using HSQL as we wanted to use the same configuration as the other untuned results.

    Dennis Leung
    Oracle Corp.
  14. TORPEDO is a scud.[ Go to top ]

    not saying that cost is the only factor, and yes Torpedo is good in its goals of putting such choices head to head, but they did miss some important points with cost / db support.

    i am sorry if this came across as if i don't like torpedo (besides its really cheezy name, thats what my reference to scud was about), its a good start and a very necessary thing to do...

    -jp
  15. TORPEDO's Next Steps[ Go to top ]

    not saying that cost is the only factor, and yes Torpedo is good in its goals of putting such choices head to head, but they did miss some important points with cost / db support.i am sorry if this came across as if i don't like torpedo (besides its really cheezy name, thats what my reference to scud was about), its a good start and a very necessary thing to do...-jp
    Hi All-
    We've really enjoyed seeing the industry's reaction to TORPEDO over the past week. We understand the desire to see other comparable elements such as cost, feature, performance and other factors. TORPEDO has been in process for nearly nine months and we drew a line in the sand on a level approach for the first iteration, while understanding that there would be a desire to see a more detailed comparison. But, we are confident that TORPEDO is on its way to achieving its goals in providing a level playing field for comparison. Althought it is one dimensional today, it was better to start simple and carefully add complexity in the future.

    Regarding the future, we've seen a lot of interesting suggestions. If there is another dimension added into TORPEDO, which one would you want and why? Features? TCO Analysis? Developer Productivity? Performance? Others?
  16. Does this sound to anyone like Oracle jumping on another "standard" implementation that wasn't meant for a full performance analysis, using it to promote themselves as the most performant?

    This happened with the Java Pet Store, which was a mess anyway. So Oracle claimed they had the best performance on the mess. Then Microsoft came along and implemented it differently, claiming that their solution was better than the best Java solution. I'd hate to see Microsoft try something similar when they finally release their ORM solution (ObjectSpaces?). The reason why is that we all look foolish. Here we have tests and implementations that are not necessarily designed to provide a fully quantifiable performance evaluation being used incorrectly by vendors to tout their own solutions. I find it to be a messy situation. If the vendors want something to write claims against, maybe we should give it to them, but my understanding of the TORPEDO thing was that it specifically did not address performance, and was a work in progress.

    So Oracle claims the top spot on a set of tests that don't do any realistic performance evaluation, and that test something entirely subjective and dependent upon the particulars of any given project (object relational mapping). Does this have significance? Can we hose the vendors down so that they stop trying to use tools we're using for our own evaluation, just for their benefit?
  17. Can we hose the vendors down so that they stop trying to use tools we're using for our own evaluation, just for their benefit?
    To play devil's advocate, why SHOULDN'T Oracle and others try to address our needs by using our own evaluative tools? I personally see no benefit to keeping vendors (open-source or commercial) guessing about our requirements are (and we certainly hate it when our users/clients do this to us, because it sets everyone up for disappointment).

    On one of my projects, my budget is extremely tight, so TopLink doesn't fit my needs regardless of its pole position. But at the same time, it's nice to know TopLink's position when dealing with heavy financial transaction loads, where more money is spent on business analysis than on all licensing costs combined.
  18. So Oracle claims the top spot on a set of tests that don't do any realistic performance evaluation, and that test something entirely subjective and dependent upon the particulars of any given project (object relational mapping). Does this have significance? Can we hose the vendors down so that they stop trying to use tools we're using for our own evaluation, just for their benefit?
    You can look at it that way, I guess.

    OTOH, Oracle is obviously working on tuning TopLink, because this is a beta release of their software that is trying to get the number of db operations down to a minimum, right? So if you are a TopLink user, you're probably pretty happy that Oracle is _finally_ working on making the product more performant.

    My understanding is that all of the Torpedo participants were hard at work getting their numbers down. Most of the results, before they were tweaked and tuned, were apparently much higher. I think it would be interesting to see someone besides the vendors running these tests _from scratch_ and submitting their own results. Look at it this way: Not every app has Gavin King (Hibernate) and Patrick Linskey (KODO) around to help implement and tune. ;-)

    Peace,

    Cameron Purdy
    Tangosol, Inc.
    Coherence: Shared Memories for J2EE Clusters
  19. You can look at it that way, I guess.OTOH, Oracle is obviously working on tuning TopLink, because this is a beta release of their software that is trying to get the number of db operations down to a minimum, right? So if you are a TopLink user, you're probably pretty happy that Oracle is _finally_ working on making the product more performant.
    I think that's an excellent point. Lots of folks are going to learn a lot about tuning the most popular O/R bridges, at least to specific requirements.
  20. Not every app has Gavin King (Hibernate) and Patrick Linskey (KODO) around to help implement and tune. ;-)
    Reading the reference docs or Hibernate in action is suffisent to optimize an "app" as simple as torpedo. Actually, those kind of docs are suffisent to understand that such "optimisation" is usually a non sense in real apps ( this is the less political correct version of Dennis arguments :-) )
  21. Not every app has Gavin King (Hibernate) and Patrick Linskey (KODO) around to help implement and tune. ;-)
    Reading the reference docs or Hibernate in action is suffisent to optimize an "app" as simple as torpedo. Actually, those kind of docs are suffisent to understand that such "optimisation" is usually a non sense in real apps ( this is the less political correct version of Dennis arguments :-) )
    Not to goad you or anything, but I don't believe you ;-)

    Seriously, how would we know if the docs are good enough, if the only published numbers are the ones supplied by the vendors? Have you tried (from scratch, not with the submitted Hibernate / KODO / CMP / TopLink work) to attain / match / beat their numbers?

    If I had a couple extra hours, I'd love to try out a couple of the ORM tools just to see. Also, besides the ones listed, I've been meaning to look at the recent revs of JDX, Castor, Cayenne, JORM, SimpleORM, and some of the other JDO implementations like JDOGenie.

    If nothing else, I'd like to see the difference between a "plain vanilla / out of the box" approach, and a tuned / optimized approach. Is the difference a factor of 2 or 20?

    Peace,

    Cameron Purdy
    Tangosol, Inc.
    Coherence: Shared Memories for J2EE Clusters
  22. Seriously, how would we know if the docs are good enough, if the only published numbers are the ones supplied by the vendors? Have you tried (from scratch, not with the submitted Hibernate / KODO / CMP / TopLink work) to attain / match / beat their numbers?If I had a couple extra hours, I'd love to try out a couple of the ORM tools just to see. Also, besides the ones listed, I've been meaning to look at the recent revs of JDX, Castor, Cayenne, JORM, SimpleORM, and some of the other JDO implementations like JDOGenie.If nothing else, I'd like to see the difference between a "plain vanilla / out of the box" approach, and a tuned / optimized approach. Is the difference a factor of 2 or 20?
    You cannot take an ORM out of the box and make a decent application in a couple of hours, this is impossible unless your app is *very* tiny. TMC claims that Tormedo is a 9 months work.
    Seriously, ORM frameworks are not (or hardly) auto-tuned engines, the configuration and thus performance highly depends on your use cases and your knowledge of ORM principes. Your performance will depends on configuration by :
     * the object graph boudaries
     * the batch loading size
     * the caching strategy
     * the relational model you've designed
     * ...
    Out of the box means nothing on actual ORMs, those tools are frameworks, not applications.

    Actually auto-tuned ORM will probably be the next (or the following) ORM generation. It will be able to decide what to lazy, what to cache, what to fetch depending on you object graph usage. This is only a dream right now :-)
    For the present days, you can dedicate someone to be an expert of ORM in your projects. It'll probably spend 1 or 2 days out of 100 of developement to tune the application. I experienced this strategy on my projects and this is the one that works. The expert does not have to be a Gavin King or so, just a guy who spend sometime to understand its tool.
  23. Emmanuel,

    You and I are quite in agreement on this. It is what you describe, this understanding of best practices for architectures using particular ORM packages, that I would like to see captured in a document. It's great to know that one can ultimately achieve about the same result with Hibernate and KODO and Toplink, but I'd like to know more about how they got there.

    Peace,

    Cameron Purdy
    Tangosol, Inc.
    Coherence: Shared Memories for J2EE Clusters
  24. This kind tunning has meaning for constant size databases only aka "readonly" database. I think it can be good to compare tools by quality, as less database features are hidden by tool as more performat data access code you can write.

    BTW, why there is no transparent SQL and object oriented stored procedures in this scientific FUD war ?
  25. Tuning Tools[ Go to top ]

    I think it would be interesting to see someone besides the vendors running these tests _from scratch_ and submitting their own results. Look at it this way: Not every app has Gavin King (Hibernate) and Patrick Linskey (KODO) around to help implement and tune. ;-)Peace,Cameron PurdyTangosol, Inc.Coherence: Shared Memories for J2EE Clusters
    Full disclosure -- I work for SolarMetric. This is another area where vendors can differentiate themselves -- how easy is it to tune, and what tools are provided to help. Kodo, e.g., provides a graphical persistence profiling tool that aids in tuning by providing, on a use case by use case basis, persistence related timing information, fetching information (e.g. which fields are loaded but never used, which fields are faulted in, etc.), query result usage information, generated SQL, and other statistics. Users can use this to effectively analyze their running applications (thus making dynamic analysis possible), and can even export the data so that they can get support quickly from people who are experienced in tuning Kodo based applications.
  26. Tuning Tools[ Go to top ]

    Hi Greg,

    I better approach would be to integrate your contextual information into industry tuning tools. This is not SolarMetrics areas of expertise and customers would like to be able to use the same tools across products and technologies (EJB, JDO, Hibernate, JDBC). Also note that tuning needs to be done in all phases and that a tuning tool offered by a JDO vendor must be able to scale under load and across deployment architectures.

    JDBInsight already has a built-in callstack classificaton for JDO. We even ship with SolarMetric snapshot to highlight this feature. What would make this better if runtime (Solar's field data and events) contextual information could be accessed by our tool during resource transactions and the informaton appended to our trace or profile. An advantage of this approach is that customers can obtain then better time resolutions as well as JVMPI event based statistics (memory allocation, thread cpu, blocking and waiting) and have the power (visualizations and transformations) and usability of a well design Swing application. JDO should focus on the raw runtime performance and the designer mapping tools.

    I need to check out how comprehensive the JMX interface is to your product's internals.

    In our next EA release (JDBInsight 3.0 EA 2) we plan to ship snapshots of the Torpedo application executing with the different vendor solutions. Much better than looking at some log file output.

    Regards,

    William Louth
    CTO, JInspired

    "J2EE tuning, tracing and testing with Insight"
    http://www.jinspired.com