The ECperf Explosion: Nine New Results Posted

Discussions

News: The ECperf Explosion: Nine New Results Posted

  1. The ECperf Explosion: Nine New Results Posted (47 messages)

    ECperf has exploded today with nine new results from companies including Sybase, Oracle, BEA, and IBM. Highlights from these results include Oracle's new record breaking Performance figure of 61862.77 BBops/min@Std and a Price/Performance figure of $5/BBops. A newcomer into the race, Sybase also posted an impressive Price/Performance figure of $6/BBops.

    The following is a list of all the new postings, categorized by Price and Performance:


    Best Price Postings
    ---------------------

    Oracle $5/BBops 24639.37 BBops/min

    Oracle $6/BBops 23109.77 BBops/min

    Sybase $6/BBops 8286.73 BBops/min


    Best Performers
    -----------------------

    Oracle/Sun 61862.77 BBops/min $28/BBops

    IBM 44294.97 BBops/min $23/BBops

    Oracle/Sun 36122.60 BBops/min $12/BBops

    BEA 25394.60 BBops/min $9/BBops



    Two other postings, both from BEA, have been revised to reflect improvments in Price/Performance Figures (new FDRs have been posted for these):

    - BEA-HP BBOPS=37,791 (NO CHANGE) NEW PRICE = $36/BBop@Std , down from $38/BBops

    - BEA-SUN BBOPS=16,668 (NO CHANGE) NEW PRICE = $22/BBop@Std , down from $26/Bbops


    Check out the new ECperf results here.

    Threaded Messages (47)

  2. Again I say ... who really cares. The price/performance tests are not useful in their own right. Only when combined with other measurements of application server value including reliability, included api's/products, ease of development, standards support, etc. can a potential buyer conduct an accurate evaluation. Perhaps I don't know enough about the purpose of ecperf, but if I can choose whatever hardware and supporting software I want to run my tests...how can I compare myself against my competitors?
  3. Regarding the usefulness of price/performance tests with
    J2EE, the leading J2EE products are starting to move into
    a fairly 'mature' phase, where issues such as APIs,
    standards are getting more difficult by the day to
    to use as differentiators in any evaluation.

    I would suggest that price/performance results today are
    much more meaningful than they would have been 12 to 18
    months ago when the various products were more
    differentiated on features alone.

    Regarding how one goes about 'comparing' with one's
    competitors, one can do things such as submitting results
    on a hardware/OS configuration that matches a previous
    submission, as you will see was the case in the Sybase
    submission.
  4. ECPerf has improved the quality of the J2EE servers, just as SPEC improved OSs, compilers and hardware. When vendors turn to the "code only for the benchmark" approach, it becomes pointless, but so far I can attest to the improvements in at least two of the aforementioned products due directly to the existence of ECPerf.

    I would like some time to have enough spare hardware and time to do a few runs myself, since I don't think I'd be daring (liability and all ;-) enough to configure the servers as described in some of the results.

    Peace,

    Cameron Purdy
    Tangosol, Inc.
  5. Thought this might interest a few...

    You can tune into this free Oracle Internet Seminar on July 17,2002 at 8.00am PDT, 15:00 GMT or view on-demand (after July 17th) to see how Oracle9i Application Server achieved these ECperf results. There is a live Q/A session following the presentation, and you are encouraged to call in with questions.

    Vineet Buch, Oracle Director Performance Product Management, Oracle Server Technologies will discuss how Oracle9i Application Server achieved record breaking
    performance and price/performance.

    To view seminar live at 8am PDT on July 17, 2002 please visit http://www.oracle.com/iseminars .

    This seminar is archived and will be available on-demand following the live event. View seminar on demand (after July 17th).

    -
    Sudhakar Ramakrishnan
    Principal J2EE Evangelist
    Oracle Corporation

  6. Has BEA submitted any ECPerf results using
    the BEA JRockit JVM?
  7. Has BEA submitted any ECPerf results using

    the BEA JRockit JVM?

    Yes - the last set (some 6 - 8 weeks ago?) that they submitted did used it.

    /david
  8. Why does oracle9ias not full Compatibility with J2EE?

    What does some J2EE function not support in oracle9ias?

    look: http://java.sun.com/j2ee/compatibility.html


    thanx.
  9. It seems to me that Oracle is the top player in J2EE at the moment.
    I tried JDeveloper9 a few weeks ago...nice tool for free! They bought TopLink, the leading O/R mapper and now they are leading both categories in ECPerf performance and price/performance...wow!
    They show that even big companies can change there strategies very fast and that BEA and IBM are not the only ones who will survive and who are setting new landmarks.
    I am wondering when BEA and IBM will fight back...and when somebody will sponsor a JBoss ECPerf test.
    I like the Sybase results. It's good marketing. I haven't known that there is a Sybase AppServer :-)

    <Mirko/>
  10. Mirko wrote:

    ...
    I like the Sybase results. It's good marketing. I haven't known that there is a Sybase AppServer :-)

    <Mirko/>

    ...

    And yet, Sybase EAS (formerly Sybase Jaguar CTS) has been out for better than five years, and was arguably the first commercial product that would be recognized as an application server by today's standards.

    Sybase always seems to be long on technology and short on mind share.

    -Tommy Phillips
    (Yes, I do work for Sybase).

    PS- I did use the word "arguably", but please don't treat this message as an invitation to argue. Mostly, I'm griping about our "stealth marketing".
  11. Sybase again proves to be the "Jason" (of "Friday the 13th" the movie fame) of the database and application server markets: it just won't die. All that's missing is the goalie mask...
  12. Good news for Oracle, even better news for Orion! Oracle don't like to make a big thing about it, but OC4J is actually Orion rebadged. If you don't need Oracle's size and support contracts (or their prices), check out www.orionserver.com.

    -Darren
  13. Darren,

    You are correct Oracle did license OrionServer about 12 months ago but I would caution against interpreting the ECperf results achieved with Oracle 9iAS today and what may or may not be able to be achieved (hypothetically) with OrionServer. I believe there are now very large differences in the codebase and I can think of at least 2 areas where this means that there are significant performace improvements in the Oracle product over what they originally received from Orion.

    Before Orion fans start beating me up I would just add that I am not making any comments or drawing any conclusions about the current Orion product I am just commenting on 9iAS.

    Tom

  14. I have done some extensive analysis of all of the submissions and came to the following conclusions;

    -IBM is putting out some really funky and unrealistic configurations that do not seem like anything anyone I know are putting into production. 12x2 CPU Linux boxes? Seems like a benchmark special to me. IBM's lack of #s for AIX, AS/400 and the mainframe platforms are highly suspicious. I suspect they would publish them if they were competitive, but I guess they aren't.

    Also, IBM's Passport Pricing that they use assumes that you are purchasing IBM hw, O/S and middleware. If you substitute anything, you don't get this big discount they are applying.

    -Oracle, while jumping into the game with some good numbers, is up to their old tricks again. This licensing scheme is something that should be banned by the benchmark committee. How can you compare 2 year term licenses with the perpetual licenses that BEA and IBM are using??? Sketchy at best. They are also using a pre-release driver, so again, comparisons are hard.

    I like the BEA numbers. While not showing the *absolute* leadership in either total throughput or price/performance, they represent the best REAL WORLD configurations, showing off Windows, Linux, Solaris and HP-UX, without any pricing specials that IBM and Oracle are doing. Anyone can put a huge hardware comparison together and get a big number. It's doing it in a real world configuration, without any funny pricing games that I'm looking for.

    As for Sybase, Pramati and others...market share and vendor viability are enough to keep me away.

    The net: these vendors should all be looked at with degree of suspicion. ECPerf is about Marketing. For now, I tip my hat to BEA for keeping it real.

    Harvey
  15. <quote>
    As for Sybase, Pramati and others...market share and vendor viability are enough to keep me away.
    </quote>

    I always find this kind of thinking interesting. It can be rephrased to say "new innovative players who aren't as big as these major companies can't play". If we automatically push minor players out of the market, then what we get is a monopoly by the larger players in a technology. Once that occurs, the major players can push a market in to stagnation, lowering prices and reducing expectations. Is that what we want?

    How can developers be so independent, and yet flock so willingly to the biggest player on the block?



    Jonathan
  16. <How can developers be so independent, and yet flock so willingly to the biggest player on the block? >

    Hey, you are forgetting something here. We are not in college any more, and this is not a game for some computer geeks. We are doing some serious stuff here after all.

    Companies are betting their core business on these products, and vendor's viability, market support, past history, and future directions are after all what counts the most. It's not about performance or price/performance after all, even though these are definitely helpful.

    It's about whether your company, and the wendor who supplied the app server to you, will still be doing business together in 1,2,5,10 years from now.

    For example, by looking at BEA's history with Tuxedo, success with WebLogic, other products, their market lead, their support by developers, their standards compliance and technological superiority - I'd say - most likely, yes.

    IBM - probably yes too, even though for them dumping the product they don't think performs the best wouldn't cost much for the rest of their business, after all all they care about is selling you more hardware.

    Oracle, with their third (or, is it fourth?) code base for their app server, the image of their CEO, tricks they've used to get those ECperf results, and interest in database business - just not good enough for a serious business.

    The rest - Sybase, Pramati, and so on - just look what happened with the Bluestone (just got dumped by HP in favor of BEA). This market is shrinking in number of players, and you want to stay with those who are the fittest. Can't see some airline, or a bank, or a stock exchange, or telco basing their business on one of those smaller players. Just too much risk.

    My 2c,

    George.
  17. <quote>
    The rest - Sybase, Pramati, and so on - just look what happened with the Bluestone (just got dumped by HP in favor of BEA). This market is shrinking in number of players, and you want to stay with those who are the fittest. Can't see some airline, or a bank, or a stock exchange, or telco basing their business on one of those smaller players. Just too much risk.

    </quote>

    Sybase should not be lumped together with the Pramati and Blustones , of the world -- and yes both the exchanges, major brokerages and banks rely on Sybase DBMS for mission critical apps that require stability and performance. NYSE recently listed Sybase as a stock because they see us for a long term player -- we've been around for 17+ years and we are not going anywhere.

    my .02 $

    Nick Rapoport
    (Yes I also work for Sybase)
  18. I stand corrected on the term license for the Oracle App Server. The fact that it is 1 year is even worst for the customer. The perpetual license for the app server is listed at $20k/CPU, which is what the proper comparison is to the other vendors. That's all I was trying to say. I see Oracle is promoting these numbers as apples to apples. In my opinion its deceptive at best.

    As for Sybase, I agree, long term vendor viability isn't the right issue. It's the fact that according to every analyst report I read, they have sub 4% market share which is the issue. My point was the long term viability of them in this space. Just like with the RDMBS, as they are acutely aware of, once the market tips, all of the packaged applications, surrounding technologies (management, security, development tools, etc.) all support the top 1 or 2 players. The same goes for the developers and administrators. This ground swell drives the local User Groups, Books, Consultencies, best practices, etc. that continues to fule the leaders and repress the laggards. Not to be down on Sybase, but the horses have left the barn and they aren't part of this race.


  19. "
    NYSE recently listed Sybase as a stock because they see us for a long term player -- we've been around for 17+ years and we are not going anywhere.
    "

    beg to differ ... Sybase shifted to NYSE from Nasdaq only as a favor for having gotten a few deals from NYSE. This has nothing to do with technology!

    being around for 17+ years is hardly an achievement. What have you achieved in those 17 years? An also-ran dbms that is languishing in the single digit market share, an app-server that is almost as good as dead! The only saving grace is your mobile db. Whatever happened to powersoft? Is it still around?

  20. They've announced that they are in an early development phase for a new PowerBuilder version that will support both Java and .NET.

    Peace,

    Cameron Purdy
    Tangosol, Inc.
  21. Could app server vendors who participate in these discussions please make sure you draw that out in your signatures? It makes it alot easier for developers watching these groups to understand the spin attached to each post if they come from an employee of a vendor. Its only fair ;)

    Dave Wolf
  22. With all the posts stating that these results are 'meaningless' due to the rules and cheats every benchmark has, I think its important we do not miss the forest for the trees here.

    The reality is the difference between $6 a tx and $5 a tx is basically meaningless. Which in and of itself is the actual importance of the results. What mean is that with all the latest results we as developers can now see that the one slide all of our vendors has about performance in their sales slide deck is basically useless, since for all intents each of the vendors is delivering nearly identical persormance.

    So, when making a purchasing decision of a vendor who has posted ECPerf results, you can more easily dismiss concerns about performance when making your purchasing decisions. Should you choose the vendor who is #1 in performance? Not necessarily. But neither should you dismiss a vendor in the #4 slot.

    This is a pretty important result of the tests that is going under emphasized. As a developer who might be struggling with an app server and its performance, if that app server has results that are 'in the running' we can step back and likely decide the app server itself isnt our performance concern.

    So it is very important to see that your vendor has posted numbers, but less important if that result is #1, #2 or #N, but that the results are ' in the running '.

    Sun wanted to create a missle race and ECPerf was their cold war kick start. It worked. In the end though, it doesnt matter who has how many missles.

    Dave Wolf
  23. Harvey,

    A few corrections.

    (1) ALL submissions except for IBM and Sybase used 1-year
        term licenses for the Oracle DBMS (this includes BEA,
        Pramati).

        (IBM used perpertual DB2 licensing, Sybase used
        perpetual ASE licensing)

    (2) All submissions running on Oracle9iAS used 1-year term
        licenses for the application server. All other vendors
        used perpetual licenses for the application server.

    You can do an interesting price-performance comparison using
    just the hardware price (excluding the software price).
    Since the ECperf review committee didn't allow hardware
    leasing, the hardware price/performance metric is currently
    quite valuable if you are attempting to make
    apples-to-apples comparisons.
    Finally, you questioned the viability of Sybase as a
    company. I'll have you know, we are alive and well, and
    working hard on pushing our market leading price-performance
    further, to the point that it can't be beaten even by
    vendors who choose to play tricks with 1-year term limited
    licensing.

    Stay tuned for SPECjAppServer2001!

  24. "...and working hard on pushing our market leading price-performance further, to the point that it can't be beaten even by vendors who choose to play tricks with 1-year term limited licensing."

    For the record, this is simply untrue (and it's something Sybase incorrectly claimed in its press release as well): Oracle is the price/performance leader at $5/BBops/min (vs. $6/BBops/min for Sybase).
  25. Sekhar,

    The Sybase press release referred to a single-node
    application server configuration, and $6/Bops/Min@STD is
    the best price-performance that has been published in this
    configuration.
  26. Alright, guys, that's enough or I'm going to have to post ECPerf bbop/$ numbers for Tomcat4/JBoss3. ;-)

    (Isn't it great to see Oracle and Sybase slugging it out again?)

    Peace,

    Cameron Purdy
    Tangosol, Inc.
  27. Sekhar (who works for Oracle) Ravintulalalalalalala:

    Enuff said dude ... you can publish all the ECperf numbers up the wazoo and nobody would care! ECperf has turned out to be yet another bogus benchmark with outlandish rules that don't resemble anything close to reality.
  28. Evan and Sekjar Ravilalala:

    Hey ... you can parse the results anyway you want!

    This precisely proves my point why ECperf is a ludicruous
    benchmark.
  29. It seems to me though that the support for the orion server has been pretty low for the past several months (since they sold the software to oracle :-).
    I'm beginning to wonder if they will continue to support the software at all.
    I see nothing new on orionserver.com;
    no new versions (I check every day for a new version, be it not stable, and especially for the ejb2.0 full compliance).

    I'm asking (myself):
    - do they intend to support orion anymore or they just gave up ?
    - do they wait for oc4j to implement the ejb2.0 specification first ? (to see if they are able to do it :-). Oracle does not seem to be able to keep up the pace by the way:
    a) they have a tradition of estimations of releases that are delayed (a lot).
    b) oracle's best java products are developed by 3rd parties: JDeveloper (based on Borland's; I think personally that the latter is a little bit better than the original) and oc4j (based on orion). I would mention also their recommandations to NOT use oracle's original j2ee server, which is probably going to be abandoned (it is called something like bc4j? i'm not sure. It still ships with 9ias, still is not used/usable).

    I personally noticed only two changes from orion to oc4j:
    - change of name of the app server :-)
    - a few bugs in orion dissappeared only to make place for others. I agree, most of the bugs appear for features that are not documented at all, like support for relationships; orion 1.5.4 claims (?) not to support relationships, still it does, with some bugs. (Or btw, hot redeployment does not work in all cases.)

    We are of course extremely interested in the policies of the 2 :-) app servers - orion and oc4j;
    Like: who is going to develop further the system,
    who will provide the best support etc.

    Stefan Morcov
    SIVECO Romania
  30. Sorry about saying that orion 1.5.4 claims not to support relationships; I think it is 1.5.2;
    they say that the 1.5.4 version does partially support them. I noticed 1.5.2 does too. Am I wrong ?

    Stefan Morcov
  31. Oracle9iAS/OC4J 9.0.2 is 100% J2EE 1.2 compliant with significant 1.3 functionallity suport.

    100% J2EE 1.3 compliance will come with 9.0.3 by the end of August, I think.
  32. How does Oracle get by with the pricing they are quoting in the benchmark?

    Even the 2-year lease option at the Oracle Store is more than they are claiming in the FDR. that price is $2000/cpu. At the Oracle Store the 2yr option is 7000/cpu!

    They must have just licensed the Rdbms and the app server just long enough to run the test!

    Matt

  33. Matt: "They must have just licensed the Rdbms and the app server just long enough to run the test!"

    Touche! In related news, BEA and IBM announce sub-$1 BBOPs. Today only!

    Peace,

    Cameron Purdy
    Tangosol, Inc.
  34. In response to what 9iAS is missing: Mainly a few EJB specs:

    Local Interfaces, EJB-QL, RMI/IIOP, CM Relationships.

    But, they just came out the an updated version (9.0.3) which supports the full spec...it's out on otn.oracle.com.

    They basically say this: Because of their total integrated 9iAS stack, the integration piece takes longer and other point products don't have this issue. Also, the other products that are 1.3 certified were only for beta versions.
  35. My testing shows 9ias does support CM relationships, even since previous versions, maybe it is not documented (like the orion 1.5.2 :-)).

    Btw: you can see in their known issues for the latest version that relationships are not supported unless you let oc4j create the tables, including some 3rd relationship table.

    <quot>When deploying entity beans which have relationships defined, the autocreate-tables attribute must be set to true. The current implementation requires the use of association tables to represent relationships and these must be created by the container at deployment time.</quot>

    Actually, it is a lot of workaround to specify your own relationship table, still I was able to do this repeatedly, by observing what it actually creates. Then I even gave up the 3rd table that they say somewhere it must be automatically created, and simply mapped my plain 1-many relationship from the table with some hard-work.

    Stefan Morcov
  36. Hi all,

    I can see what all the brouhaha is about the results as it relates to the price used to compute the overall results. After going to Oracle Store and looking at the $15,000 price per CPU vs. the 3,000 used in the test, and doing the calculations, the Oracle results jump up to 6.8 and higher after adding the additional support costs.

    What a lot of you seem to miss is that Oracle Standard can't hold a candle to DB2 or Sybase Enterprise version of the database and the really useful replication and other features they have. If you were to switch to Oracle Enterprise, the Oracle results jumps up to around $9 per transaction and is more representative of DB apples to apples comparisons. When Sybase 15 and IBM releases their latest DB version, Oracle 9i marketing will have to switch to something other than performance and easier management.

    Companies learning how to use Linux and learning how to judge a DB and app server on merits, not propaganda, will have the profitability and efficiency edge. Large companies trying to make all their applications run under "premium brand" monolithic DB, app. server and hardware platform are going to get killed by companies like the one I work for that use inexpensive solutions where possible (95% of the time) and the apps work just as well.
     
    Maybe that's why I've noticed a lot less job listings for Oracle and BEA, and more for Sybase, MySQL and JBoss lately. The last couple of years have shown that buying the most popular product (expensive) doesn't translate to lower operational cost or better operational performance (Today's highest priority to improve profitability).

    Kevin...No I don't work for any of the companies listed.
  37. All the noise about the results, is allways very funny. The thing that matters at the end of the day, is which product would you choose if the budget and responsability is yours.

    IBM´s products include "bundle" costs...
    Oracle´s product is the best RDBMS

    Our experience is much better, less TCO costs, more performance, more robust on weblogic... (real experience big system)
    regards
  38. All the noise about the results, is allways very funny. The thing that matters at the end of the day, is which product would you choose if the budget and responsability is yours.

    IBM´s products include "bundle" costs...
    Oracle´s product is the best RDBMS

    Our experience is much better, less TCO costs, more performance, more robust on weblogic... (real experience big system)
    regards
  39. When Sybase 15 and IBM releases their latest DB version, Oracle 9i marketing will have to switch to something other than performance and easier management.

    As a beta site for ASE 15, We are finding Sybase's RDBMS features to be dated. Much of the hype features of ASE 15 are just that hype. We have done some benchmarking against Oracle and DB2 inhouse and simply put, Sybase ASE 15, comes up short. If Sybase is betting on this to get them back into the RDBMS market, they better think twice.
    Maybe that's why I've noticed a lot less job listings for Oracle and BEA, and more for Sybase, MySQL and JBoss lately. The last couple of years have shown that buying the most popular product (expensive) doesn't translate to lower operational cost or better operational performance (Today's highest priority to improve profitability).

    Kevin...No I don't work for any of the companies listed.

    And while true that there seems to be more job listings for Sybase lately, most of them are conversion jobs to Oracle, DB2 or MySQL.

    And no I do not work for any of these companies either.
  40. No newcomer I am sure[ Go to top ]

    Welcome back old Sybase friends!
  41. No newcomer I am sure[ Go to top ]

    From what I have seen is EAServer really a cool thing.

    So bad that there is not such a huge user base out there than for some of the other servers, even though the developer version is a free download.
  42. One thing the postings are missing is whether version 1.0 or 1.1 of ECPerf was used. I think I am correct in stating that neither of these requires J2EE 1.3 compatability.
  43. All the posted results were with ECperf 1.0.

    J2EE 1.2 (or 1.3) certification of products is required.
  44. this is ECperf 1.0

    best==>
    prashant
  45. Hey all you IBM'ers out there. I am interested to know something. Why does IBM do thier ECperf benchmarks for WebSphere on Windows 2000!!! and AIX 5L Unix (DB2 database)?

    They have a strategy for so called "right-sizing" that they are trying to push into industries like FSI to protect thier patch from the onslaught of Unix. Basically Linux on S390 (AKA Raptor). WebSphere is also available on this platform. They are also quite focussed on Linux and making it available as an OS option on all thier hardware platforms.

    Correct me if I am wrong but the processing power of mainframes is theoretically way beyond that of the largest mainstream Unix based server systems available today.

    If IBM is serious about protecting thier patch and keeping the FSI marketplace under wraps then why don't they post an ECperf benchmark using S390 WebSphere and DB2?

    Also, if they are serious about squeezing Microsoft then why the hell do they use a Windows 2000 operating system (instead of Linux which is arguably more scalable) for thier application server rather than thier supposedly superior RS6000 and AIX?

    Maybe the mainframe is not all it is cracked up to be...

    Maybe the RS6000 is also not what it is cracked up to be...

    I don't want to make any comments about my assumptions. What do the IBM'ers think?

  46. Congratulations to Sybase for sliding into the leaders list. Nice to see my old friends still kicking ass.

    Dave Wolf
  47. I thought I read somewhere that the deadline for submitting ECperf v1.0 results was June 17 2002. Is it that these new results have only just been approved by the Review Committee or am I wrong about the submission window?

    Also, does anyone know anything about the SPECjAppServer roadmap? (or why there is so little information available?!)
  48. The deadline was June 17, and these results have just been approved by the review committee.