New ECperf Results: IBM Bounces Back with Best Price/Performance

Discussions

News: New ECperf Results: IBM Bounces Back with Best Price/Performance

  1. New ECperf results show that IBM Websphere has taken the lead in the Price/Performance category with an astonishing $13/BBops (BEA Weblogic had $18/BBops). Performance-wise, Websphere came within 62 BBops/min@Std of Weblogic. I called up Matt Hogstrom, ECPerf J2EE Performance Specialist at IBM, to get some more insight:

    Nitin: Your results have improved dramatically since your last ECPerf tests in January. What were the decisive changes you made? How was IBM able to achieve such a low Price/Performance figure?

    Matt: "Well, there are several reasons that these results are better than our first submission. First, IBM is committed to performance improvements in our product as part of the product lifecycle and these results reflect those improvements. This includes improvements to WebSphere itself as well as our JVM technology. These results are using the 1.3.1 version of our JVM. Also, these results are based on a different hardware platform than the January submission. Given the difference in processing power of the hardware the boost in performance is expected."

    Nitin: Performance-wise, you fell short about 60 BBops of BEA. Does your exceptionally low Price/Performance figure overshadow this fact?

    Matt: "Yes, ECperf is tested by setting an Injection Rate. These results used an injection rate of 161 and so any submission at this rate will yield approximately 104/BBops per Injection Rate. Missing the mark by 60/BBops a minute isn't really significant since the overall BBops are over 16,000/minute. What we would like to show customers that Websphere performs just as competetively as WebLogic. It is interesting to note that for these tests we did as much work as Weblogic but with less hardware using Websphere. We used approximately 60% of the physical memory on all systems, half of the application server instances, no processor affinity required, 1/4 of the Java heap, and half the number of processors on the database as the WebLogic submission. With WebSphere you don't need as much hardware or complicated setup to complete the same amount of work. This translates into a lower cost of ownership and simpler operation than compared to the BEA submission."

     
    Check out
    http://ecperf.theserverside.com/ecperf/

    Threaded Messages (100)

  2. Here we go. Let the flip flopping begin!
    Good job to the IBM team though.
    I wonder how much of the price/perf is due to bundled
    software? ;)
  3. Further to Matts comments[ Go to top ]

    Both WebSphere application server boxes and the database box use 2.5Gb of RAM versus 4GB for the WebLogic ECPerf run as well as the database box (DB2) using only 2 CPUs rather than the 4 required by Oracle in BEA/HPs ECPerf submission.

    We can use less RAM on the application server because WAS would appear from the published numbers to scale better on SMP hardware than the competition. BEA, despite running 2 instances with each instance pinned to 2 specific processors and significantly more RAM as a result, only gets a 0.003% performance advantage over simply running a single WAS instance on the 4 CPUs.

    We're using practically the same hardware platform as BEA used for their numbers, albeit with less CPUs and less memory ;-)

    This is probably a unique event which we have two benchmarks on the 'same' hardware platform (ignoring less memory and CPUs on our part) allowing an apples to apple comparison of two competing software stacks (BEA V7/Oracle against WAS V4/DB2) on both performance and price. So, if customers want an apple to apple comparison then I believe this is probably as close as they will get.

    So to summarize:
      * Less hardware required for the same performance level.
      * Less dollars required for the same performance level.
      * No playing around with CPU pinning etc.
      * Better vertical scaling.
      * IBM's number's are using RMI/IIOP, the industry standard
        protocol.
      * DB2 supports this level of performance level
        with half the CPUs and almost half the memory when
        compared with Oracle.

    Billy

    (I work for IBM but my comments and opinions don't reflect the views of IBM).
  4. Further to Matts comments[ Go to top ]

    ok ... all the IBM trolls will soon start popping up here :)

    As I mentioned in my earlier postings, just try telling IBM that you want to use WebSphere with Oracle as the backend and watch the WebSphere prices start increasing exponentially. If IBM really wants to play this straight, they shouldn't indulge in fancy nimby-pamby pricing.

    Of course, knowing IBM this will never happen!

    Pramati rules!!!!!!!!!!!!!
  5. Trolls[ Go to top ]

    Andy,
    I'm hurt, "Trolls" ;-)

    Ignore the pricing if you want, this shows we use less hardware than their solution for the same problem.

    Billy
  6. Benchmark specials anybody?[ Go to top ]


    This is getting interesting. First, they have a fancy pricing scheme and now, they have benchmark specials! On page 3 of the FDR, there is mention of a cmpopt tool that was used to optimize ejbStore operations based on the application access patterns!

    Whoa!!!!!!!!!!!!!

    Doesn't this violate the spirit of the benchmark? I thought the ECperf reviewing community was against allowing benchmark specials and avidly disallowed any hints based on advance knowledge of the application execution profile! Why have they capitulated to IBM in this case?

    As far as I'm concerned, this isn't good news if every vendor starts implementing benchmark specials for ECperf. Very soon, this will start resembling TPC-C where transaction rates of 400,000/minute are routine and bears no resemblance to reality whatsoever.

    I had great hopes for ECperf to be an independent benchmark, for a brief while it looked as if this might be it ... but now I'm not so sure after this IBM result which has singlehandedly destroyed any credibility claims that ECperf may have had.

  7. Benchmark specials anybody?[ Go to top ]

    Andy,

    Just to set the record straight (doesn't sound like you had a chance to complete your analysis). CMPOpt analyzes the java bytecode of the individual beans and not their interactions. This means optimizations are not based on how the driver runs, simply what the beans do. We save the results of the analysis (ie. this method only refers to CMP fields as opposed to updating them) and use this information at runtime to optimize performance. The analysis is limited to the remote method invocations on individual beans and quite comprehensive.

    Hope this helps.
  8. Benchmark specials anybody?[ Go to top ]

    Matt:

    I presume this "optimization" phase happens outside the measurement interval of the benchmark, yes? If so, I would still consider this a "benchmark special" as the previous poster put it. Without the use of this tool other app servers could be unfairly penalized as their measurement interval includes the runtime optimization.

    Wasn't the goal of ECperf to provide a level playing field by not allowing the use of these tricks?
  9. Benchmark specials anybody?[ Go to top ]

    The issue is whether an Application Server has technology that a customer can use at deploy time which makes the runtime faster. If the tool was specific to ECperf then your comments are accurate. However, the tool is shipped with WebSphere and is a deployment utility that analyzes EJB 1.1 jars (of which ECperf is one EJB 1.1 application). Characerization as a benchmark special does not have any merit. The utility simply does analysis at deploy time and saves the results of the analysis so the run time is optimized. Those methods that change data ensure that ejbStore is driven and those that simply refer to data tell the container a store is not necessary. This doesn't violate the ECperf or EJB 1.1 specifications and produces accurate results. I think its quite clever. Think of it like the optimizing COBOL compilers of yester-year where more time was spent analyzing the COBOL code so more efficient assembly code was generated to optimize the actual runtime. (CMPOpt DOES NOT modify bytecode!!!) The result is a faster runtime and isn't that the name of the game?
  10. Benchmark specials anybody?[ Go to top ]

    http://www-106.ibm.com/developerworks/ibm/library/i-optimize/?dwzone=ibm

    If I ever needed proof that this is a benchmark special, here is it! I rest my case.

    Q. What platforms can be used to run the CmpOPT tool?
    A. Because this is a technology preview, we've limited the tool to the Windows platform.

    Q. Can I use CmpOPT on EJB 1.0 JARs?
    A. No. The utility is specifically designed for EJB 1.1.

    What about EJB 2.0? Oh, never mind, we're talking about WebSphere here :-) The name of the game is higher throughput, but not as a result of circumventing the rules or inventing seemingly clever benchmark specials.
  11. Benchmark specials anybody?[ Go to top ]

    The utility is included with WebSphere and supported on all platforms WebSphere is. The article was to make it available to our customers before inclusion in the product.

    BTW, Pramati makes a very respectable AppServer as well.
  12. Benchmark specials anybody?[ Go to top ]

    http://www-106.ibm.com/developerworks/ibm/library/i-optimize/?dwzone=ibm

    "
    CmpOPT runs for a long time. This is because the process of analyzing multiple classes is time-consuming. It also requires significant amounts of JVM heap memory.
    "

    How long is "long"? Your FDR says the ECperf benchmark measurement interval was 30 minutes. Surely, if you had factored in this "long" time into the benchmark, your results would have been lower (due to the occurrence of full GC!). Billy made some tall claims about WebSphere using less memory than anybody else on the planet -- guess he forgot to add in the "significant amounts of JVM heap memory" needed for this benchmark special tool.

    "CmpOPT is currently supported on the Windows platform only"

    What if I have a Solaris (or God forbid, an AIX) setup? I need to add the cost of an additional Windows box to run this benchmark special tool. Granted they are cheap, but will that not add to my overall system cost? Again, Billy employed some very creative math when he claimed that WebSphere uses less resources -- guess he forgot to add in the additional box!

    "
    The analysis is static. Changes to class files might invalidate previous results. If you make changes to the application, CmpOPT must be re-run against the target JAR file to update the results.
    "

    Let's see -- it runs for a "long" time, uses a "lot" of heap and "needs" to be rerun every time. So inspite of your protestations to the contrary, it does make use of the prior knowledge of the applications execution profile!!! Sounds like the perfect description of a benchmark special.

    All I'm saying is that this result doesn't appear to be "trustworthy". Has too many clever tricks which doesn't make me feel very comfortable trusting IBM's claims of superior performance.

    Pramati still rules!!! :-)
  13. Benchmark specials anybody?[ Go to top ]

    Does anybody care about these benchmark numbers during the application server selection process?

  14. Benchmark specials anybody?[ Go to top ]

    <snip>
    Pramati still rules!!! :-)
    </snip>

    Andy,

    I & Pramati are certainly flattered at the above reference. But I dont understand the context.

    Cheers,
    Ramesh
    - TeamPramati
  15. Benchmark specials anybody?[ Go to top ]

    <snip>
    I & Pramati are certainly flattered at the above reference. But I dont understand the context.
    </snip>

    nor do I ;-)
  16. Benchmark specials anybody?[ Go to top ]

    Andy,
    We gotta be talking past each other here. But, you don't seem to understand what the tools does or how it works. Please read the article about it on developer works and you should understand it better. I hope Matts amplifications help also.

    You saying that for, say Spec2000 a well known benchmark, we should include the compile times with the benchmark run times so that vendors with very good (and usually computationally expensive optimizers) get penalized for making products that make faster running applications? Is this really what you are claiming?

    All the 'claims' stand up that have been made, look at FDAs from the submitters and everything to back them up is there.

    Have a great evening.
    Billy
  17. Benchmark specials anybody?[ Go to top ]

    "How long is "long"? Your FDR says the ECperf benchmark measurement interval was 30 minutes."

    Look, EJB compilation steps are often slow. The optimizer is just an add-on to that step. It's a build step. Not a runtime step. It wouldn't matter if it took 24 hours.

    Do you use Pramati? How does it perform in terms of ecperf?

    Peace,

    Cameron Purdy
    Tangosol, Inc.
  18. Benchmark specials anybody?[ Go to top ]

    Wasn't the goal of ECperf to provide a level playing field by not allowing the use of these tricks?


    ECPerf requires that you do not modify the _application_ in any way. The exact same application is tested in every case. Other than that, any vendor optimizations are fair game. That's the whole point...
  19. <quote>
    ECPerf requires that you do not modify the _application_ in any way. The exact same application is tested in every case. Other than that, any vendor optimizations are fair game.
    </quote>
    Not entirely correct. Meaning, the vendor optimizations used must ensure that the Ecperf specification reqts are honored at all times. For instance the spec requires commit-option A not be used- so any opts that result in using locally (avoiding DB calls) cached objects is not valid. Or, for instance optimizations to avoid ejbStores are disallowed, so any such opts also must not be used.

    These reqts ensure that the application developer can expect identical semantics & behaviour from all vendors. In short, the optimizations that will be allowed are internal opts that get kicked in automatically without altering any semantics that applications may expect.

    Cheers,
    Ramesh
  20. CMPOpt[ Go to top ]

    More information on this cool technology can be found here in a developworks article from last year by Matt. Customers can use this utility to optimize their own CMP applications also of course. Check out the article, clever stuff.

    We basically added it in using an external utility rather than the main deploy code path due to the timing of when this feature was available etc. It'll be integrated in the next major release. Similar technology is already integrated in the latest competitor products as you may have already guessed.

    Billy
  21. I have had the opportunity to look at performance issues in J2EE and would like to provide an explanation of what the motivation behind the optimization (CMPopt) is.

    The EJB container is responsible for calling ejbLoad() to load an Entity Bean and ejbStore() later to persist the object. You may wonder what happens with a ECperf transaction such as "orderStatus". This transaction is read-only and is intended to provide status information. Will the container call ejbStore() even though the OrderEJB is not modified?

    In the old days, the surprising answer was yes - the EJB would be written back to the DB. It was widely acknowledged that this bad and the fix was to use a flag - isDirty or isModified.

    Modern containers such as Weblogic6 the container will make a copy of the object and check field by field to see if it has been modified. Only if it has been modified will it be written to the db. You may think that this is overkill, but please remember that writes are to be avoided in transactional systems since they lead to increased lock contention.

    WebSphere provides a much more elegant yet problematic solution. One can demarcate transaction semantics of a method such as getOrderStatus() to be read-only. In the C++ world this is like marking a method as "const" - nothing gets changed. This allows the container to skip any calls to ejbStore(). I think this is an excellent solution except that it creates the following challenge: who decides what transactions are read-only? If you leave it to people, they will make mistakes. It is also a tad tedious. CMPopt monitors runtime behaviour to figure out what the rights settings are. I think this is a reasonable approach to this problem and will probably get less clunky in future versions.

    In closing, I think that this optimization is consistent with good practices when using Entity Beans. Further this could even be construed as a weakness in the specification since it does not address this issue and leaves it to application-specific extensions.
  22. Good explanation Michael and basically right on in all aspects except that we don't monitor the runtime behaviour like a profiler for example does, we instead do a static analysis on the jar beforehand public ejb method by ejb method and thus CMPOpt has no direct runtime impact or memory impact. It, of course, has an indirect effect in that your application runs faster without you needing to hand tune the application for these particular optimizations.

    Billy
  23. In future versions clunky tools like CMPopt will probably disappear - CMP 2.0 allows container to perform all these optimizations automagically.
  24. Hello everybody,

    this is the same heating discussion as with the Bea-ECPerf Benchmark. Every camp (Bea vs. IBM) here accuses the other that they has used cheap tricks to get their published results. IMHO it's absolutley boring and of no help.

    For me, who is not an adherent of any camp, this ECPerf-results only shows that with tuning and choosing the right tools J2EE is able to compete in cost and performance.

    Just my 2 cents
  25. Two Words: Who Cares?

    This is about as useful as those hokey RDBMS tpc-c benchmarks.

    Reality is that Java middleware is evolving beyond the application server. J2EE is evolving too slow and meanwhile the battle is moving to integration, content management, portal, Web services, etc.
  26. I agree with your post: this is good news for J2EE and for enterprise application managers who have chosen Java over other technologies like .NET. Of course, this is only showing a few abilities: performance and price, which by no means are the only metrics one should use when choosing an application server.

    The other remark we can do here is that none of those high-scorers used any Sun hardware, which can be interpreted as a bad news for Sun. The only configuration using Sun was Borland's and it proved extremely expensive for a poor performance. In my own experience, Sun hardware seems to be the number one platform for deploying Java enterprise applications. Knowing that hardware is their core business, they have much to lose, which would be sad knowing how much they have contributed for Java so far.

    Now, knowing that both BEA and IBM application servers can be compared if well-tuned, is it correct to conclude that for Oracle back-end systems, it is recommended to use WebLogic, whereas for DB2, it is recommended to use WebSphere ?

    It's probably also important to note that the IBM solution smells like a potential vender lock-in at all server levels. Some may feel comfortable with such a solution because it offers better overall integration of the different components. Others may not like that, it depends upon the projects, the people.

    Instead of resorting to the same old ludicrous religious anti-productive biased arguments for bickering, I think you guys (IBM/BEA/...) should focus on increasing your J2EE market against .NET and steal user-friendliness monopoly from Microsoft for instance (*). At least, war for freedom of choice is a more valuable argument if you really want to play wargame. The bigger the cake, the bigger the shares, there's room for everyone in J2EE.

    (*) By the way, I noticed some nice efforts were done on WebLogic WorkShop's UI. It's fast and well designed, good job! That's sort of what I mean by user-friendliness.
  27. <quote>
    In closing, I think that this optimization is consistent with good practices when using Entity Beans. Further this could even be construed as a weakness in the specification since it does not address this issue and leaves it to application-specific extensions.
    </quote>

    Ahem... hasn't this weakness been addressed a year ago with the EJB 2.0 specification?
  28. Stop this marketing nonsense![ Go to top ]

    Ahem... hasn't this weakness been addressed a year ago

    >> with the EJB 2.0 specification?
    Exactly! I cant work out how a company with the resources IBM has, still takes about 12 months to come out with a PRODUCTION implementation of the EJB (J2EE) specifications.

    I have a fair amount of respect for IBM and Websphere - but all this marketing spin is starting to wear thin. IBM seem to be more intent on winning a marketing war than improving their product.

    I am becoming more and more amazed at the sheer number of IBM employees that are now supporting Websphere on TSS.
    Perhaps the number of IBM employees required to support Websphere on theServerSide forums is indicative of how many are required to support it in production...

    Just develop a good product dammit and it will sell itself!

  29. Stop this marketing nonsense![ Go to top ]

    <snip>Just develop a good product dammit and it will sell itself!</snip>

    Sure. Look at the history of Apples WebObjects and say this again ...
  30. Ahem... hasn't this weakness been addressed a year ago

    > with the EJB 2.0 specification?

    Hi Cedric,
    I'm trying to follow the discussion here. I'm relatively new to EJB, so can you please point me to the changes in the spec you refer to?

    Thanks a lot,
    Pieter.
  31. <quote>
    Modern containers such as Weblogic6 [...]. Only if it has been modified will it be written to the db. You may think that this is overkill, but please remember that writes are to be avoided in transactional systems since they lead to increased lock contention.
    </quote>
    If one needs to maintian consistency and data integrity (prevent one transaction overwriting updates from another transaction), the beans have to anyway be locked, from the time it was loaded thru the end of the transaction, regrdaless of whether it is modified or not. (Different matter that most containers most of the time dont do it!)

    <quote>
    WebSphere provides a much more elegant yet problematic solution.
    </quote>
    Just curious: Wonder why a static way of setting read-only is more elegant than a runtime deterministic way (where, if no fields are modified, no store is performed)?

    Cheers,
    Ramesh
  32. Ramesh: "If one needs to maintian consistency and data integrity (prevent one transaction overwriting updates from another transaction), the beans have to anyway be locked, from the time it was loaded thru the end of the transaction, regrdaless of whether it is modified or not"

    Definitely not true. Optimistic concurrency (deferred concurrency control) is usually a much better approach for high-scale systems.

    Peace,

    Cameron Purdy
    Tangosol, Inc.
  33. <quote>
    Definitely not true. Optimistic concurrency (deferred concurrency control) is usually a much better approach for high-scale systems.
    <quote>

    Sure Cameron. (Infact we have implementations (a few flavors) of Optimistic Cnncurrency (OCC) as well.) My stmt was in just that context of locking when updates happen.

    But about being a much better approach, guess it is not that one could use it always. OCC & PCC, the tradeoff (ofcourse, being in the business you are in, am sure you know this better! :-) is that OCC while gaurenteeing integrity can in that process result in rollbacks if the concurrent access is very high. So OCC can be effectively used only when there is read-mostly access to the beans.

    Cheers,
    Ramesh
  34. Good points Ramesh and Cameron. The issue of optimistic/pessimistic concurrency becomes difficult to resolve because it often needs to be set at the application level, not the bean level. You may want to access an EJB optimistically in one application, but pessimistically in another. Are there any app servers that allow that?
  35. "You may want to access an EJB optimistically in one application, but pessimistically in another. Are there any app servers that allow that? "

    You can deploy the same class as two different beans, i.e. with two different dd's.
  36. I assume you mean concurrent access to the same bean (data). Otherwise OL is still better than PL - that is, if your database uses row locking.

    Re: runtime vs. compile(deploy) optimization.

    Both have advantages and disadvantages. In this case, the advantage is that compile one is faster in production (no overhead of the runtime one). Its disadvantage is that if you have a method that modifies a bean only 5% of time, the compile time would still recognize this method as "rw".
    It might not be recognizable at runtime, as it could be due to data distribution. That's when (smart) runtime optimization can make a huge difference - something like Sun has attempted with HotSpot.
    On the other hand, if you're doing CMP, having your internal isModified/isDirty flag (set propertly by the container) should be quite easy. This would be something aspects would be really great for ;).
    Regards,
    Vlad
  37. Benchmark specials anybody?[ Go to top ]

    Dear Andy, you are the real troll here, not IBM guys. And you do not have a clue - your failure to understand what CMPopt does proves this. Are you one some kind of a personal vendetta against IBM? Did they fire you for incompetence?

    Anyway, I cannot wait for Oracle results. It is supposed to be 4x faster or something ;-)
  38. Benchmark specials anybody?[ Go to top ]


    If you look at the recent posting by George Northon, it confirms what folks here (atleast the non-IBMers :-) ) have been saying all along ... that the pricing IBM used for their recent ECperf publication is somewhat questionable! It appears that you cannot really avail of these prices ie: these are NOT list prices, but are dependent upon buying a fixed quantity of IBM products (aka volume pricing).

    If IBM really wants to play this game fairly, they should disclose list prices and recompute the costs for this submission.

    Standard disclaimer : I work for J2EE ;-)
  39. Data Corruption?[ Go to top ]

    A while ago, i saw a reference to ecperf submissions with data-stomping. Am new to the EJBs; can someone tell me if this data-stomping will happen in all applications or is it only to do with Ecperf?

  40. Data Corruption?[ Go to top ]

    <snip>
    .. can someone tell me if this data-stomping will happen in all applications or is it only to do with Ecperf?
    </snip>

    Data stomping basically means that concurrent transactions access the same bean instance and modify the bean at the same time. In effect one transaction overwriting the changes of the other transaction (imagine this happening to something like bank-accounts or factory inventories!).

    This can easily be avoided by enabling concurrency control where the container ensures that concurrent access to the bean instances are serialized (via locking). Some servers support Concurrency Control by default. And some support it via deployment flags.

    In short, the data stomping need not happen in all EJB applications. The deployer decides if it shd be allowed.

    Cheers,
    Ramesh


  41. Benchmark specials anybody?[ Go to top ]

    Hi,
      Define list price, and I get the price for you. But I can't see it being more complex that the price that an individual would pay if he/she walked up to a vender and just baught a single copy. Also which country/region price would you like? Direct or from a Business Partner, Download or Boxed? Support or no Support? Volume or Single?

    Paul Anderson

    My views are my own and not IBM's
  42. Benchmark specials anybody?[ Go to top ]

    <snip>
    Define list price, and I get the price for you.
    </snip>

    List price is quite simply the price that any customer can get with no expectations of past or future business. And it is not a promotional price. This list does include any package offerings available to all customers.

    Cheers,
    Ramesh
  43. Benchmark specials anybody?[ Go to top ]


    Re: list price

    To add to Ramesh's points, it should also include prices for support that are NOT dependent upon past purchases. I'll wager that the only reason they were able to get this low $/BBops is that they conveniently ignored the support costs (which can be pretty substantial, in reality).

    I hope IBM would step up to the plate and republish this result without all the fancy pricing schemes. I don't really know much about the ECperf review process, but by any chance, were the results audited by the folks from Andersen? ;-)
  44. Benchmark specials anybody?[ Go to top ]

    Oracle results. It is supposed to be 4x faster or something ;-)


    Right, because of shorter code paths (if I remember Larry's words correctly).
  45. Benchmark specials anybody?[ Go to top ]

    Oleg and others:

    Sorry if my postings have come across too strong ... I guess I was just baffled by the whole "passport advantage" pricing scheme and how the ECperf folks have now given it a veneer of respectability by approving the result.

    I've said enough and now it is time for me to leave this thread.
  46. I suppose these results are due to IBM is using their own VM which is ever so much more scaleable than the Sun VM.

    I expect now when BEA bought the swedish JRockit VM they'll be back with a vengance. And since JRockit VM is beating the IBM VM (even if ever so close)...

    What VM was used for the BEA test?
  47. jRockit ;-)
  48. Further to Matts comments[ Go to top ]

    Is there any significance to the fact that the hardware and operating system software was different during the WAS 4.0.3 test (2nd test) versus the WAS 4.0.2 test (1st test)? AIX was used for the first test while MS W2K server was used for the 2nd test [on a set of Intel cpu's]. Would we expect to see better or worse throughput for the 2nd test using the first test's hw and OS sw? What about with Sun / Solaris and/or Linux?
  49. Good job, BEA and IBM will go back and forth on this. BEA will release new numbers when 7.0 goes public, WebSphere 5.0 is around the corner. Its really great to see this competition because both products will strive to make their product better.

    One definite thing to note is WebSphere 4.0.3 is a production version as and WLS 7.0 is beta. I thought IBM would bounce back with 5.0 beta but bouncing back with a production version I think is great.

    Finally, with all the Sun vs. .NET and J2EE talk, the lack of other vendors are just proving J2EE is IBM and BEA.


    Views my own, and not necessarily IBMs
  50. Roland,

    > One definite thing to note is WebSphere 4.0.3 is a production version as and WLS 7.0 is beta.

    Nice spin, but 4.0.3 is still beta.

    Page 6:
    Availability Date: WebSphere 4.0.3 will be available April 30, 2002. All other components are available as of February 25, 2002.

    What's more interesting is that IBM does not use standard list prices for their App Server whereas BEA does.

    Page 20:
    Note: Software pricing based on Passport Advantage Agreement.

    Cheers,

    Mike Murphy
  51. 4.0.3 is just a fixpack, not a full release. The fact that the release date is 4/30 does not mean that it is beta code. I cannot download it now to play with from a public site like I can with WLS 7.0 or WAS TD(5.0). But on 4/30, the package is a production fixpack, not a beta fixpack.

    views my own, not IBM.
  52. Roland,

    > 4.0.3 is just a fixpack, not a full release.

    Spinning again...

    I'll use your words this time. 4.0.3 is not in production, not available, not downloadable.

    Let the audience decide.

    Cheers,

    Mike Murphy
  53. Roland,

    > 4.0.3 is just a fixpack, not a full release.

    Spinning again...

    I'll use your words this time. 4.0.3 is not in production, not available, not downloadable.

    Let the audience decide.

    Cheers,

    Mike Murphy
  54. Benchmark specials and hokey pricing[ Go to top ]


    That should just about summarize the recent IBM result. Why doesn't IBM use list prices (like the other ECperf scores)?

    Just look at the sheer number of IBM employees on this thread trying to defend their ECperf publication -- does this tell you anything? These were the same folks who were desperately trying to disparage the earlier BEA/HP result as well.

    Isn't it surprising that the only folks defending WebSphere are IBM employees?

  55. I think in case of this product the benchmark should definitelly include the cost of consulting required to make Websphere work, plus cost of time developers have to spent on memorizing workarounds documented in Redboooks.

    Yeah, yeah, I know we have now, new, better, improved WS version (n+1). The same promisses once again ...

    Not to bash IBM - Thinkpad laptops are excellent ;-).

    Michael


  56. Benchmark specials and hokey pricing[ Go to top ]


    Can IBM publish a score on Solaris using Oracle as the backend? This will tell us the true cost of WebSphere (among other things). IBM is using the "passport advantage" program to effectively mask the real cost of WebSphere by making it look cheaper than it actually is.

    "hokey pricing" indeed!
  57. "Can IBM publish a score on Solaris using Oracle as the backend? This will tell us the true cost of WebSphere (among other things)."

    But people don't really care about "the true cost" of the app server. I think they care more about the total cost of ownership.

    I think this comment underscores the fact that because IBM can sell you everything you need they can make complex promotions that are not as easily available to BEA. Perhaps BEA is/can get in bed with Sun and server makers to counteract this?
  58. Googs:

    I disagree! The last thing I want to do is to bet my entire business on a single vendor (in this case IBM). I want to have the flexibility of being able to choose the best of breed solution. This doesn't mean I should get ripped off if I want to run WebSphere on Solaris/Oracle -- WebSphere costs will start to escalate *dramatically* for all other configurations. Why do you think IBM is reluctant to use list pricing in their benchmarks?

    If you believe you have the $$$ to shell out to the hordes of IBM Consultants whose help you will need most certainly to get going, then by all means go ahead and use WebSphere/DB2/AIX.
  59. I have to agree a bit here. I am sure that WS with DB2 is great. But say I work for a company that already has 30 or 40 enterprise Oracle Licenses (Just left that job actually, but pretend I was still there). I know for a fact that that company would not have let me choose DB2 instead of Oracle simply because of localized cost for the app-server. Now my current company would only let me use WS on Solaris.

    These scenarios are not uncommon. I don't actually care that much about the performance benchmarks, as they're not the end-all to app server selection. I *would* like to see costs with other DB servers (Oracle? Sybase?) and OS's (specifically Solaris or HPUX).

    -Newt
  60. The last thing I want to do is to bet my entire business on a single vendor (in this case IBM).


    Please leave IBM alone for a moment. They at least contribute to the independant ECperf benchmark. Perhaps you could take a look at how Oracle always has been playing the game:

    their own benchmark..., their claims against BEA, the same boring claims (after BEA had responded). They even try to piss of Microsoft, which is not a good idea :)

    Cheers,
    Pieter.
  61. WebSphere costs will start to escalate *dramatically* for >>all other configurations. Why do you think IBM is

    >> reluctant to use list pricing in their benchmarks?


    You are assuming this. There are no numbers to prove the Solaris solution is drastically different. So from the ECPerfs result I am too assume BEA solution on Solaris is also far worse than HP.


    Also many people see the benefits of a Single Software Platform Solution, especially with IBM since IBM is not a small company who will go out of business tomorrow.

    Many of the IBM products are the best of breed or at least close to the best of breed. MQSeries, WebSphere, DB2, Policy Director and AIX are top notch products, you are guaranteed to be in the top 5 with any of these products.
  62. Roland:

    Spoken like a true IBMer!

    Why doesn't IBM take the lead in publishing a result using components from other vendors (there is really a world outside IBM -- you know that, right?)

    If you read my post carefully, you will see that I'm referring to the fact that just to get WebSphere running on Solaris and Oracle, I will have to fork out a ton of $$$ (we're not even talking about performance, just getting started). Can you please let us know the license costs for WebSphere if I *don't* want to use DB2/AIX?

    And then there all those support issues to deal with. On several occasions, I've tried to get help from IBM with a few issues I was seeing with WebSphere on Solaris/Oracle and was flatly told I'm better off moving to AIX/DB2. They wouldn't even listen to my problems -- so much for IBM customer support!!!

    Needless to say, we dumped IBM and moved to BEA immediately after this crappy experience and am glad not having to deal with IBM again!

    If you take off those big blue glasses that you are wearing (or being forced to wear) :-), you will realise that there are several valid reasons why everybody does not wants to go with one vendor.
  63. http://www-3.ibm.com/software/swprod/swprod.nsf/H2BPages/TDUN-49F2DQ?opendocument

    You can buy it online from here, the cost of the App Server does not differ by platform.

    I am a WebSphere consultant and have gone on many client engagements where they need help for WAS on Oracle and Solaris. Our consultants are trained on both platforms. I don't know who you called for help?

    views are my own and not IBMs
  64. "If you believe you have the $$$ to shell out to the hordes of IBM Consultants whose help you will need most certainly to get going, then by all means go ahead and use WebSphere/DB2/AIX"

    Ok - you lost me here. I thought the latest benchmark was running on intel and Windows 2000. The box is probably built by IBM, but it's still wintel, not aix-risc.

    Obviously the cost per transaction with this workload is lower on wintel, so I wouldn't go with AIX. But, if IBM sells this entire configuration for less than the sum of the parts, then more power to them ...
  65. The reference was wrt to a real world scenario ;-)
  66. Can IBM publish a score on Solaris using Oracle as the > backend? This will tell us the true cost

    > of WebSphere (among other things).

    Aha, we publish those numbers and next day you see Oracle or Sun ads in all IT magazines: "when IBM wanted to show off their WebSphere performance they have chosen Oracle database (or Sun hardware)" - this has happened before.

    Actually - there was an ECperf running on Sun hardware with Oracle database on HP (this last fall). I wish I could tell you results from that run...

    > IBM is using the "passport advantage" program to
    > effectively mask the real cost of WebSphere by making it
    > look cheaper than it actually is.

    Passport Advantage is not a discount. It is normal IBM pricing model. Go to ibm.com - pick same software/hardware as in ECperf report - and you will get exactly the same price. Even if you work for BEA or Oracle. Again - no special negotiations are needed - it is automatic feature built into the IBM price list.

    If this was "special" discount (which is prohibited by ECperf) - do you think BEA people on ECperf committee would let IBM publish this benchmark beating BEA so strong?

    Yes, I do work for IBM.
  67. Aha! Another IBMer on board! You folks are getting better at spinning this hokey pricing model.

    I notice that you conveniently forgot to mention the support pricing. Is it true that the subsequent support renewals aren't included in passport advantage price? ie: the classic bait model. Start with a really low price for support and then when it is time to renew the contract, wazoooooom! The old prices are no longer valid and the poor customer who doesn't have a choice now is stuck with forking out gargantuan sums of dough!

    I am still looking for those non-IBM mails saying how great WebSphere is! The silence is deafening! :-)
  68. Common guys the facts are on table, both BEA and IBM. Why crib, so much. The good news is3 both IBM and BEA are working hard at beating each other. And they are doin well at that.

    And hey i am too waiting for more non-Andy posts to question these results. ;)

    Take it easy pal :)

  69. Kapil:

    Good point. But as many of the postings have shown, this particular IBM result just doesn't feel right! Too many inconsistencies.

    I don't work for IBM or BEA; just interested in J2EE!
  70. Can anybody explain me, how I can compare the benchmarks, if I have a different type of machine?

    Let them all run the test on the same machine and we can compare them...

    Mirko

  71. That ain't going to happen Mirko! IBM will never publish a number using Solaris or Oracle.
  72. Benchmark specials and hokey pricing[ Go to top ]

    I'd like to see BEA's performance on DB2/AIX, if BEA is really willing to beat IBM's result with RMI-IIOP.
    Does BEA support DB2 as CMP DataSource?
  73. Benchmark specials and hokey pricing[ Go to top ]

    If DB2 has a JDBC driver, it will work with WLS CMP, obviously.
  74. Gary,
    Whilst, it may seem obvious, unfortunately, it's not the case. Just having a JDBC driver is not sufficient for full CMP support on anybodies J2EE server whether it's WAS or BEA or anybodies.

    It may work but we (and the other vendors) do a lot of work working around slight differences in databases to get them 'supported' for CMP.

    As CMP implementations become more sophisticated this is going to become more so as the CMP implementations start using database features to chase performance to higher levels. A lot of these tweaks are usually vendor specific.

    If only it was so easy.

    Cheers,
    Billy (IBM)
  75. From my understanding the Sponser publishes results, and it seems that the hardware vendor has been the sponser. So if I want to see WebLogic or WebSphere results on Solaris, doesn't Sun have to sponser that. So from looking at the results, BEA never sponsered a test, only HP did. Will HP publish their app server next to get an edge over BEA if they can?

    I don't understand the process, but can anyone sponser (software vendor, hardware vendor, db vendor)?
  76. How to compare results?[ Go to top ]

    <snip> Can anybody explain me, how I can compare the benchmarks, if I have a different type of machine?
    </snip>

    The benchmark is designed to be comparable across platforms. Hardware pricing generally absorbs computing power. Meaning a hardware that performs better will be priced higher and that doesnt will be priced lower. This to some extent also makes comparisons across heterogenous hardware easier.

    At the end of the day, a customer needs a complete J2EE platform (app server, hardware servers, OS, VM, DB) and not just an App Server. Further, for various reasons certain combinations of the above will perform better. So to compare results here one will need to look at the whole stack (even if App Server is the most differentiating aspect of the stack from any perf difference wise). Also, the stack price is what is relevant to the customer (ultimately customer wants the complete solution and its price).

    On the whole the benchmark does allow comparing complete J2EE platforms (though some may argue it will be good to compare just the App Servers).

    Cheers,
    Ramesh
  77. How to compare results?[ Go to top ]

    <snip> Can anybody explain me, how I can compare the

    >benchmarks, if I have a different type of machine?
    ></snip>
    >
    >The benchmark is designed to be comparable across
    >platforms. Hardware pricing generally absorbs computing
    >power. Meaning a hardware that performs better will be
    >priced higher and that doesnt will be priced lower. This
    >to some extent also makes comparisons across heterogenous
    >hardware easier.

    I think the only way to draw conclusions for a particular customer is to install ECperf on the hardware the customer can afford. I'm running ECperf with WebLogic and JBoss on the same hardware.

    My actual opinion is that Borland remains the true winner, because of:

    >The Borland AppServer Configuration Kit for ECperf 1.0
    >Benchmark is being provided for customers to test for
    >themselves the leading scalability and performance
    >available from Borland AppServer. The Kit download is
    >accompanied by a step-by-step guide for customers to
    >install, run, and analyze ECperf results for Borland
    >AppServer.
    (source: http://www.borland.com/appserver/ecperf/)
  78. How to compare results?[ Go to top ]

    Pieter : I absolutely agree with you.

    Where are BEA and IBM kits so anybody could perform its own test, and compare on its own hardware?

  79. How to compare results?[ Go to top ]

    <snip>
    I think the only way to draw conclusions for a particular customer is to install ECperf on the hardware the customer can afford. I'm running ECperf with WebLogic and JBoss on the same hardware.
    </snip>
    If you look at the numbers coming out now (and due out in the coming weeks), I think it enables direct public domain comparison.

    About any customer running the benchmark, the benchmark IS designed to enable this easily. The Ecperf kit, combined with the disclosures in the submissions is sufficient information to make benchmark runs. If you know of any difficulty in this pls send your feedback to the ecperf group and we will fix it.

    Cheers,
    Ramesh
    - TeamPramati
  80. How to compare results?[ Go to top ]


    The FDA and the FDRs have all the information you would need to reproduce the result.
  81. Dear Roman,

    I did follow your advice, and checked ibm.com out. In fact, it's http://www-4.ibm.com/software/howtobuy/ppa.html

    The tag line on that page is:
    The way to do worldwide VOLUME licensing of IBM software.

    Here is a quote from that page:
    Please note: There is a 10 point minimum requirement to join the program. For details, ask your IBM phone representative.

    What do you think the representative will tell you? Well, I did try.
    First, he/she wants to know the size of your enterprise.
    Second, you have to purchase all the hardware from IBM, and only use their IGS.
    There are other conditions (as per the quote, 10 :-)

    This is absolutely OK for a vendor to push their products by offering a special price. It's just should not be OK to used it as a regular price - it's not available to any customer who walks into IBM's door.

    And, as for BEA voting for it - I don't think so. Remember, it's a vote, and the majority can win even with one vote.

    I don't work for BEA. And obviously, for IBM either. I work for real cutomers.

    Best,

    George.
  82. Dear George,

    <george>The tag line on that page is: The way to do worldwide VOLUME licensing of IBM software. Here is a quote from that page: Please note: There is a 10 point minimum requirement to join the program...</george>

    I disagree. Here is the fact-> "Volume" means "package", not "discount" (which is offered as a result of a sales negotiation). Remember what ECperf spec says: "All pricing used must be list pricing. No form of discounts are allowed... It is permissible to use package pricing as long as this package is a standard offering." If vendor is able to offer such packaged price - that is as important as vendor's ability to achieve high transaction rates. Why BEA doesn't offer packaged pricing? They don't own unbreakeable Oracle DB yet, neither they own Sun. This is the only reason why they have lost pricing round this time.

    ECperf is about real world - and this pricing is real world and advantages that IBM has as a single vendor are real world (yes, I know some folks hate this "single vendor" statement, but that is separate discussion).

    <george>What do you think the representative will tell you? Well, I did try. First, he/she wants to know the size of your enterprise.</george>

    Are you surprised that Sales rep wanted to know this? ;-)

    <george>Second, you have to purchase all the hardware from IBM, and only use their IGS. There are other conditions (as per the quote, 10 :-) </george>

    Hardware, IGS... Have you really called shop IBM number?

    <george>This is absolutely OK for a vendor to push their products by offering a special price. It's just should not be OK to used it as a regular price - it's not available to any customer who walks into IBM's door. </george>

    ANY person from the street can get prices quoted by IBM in ECperf submission. This has been verified. I do want to protect IBM ethics (which I believe are quite high), but I don't really want to waste any more of my time on things like this. Therefore

    SUMMARY: If you can't get those prices from IBM (for whatever reason), send your complains to ECperf and they will check. Why do we have this worthless non-technical discussion on this forum? If we are all "guys working for J2EE", so lets think how to make it better than .NOT.

    I'm too waiting for Oracle to publish their unbeatable numbers...

    Now on a creative side - What do you think is a proper way to compare J2EE and .NET performance? I wonder who will create that spec (web services based?)...

    Roman
  83. Well-done Big Blue. I am curious to see what BEA will come up with now. I believe the fact that BEA and IBM are at each other's throat is proof that ECPerf is a success. We are actually going to be able to use this benchmark to make purchasing decisions!
  84. "Well-done Big Blue. I am curious to see what BEA will come up with now. I believe the fact that BEA and IBM are at each other's throat is proof that ECPerf is a success. We are actually going to be able to use this benchmark to make purchasing decisions!"


    Hang on, Googs. Consider WebSphere only if you're going to deploy on IBM hardware. And if you do, make sure you get a very good discount off the software from the bundled package! Possibly even free! That explains the price/performance results!

    It's the same reason why nobody actually pays for Oracle 9iAS. We get it free with the database servers since 9iAS is not worth spending a cent on...

    Catcha later,
    Jason

  85. "Hang on, Googs. Consider WebSphere only if you're going to deploy on IBM hardware."

    In this case though "IBM hardware" is really Intel hardware ...
  86. This is very good news!

    They are all proving, it seems to me, that Windows platform is 'the way to go'. I wonder when will IBM release results using Linux ... or Linux still has a long way to go. Would love to see something on this ...

    Anyone any thoughts?

    Great job both IBM and BEA! Cant wait to see the results of Oracle9i, since it runs 2X or 4X faster than WebSphere ;)

  87. I've heard something suggesting that Linux JVM's weren't using "real" threads, so their scalability was very limited.
    It was somewhere around JBoss community (load testing JBoss), so I didn't consider it just MS/IBM/Sun/put-you-favourite-here marketing.
    Regards,
    Vlad
  88. Vlad,

    It depends on the vendor of the JVM running on Linux and how they have implemented it against Linux's threading model. Currently, Linux supports a one to one threading model. THat is, every user thread spawns a kernel thread. This is great for scalability, but really poor in terms of thread scheduling. Up to 30 percent of the CPU time could be spent in the thread scheduler. I have considerable experience with this mess using WebSphere 4.0, and the IBM JVM 1.3 on the Linux 2.4 kernel. The performance of the JVM was outstanding, however, it could not scale past 450 plus concurrent users because it seems that the IBM JVM on Linux is just a recompiled hash of their JVM implementation on Windows and AIX which supports a many to many threading model. THere are other vendors out there (JRockit for example) that has a very scalable JVM on Linux, however, these will not run with WebSphere (which is currently my biggest gripe about WebSphere). Anyhow, the same test runs we did using WebSphere and the IBM JVM on Windows went well past 1400 concurrent users. Check out the following benchmarks and you'll get what i'm talking about:

    http://www.volano.com/report/index.html

    BTW, this should all change in the Linux 2.5 kernel which finally supports a many to many threading model.

    To the rest of the folks who are gripping about needing consultancy services to get WebSphere up and running at optimal speeds should consult the IBM Solution Partnership Centers: http://www.developer.ibm.com/spc/index.html
    If you are a member of PartnerWorld for Developers, or an IBM partner, you can put in a request to get into one of these centers for up to 2 weeks with access to tons of hardware, a customizable network infrastructure, and IBM consultants. It used to be free, but now they're charging 1000 bucks. I mean, what's a 1000 bucks for 2 weeks of all that versus 100 bucks an hour for a consultant?


  89. Vlad: "I've heard something suggesting that Linux JVM's weren't using "real" threads, so their scalability was very limited."

    There are fake (green) threads implementations, and there are native threads implementations and there are mixed (each native thread supports several Java threads) implementations for Linux. See Sun, IBM and Blackdown as examples.

    Peace,

    Cameron Purdy
    Tangosol, Inc.
  90. BEA/IBM Vs. .NET[ Go to top ]

    Hello,
       The technical directors I have spoken to have often either directly stated or hinted the WebLogic is clearly a better product and that if it was not for the IBM marketing $, they would dump it and build solely on BEA. This marks the remarks of 2001 mostly. The trend has been changing. In 98, BEA set out to own this market, and clearly had a huge headstart. IBM has thrown massive resources at the challenge and is now about evened up the score.
        I am impressed with the result here by IBM, though the pricing issues are suspect. I spoke to an IBM executive a week ago that brought back memories of the middle-management waste that was associated with large corporations. He actually claimed BEA came to IBM asking for IBM to buy them out several times. I would like to know if this kind of nonsense (which I am certain is not true) is still corrupting the minds of IBM execs. If so, they are not only going to get whipped by BEA, but also by MS, who is more of a threat. If .NET wins, BEA and IBM will be fighting over scraps.
       Having said that, the IBM people on this chat thread have taken some abuse. As a stockholder, I hope that there are more like them within IBM. Keep up the good work!!

       About Oracle. Having worked there when the "Network Computer" was the talk of the day. There is a rule regarding Larry Ellison that is well known internally. "If Larry is saying it and wanting for people to believe it, you can be rest assured it is a lie." Oracle's (more accurately Larry's) absurd overselling works great when everyone has rose colored glasses on. But when the market is down, ....well, check out Oracle stock prices.

       About this Data Stomping. Forgive me, I am moderately technical, but I have not found anything in the disclosures of either BEA or IBM that hints that they were cheating on data integrity on this benchmark. (By cheating I mean a BM special with no Isolation or Consistency.) Often real world Apps (I have seen plenty) use read committed Isolation mode. This test seems to test only CMPs as the BMPs are very poorly written, perhaps intentionally. BTW, in IBMs defense, what goes interesting unsaid is that they are whipping Oracle in this test on DB2. People often say, and I have heard, that DB2 can not run outside of AIX. Well, 2 CPUs where Oracle needed 4...comments?
                 Keep it up BEA and IBM....GS

           
        
  91. BEA/IBM Vs. .NET[ Go to top ]

    Gary,

    1. Check TPC/C results (non-clustered) and you will see that Oracle is way better on a single machine. Every time when IBM compares DB2 with Oracle they( IBM ) present their clustered result ( Shared nothing cluster ) IBM doesn't sat anything for their results on a single machine . To get high results on TPC/C results with shared nothing cluster is really easy ( 9 tables , 7 of them have same column in the PK, so after you partition using that column the SQL is mostly local for each node and as far I know no alternate index access is used in the benchmark) but I haven't heard yet for OLTP implementation of that architecture ( Only TPC/C tests ) . If you check IBM's TPC/H benchmark you will see that their high results are clustered again. But for DSS at least this makes sense.
    2. Check www.eweek.com - they did some testing for databases recently and IBM's score is worse that Oracle's
    3. Ask yourself why 70% of SAP installations use Oracle ( As far as I know Oracle and SAP are not best friends ), around 60% of Siebel's installations use Oracle and etc.

    I don't work for Oracle or IBM.

    Regards,
    Alex
  92. BEA/IBM Vs. .NET[ Go to top ]

    Hello Alex,
        I will try to ignore the painfully condescending way you replied to me. Yes, we know Oracle owns a large chunk of the DB market. They do have a $17B market cap, even though they are not doing well.
         You never know who you are speaking to in chat. For a non-specialist, I am actually quite up on the TPC-C. I know the benchmark and the transaction mix quite well. People find it enlightening often when I point out two huge failings of the test: 1, that it avoids almost all contention with the scaling rules, almost no collisions, except for the orderID, which must be run in serializable mode (or with an advanced non-standard feature); 2, and more importantly, that it is a disk I/O test as much as it is a DB test. With disk reads 40x over memory reads and the DB so large and randomly hit that caches don't matter, all it matters is how well the DB scales and how much was spent on disks.
        You made a good point about the partitioning being silly. The system configurations for the clustered results are from another world.
        You correctly pointed out that Oracle dominates the TPC-C numbers as well and the IBM numbers are low. IBM should be inferior to Oracle. This is interesting to ponder. I think I need to read the IBM CMP configurations again. It is very dubious that IBM could be getting 16K BBops on a 2 way while Oracle needs a 4 way. Hmmmm. I think it is unlikely that WebLogic uses the DB at less the 50% efficiency of WebSphere, especially since the application is coded to dumb down the DB. The BMPs are especially poor. Are they avoiding keeping the DB informed by having "lazy" CMP. The Data-stomping is a lot more likely if the DB is doing less work that it should.
        As opposed to the TPC-C, this test has high contention and a moderate transaction rate against a reasonably sized, thought small, DB. I am starting to wonder about this....There is something somewhere not right. Let's look into this.
                   GS
  93. BEA/IBM Vs. .NET[ Go to top ]

    Gary,

    I am really sorry if my reply sounded condescending, I didn't mean to.

    Regards,
    Alex
  94. Data stomping? Sure!![ Go to top ]

    <snip>
    About this Data Stomping. Forgive me, I am moderately technical, but I have not found anything in the disclosures of either BEA or IBM that hints that they were cheating on data integrity on this benchmark. (By cheating I mean a BM special with no Isolation or Consistency.) Often real world Apps (I have seen plenty) use read committed Isolation mode. This test seems to test only CMPs as the BMPs are very poorly written, perhaps intentionally.
    </snip>
    Presently most servers dont provide any form of locking support directly. The EJB spec has a section that specifies the concurrency control requirements; wherein it is expected that the server will lock access to the bean instances to ensure that two transactions dont go and update theinstance at the same time. But presnetly this is a bit ambiguous and generally argued as not being mandatory.

    So, most vendors by default dont provide this locking. In both WAS & WLS cases, they rely primarily on database REPEATABLE READ isoltaion level to get this concurrency control. But th ebenchmarks are run in COMITTED READ iso level. When run in COMMITTED READ isolation level, there is absolutely no locking at all (either in the db nor in the app server). This means that the access to the beans is completely uncontrolled. Thus resulting in certain data stomping!

    Ramesh
    - Pramati
  95. Data stomping? Sure!![ Go to top ]

    WebLogic allows for "Exclusive" and "Database" settings for the locking. Exclusive was all you had in the old days. It restricts access to the EJB. "Database" is the optimistic concurrancy where it sends back to the database, relying on the db to handle the concurrancy.
  96. Data stomping? Sure!![ Go to top ]

    Hello Dion,
        Here is the link for the WebLogic 7 (used in ECPerf) documentation that is relevant:http://e-docs.bea.com/wls/docs70///ejb/EJB_environment.html#1145146

    If the link doesn't work, go to the 7.0 Doc and search on 'EJB Concurrency Strategy'. There are 4, including Exclusive, DB, Optimistic and ReadOnly.
                  GS
  97. Data stomping? Sure!![ Go to top ]

    <snip>
    "Database" is the optimistic concurrancy where it sends back to the database, relying on the db to handle the concurrancy.
    </snip>

    Relying on database is not exactly optimistic concurrency! When relying on the database for concurrency control, database is generally told what kind of concurrency is needed. Normally READ COMITTED is used (WLS & WAS used this in their Ecperf runs). And databases DONT do ANYTHING about locking in READ COMITTED (all this does is to ensure that no 'dirty' data is returned). For proper concurrency locking, one has to use REPEATABLE READ isolation level in databases. (And this is not used as this being a highly serializable mode, the preformance can go down several fold!)

    In effect, the optimistic cincurrency you are talking about is basically NO LOCKING!

    Cheers,
    Ramesh
    - Pramati
  98. Data stomping? Sure!![ Go to top ]

    Hi Ramesh,

    Ramesh: "Relying on database is not exactly optimistic concurrency!"

    Definitely true!

    Ramesh: "When relying on the database for concurrency control, database is generally told what kind of concurrency is needed. Normally READ COMITTED is used (WLS & WAS used this in their Ecperf runs)."

    This needs clarification, and I know that you know, but you should have kept concurrency control and isolation as two separate issues. Read committed is an isolation level, which basically describes how multiple simultaneous connections can use the database with or without seeing the effects of other simultaneous connections (hence the term isolation). While it is closely tied to the concept of concurrency, it is not the same.

    Ramesh: "And databases DONT do ANYTHING about locking in READ COMITTED (all this does is to ensure that no 'dirty' data is returned)."

    Databases may not do any locking in repeatable read and serializable either -- see Oracle which uses an internal "optimistic" approach to serializable transactions to ensure the highest level of concurrency and the highest level of developer frustration. ;-)

    If you use read committed isolation, you can lock on most databases by issuing an update to the desired rows within a transaction. Also there are "for update" clauses that allow you to place what is typically a "lesser" (shared or upgradable, see CORBA/OCS) lock. These can be used to implement pessimistic concurrency, even with read-committed isolation.

    Otherwise you can implement optimistic concurrency control either by checking all column values of "optimistically locked" rows, only those values that are being updated, only those values that you "care" about, or an incrementing or timestamped version column.

    The primary benefit of read committed isolation is that most databases use this model internally, so you get the highest level of database throughput, and because it doesn't limit concurrency, you get great scalability.

    Ramesh: "In effect, the optimistic cincurrency you are talking about is basically NO LOCKING!"

    I like to say that most people use optimistic concurrency, as in "Gee, I sure hope that my data doesn't get corrupted!"

    Peace,

    Cameron Purdy
    Tangosol, Inc.
  99. IBM. just on 2cpu DB?[ Go to top ]

    <snip>
    IBM should be inferior to Oracle.[...]This is interesting to ponder. I think I need to read the IBM CMP configurations again. It is very dubious that IBM could be getting 16K BBops on a 2 way while Oracle needs a 4 way. Hmmmm. I think it is unlikely that WebLogic uses the DB at less the 50% efficiency of WebSphere, especially since the application is coded to dumb down the DB.
    </snip>

    One reason forlower DB reqt than Weblogic cld be that the DB load is reduced because of using the cmpOpt tool and setting the deployment flags of intent=READ, in the IBM benchmark runs; for all these methods the ejbStores dont get called at the end of the transactions (not involving any of the other non READ methods). If you run WebLogic with such a flag (if available) manually set, you may see a much lower DB reqt than was needed in their runs.

    (Further, on the whole the Db reqt is lower than may be expected from similar transactional apps is because of intent=READ and also due to there being no concurrency control in place. TPC-C is always run with absolute concurrency control (atleast SELECT.. FOR UPDATE locking))

    Cheers,
    Ramesh
    - Pramati
  100. IBM. just on 2cpu DB?[ Go to top ]

    Hello Ramesh,
        I agree with your last paragraph. That was the point I was suggesting, but let me start with the first.
        If WebSphere is tuned so well that it calls the DB less (a lot less if that is the reason the DB is used Approx. 40% as much), it should have much less DB latency (expensive) involved in the transactions and thus a much higher BBops #, which is not the case. In addition, reads (as in read-only) are cheap as heck (at RC Isolation Level), so why would this strain the DB. If you look at the actual atomic DB transaction rate/ second, it is not that high. It is the contention (many updates and inserts) that causes the DB problems. If WebSphere is doing tricks such as each bean assuming it has unique access to a table, then it will be ignoring the contention without telling the DB about it. Even though the DB is set at the correct Isolation level, if the beans are all doing Update, Commit, the DB will be doing everything correctly but the data will still often be corrupted. At any rate, I agree that I don't see what is happening underneath the CMP covers, but making basic assumptions about CMP implementations it is hard to envision this scenario being all that it appears.
        Ramesh, you pointed out that the TPC-C goes through steps to verify data integrity. Well said. Since Data Integrity is not measured in any rigorous way, one must be Sherlock to find where it is being stomped. I will try to work on this next week.
        Thanks for the replies gentlemen,
                      GS
  101. IBM. just on 2cpu DB?[ Go to top ]

    <snip>
    Ramesh, you pointed out that the TPC-C goes through steps to verify data integrity. Well said. Since Data Integrity is not measured in any rigorous way, one must be Sherlock to find where it is being stomped.
    </snip>
    One place you can start looking is InventoryEnt. This is very sensitive to concurrent updates- to process orders fulfilment as well as to process manufacturing (the only value updated is inventory qty and there are only 2200 parts for upto tx200)

    Cheers,
    Ramesh