BEA takes the performance lead with new ECPerf results

Discussions

News: BEA takes the performance lead with new ECPerf results

  1. In an HP sponsored ECperf posting, BEA Weblogic Server 7.0 beta has leaped ahead of IBM Websphere in both performance and price/performance metrics. Also in recent news, Sun has withdrawn the original Sun/Borland ECperf result last week.

    After seeing these results I called up Weblogic to get the scoop. Eric Stahl, Senior Product Marketing Manager, Weblogic Server answered a few questions:

    Floyd: HP sponsored these results. What is HP's involvement in this benchmark?

    Eric: Our platform independent and agnostic attitude towards the underlying hardware/OS/DB means we are going to publish as many results on as many platforms as possible to reflect real world deploy environments common in our customer base.

    Floyd: Is this a pattern you expect other vendors to follow?

    Eric: One of our key value propositions is platform neutrality. Whereas IBM is trying to sell a single vendor solution, I don't think even they will only benchmark on IBM hardware and software.

    Floyd: So how did you guys do so well over IBM?

    Eric: It was purely a matter of getting our engineers focused on performance and getting many performance enhancements into Weblogic Server 7.0.

    This will be the death nail for vendors who can't keep up on the performance side. Cause if you can't get the numbers in the game, I can't imagine a customer who would buy from you.

    Ultimately, you look at performance numbers and price/performance numbers, and these are the metric that customers need to be evaluating when choosing a J2EE server.

    These numbers show that it requires more hardware to get the same performance for a J2EE application running on Webpshere, than on Weblogic. IBM has a lower license/cpu cost than Weblogic, but the overall cost of running an app on Websphere will be higher than that of Weblogic, due to the need to buy more hardware.

    Check out http://ecperf.theserverside.com/.

    ********************************
    * Editor's Addition - Feb 21nd *
    ********************************
    Sun asked us to host this official reason for the Sun/Borland widthdrawal:

    "Sun decided to withdraw the Sun Fire V880 result using the Borland AppServer as it does not reflect the performance of this combination of hardware and software today. Stay tuned for new results in the near future."

    ******************************
    Feb 22nd:
    Summary thread changed to reflect that it was Sun who withdrew the results, not Borland

    Threaded Messages (139)

  2. Finally! This is what everybody has been waiting for, and it's great. It's so simple and clear now who is who.

    Anybody knows when this WLS 7.0 beta becomes publicly available? Possibly, at BEA e-world expo next week?

    Thanks for any info,

    George
  3. WLS 7.0 Beta and Cajun Beta available for download to the public on Monday!

    Tyler Jewell
    Director, Technical Evangelism
    BEA Systems, Inc.
  4. And plan to GA at the end of April
  5. Awesome ... looks like Borland thought it was better to withdraw their results than suffer the inevitable comparison!

    http://ecperf.theserverside.com/ecperf/index.jsp?page=results/withdrawn
  6. So far all these results prove is that the Sun JVM is slowest - why do the vendors go to all the bother (and cost) of using different JVMs otherwise :-)

    E.g. Fastest result is using JRockit, next fastest IBM JVM, slowest (whoops, can't refer to that result anymore...).

    This confirms our recent testing that the Sun JVM (1.3.1 anyway) on quad cpu server boxes running Windows 2000 performs poorly.

    See intial results at:

    http://www.cmis.csiro.au/adsat/jvms.htm

    I wouldn't (yet) like to make any inferences about relative AppServer performance on a level playing field.

    Paul.

    PS How about BEA publish a result using the Sun JVM?!
  7. Hi Paul,

    You are really the guys I've been looking for! The ones that do testing with the same HW/JDBC/JVM on different AppServers. ECperf might have a future within the field of innovention for best Java performance on the server side as a whole, but I'll stick to your performance benchmarks when I need to compare different J2EE-servers for a specific platform.

    You are doing everything I thought that ECperf should do. Now that an objective (I hope you are) 3rd-party has taken the responsibility to provide benchmarks results with equal operating environments, my wishes from my last posting has come true.

    Thanks Paul Brebner (and all the others on CSIRO)!This will be my new bookmark for benchmarking:
    http://www.cmis.csiro.au/adsat/j2eev2.htm

    Regards
  8. Thanks for the positive comments - makes it all seem worthwhile. Please tell all your workmates, friends, relatives, etc - we need to sell lots of copies of the report to keep us in business :-)

    Regards,

    Paul.
  9. Bravo[ Go to top ]

    Congrats in finally doing an independant benchmark ;-)

    But, some points, and up front, please forgive me.

    This is the first ecperf test done with a beta product. All other tests are on regular GA level shipping code that you can buy. But, if ecperf allows this then fair enough.

    Eric's comment on less hardware with WebLogic than WebSphere.

    Two app server instances per box. Is there a problem with scaling to 4 CPUs maybe? Worth paying the penalty of twice as much RAM as an app server that can scale to 4 CPUs. Remember, an app server is a java application. Almost all of the RAM used by the app server cannot be shared across processes.

    So, if a JVM won't scale well on an SMP box then if your app server instance needs 1.5GB heap then for 2 instances that means 3gb of RAM, for 3 thats 4.5GB of RAM and so on. You can save a lot of memory by having an app server thats scale to 4 or 8 or even 12 CPUs like one I could mention.

    Plus, all these seperate processes causes CPU cache thrashing. Your CPU cache is no where near as effective as simply running a single process on the box. Its got n times the memory to cache than the alternative.

    So, you get in to the needing to assign the process to a subset of the processors to minimize CPU cache thrashing as BEA did in the test.

    Look at the WAS AIX benchmark. We (yes, I work for IBM) used a single instance and maxed out a 4 CPU box with only 2GB of RAM. Thats a simpler configuration, no CPU pinning and less RAM to buy.

    Different JDK. Fair enough, we use our own JDK on W2K also. But, I'd like to know if T3 or IIOP was the protocol used in the test.

    Anyway, congrats on the first, only and fastest for now number on W2K.

    Billy


    (views are my own and don't represent my employer IBM).
  10. Billy Boy:

    A case of sour grapes, eh? :-)
  11. Sour grapes, indeed[ Go to top ]

    Why on earth would you say that! ;-)

    But seriously, it's a good number and no one is going to beat it before BEA eWorld and thats what they wanted so hats off to them, nicely done and well timed.

    The comment about WAS needing more hardware than WL needed a response given what they were forced to do in the test to get the numbers.

    Have fun,
    Billy
  12. Let the Leapfrogging Begin[ Go to top ]

    I think you are spot on with the timing remark (for eWorld).
    Surely they will show the numbers and get a big cheer from the crowd at some keynote.

    We have to take these numbers with a pinch of salt. I think that vendors will keep leapfrogging eachother. There are so many things that come into play (hardware, JVM, protocol, etc etc).

    How important do you think ECPerf is?
  13. Sour grapes, indeed[ Go to top ]

    Just a general reminder to folks not to be so freaking naive. You don't know why Borland pulled their results. And do you think it really makes a difference how well anyone does on these numbers at the level they're running at? So far every vendor reporting results has done very well and the differences are more due to hardware and pricing than they are to app servers.

    What you should be looking at is the performance of ECperf versus the TPC-C benchmark at www.tpc.org. Sun wants play down the similarity to TPC-C, but it's definitely there (all I can say is read the ECperf and TPC-C specs and compare them), and many of the same hardware and software vendors are involved in both benchmarks.

    ECperf and the vendors publishing results so far are going after a very high end piece of the J2EE market. For those projects and decision makers and influencers on them, ECperf in general is a good thing (though it has a rather serious database consistency problem in its current implementation). But for most departmental projects, ECperf rates and prices reported in the official results so far are pretty irrelevant -- way beyond the price or performance affordable to or needed by many J2EE users at the low to middle tier.
  14. What I would like to see...[ Go to top ]

    I think its great that all of the biggies are finally coming out with ECPerf results. However, once this first round of results wears thin, I'd like to see an independent testing house put these app servers through a battery of ecperf tests on low, middle, and high-end hardware solutions, with a variety of database servers as well. I would then like to see a set of results out-of-the-box and a set of results after tuning (the company can send in its people to do the tuning). Then I would like to see the companies have the cajones to publish the results. Right now, the results are interesting, but only mildy so. Being able to compare a range of app servers that have all been tested the same way on the same platforms would be far more valuable to organizations than the current set of results, in my opinion.

    Like I said - what I'd like to see (yeah, I know - in my dreams)

    Cheers
    Ray
  15. What I would like to see...[ Go to top ]

    Hey, why don't you all give these guys, and I mean all the vendors here, some credit for the efforts. Otherwise, it's a lose-lose situation for them: run Pet Shop, or Trade 2 - bad, now run Ecperf - bad too? Show low results - bad, now show the best results - still bad?

    And why would you care so much which JVM they ran, or which RMI protocol they used? Isn’t it the result that matters in this case, and the compliance to the specs?

    At some point, we all wanted a standard independent benchmark. We more or less have it now, so let's respect it.

    ECperf is not perfect, but is a good reflection of 95% of applications out there. Even with its bugs J

    By the way, keep in mind that with ECperf, you only see results that have been approved. You don’t see those that have not met the guidelines, or were just plain wrong, or even volunterely withdrawn before they even could be approved.

    Right, at the end of the day it's not just performance and price. There is much more: functionality, administration, reliability, scalability, standards compliance, integration, Web Services support, partners’ and SI support...

    And, if you think about it - BEA with its WLS is either the best, or very close in all of these. That's why it's considered de-facto standard today, with the most following from customers and developers.

    It's the whole combination of factors, performance and price included, that make application server. Through my career I've been playing with pretty much all of them.

    Not any more, though. Yes, you guessed it - WLS was the last one. My clients are not complaining.

    George

  16. What I would like to see...[ Go to top ]

    G'day George and others ,

    It probably is worth mentioning again that ECperf has been developed by Sun under the JCP in close partnership with leading J2EE industry vendors. It was designed as a workload to test the middle-tier of an enterprise application and it was designed from day 1 with to allow comparing J2EE application server against J2EE application server. Other performance indicators e.g. the Java Pet Store has been used might provide useful information BUT they are not benchmarks and simply are not strict in implementation and run rules to allow valid app server to app server comparisons.


    It is also worth pointing out that the BBops metric of ECperf was kept deliberately numerically low in part to reflect the fact that the transactions being performed by ECperf are complex interactions, and we believe typical of those found in fortune 100 companies today. These full complex transactions are what we count and report in the metric. So "low results" is a very relative term. If you mean low relative to other industry benchmarks, this is correct and it is by design.

    Regards
    Tom Daly (Sun and yep part of the ECperf team)





     
  17. What I would like to see...[ Go to top ]

    Hi Tom,

    How nice to see a ECperf team member in this thread. I have a few suggestion for the ECperf team, that would make life easier for the customers. Remember, you are providing a test suite that should help customers make the right choices about J2EE servers, and not for vendors to show-off, right?

    1) Make ECperf mandatory in the CTS test suite. It requires some synchronization though. The current ECperf is EJB1.1/J2EE1.2, while the current spec level is EJB2.0/J2EE1.3.

    2) Strict pricing. Remove the clauses about bundles, packaging etc. The pricing today is not fair as only a few of the vendors have a stack and therefore can have bundled pricing. Remember; the system under test is the J2EE-server.

    3) Strict products. No Alphas, no betas, developer editions. Just released productional products. The 3 month "quarantine" is not of benefit to anyone except the vendor. What happens if the HW/SW is not released at the date specified? Are they fined? The vendor has already had their walk in the sun, marketwise.

    4) Last but not least: You are so right, Ray! How can the test come close of being comparable, when everyone uses different HW/DB/JVM/JDBC, which is irrelevant in the tests. Of course it's not irrelevant for the customer, but in these tests they are. So your utopic suggestion of a third-party vendor (why not The Middleware Company?) setting up the tests on 1 (or more) standard platform(s) of their choice, and let the respective vendors come and tune their AppServer for the tests.

    It is a utopia, though. Do anyone expect IBM to release numbers on a Sun box or vice versa?

    Rumors are saying that an update of ECperf to reflect EJB2.0 is not very likely, because there are too few released benchmarks. Is this true, Tom? If so, why can't Sun come out of the closet and show what iPlanet has to bring?

    Regards

  18. What I would like to see...[ Go to top ]


    BG-

    It seems to me that ECPerf is measuring the correct metrics, and a lot of the frustration is misunderstanding and misstatement.

    The fundamental problem is that ECPerf claims "to measure the scalability and performance of J2EE servers and containers." But as you pointed out:

    >>How can the test come close of being comparable,
    >>when everyone uses different HW/DB/JVM/JDBC, which
    >>is irrelevant in the tests. Of course it's not irrelevant
    >>for the customer, but in these tests they are.
    >> <>>Do anyone expect IBM to release numbers on a Sun box
    >>or vice versa?

    The fact of the matter is that the app server is BOTH an integral part of the stack (IBM, sometimes Sun, sometimes HP) that include hardware, OS, JVM, etc. AND a standalone, third-party product like BEA, JBoss, etc.

    It would be easier to do a "clean" ECPerf for solution stacks - IBM top to bottom vs. Sun top to bottom, vs BEA on Sun, etc. - and more meaningful for a lot of customers. ALSO, some customers would benefit from an agnostic ECPerf with multiple testbeds and the same app server profiled across the platform. That's where WLS vs. JBoss would be meaningful.

    There are too many independent variables in the complete stack to draw meaningful conclusions from results. The only way to solve that is to isolate the important variables - for some customers (and some vendors) - that means the entire stack. For other customers - the other pieces of the stack have to test heterogeneity.

    -Paul
  19. What I would like to see...[ Go to top ]

    G'Day BG,

    Some hopefully helpful responses:-

    1) Are we trying to develop an app that helps customers to make the right choice ? Yes, and one way to do this is by allowing vendors to compete, when this happens you get innovation and improvements across the entire J2EE application server market, sure this happens incrementally as each vendor has enhancements added to their product but overtime this lifts the entire platform space and the idea is that everyone wins.

    2) As to CTS, ECperf is not and does not have any direct relationship to compatibility testing, they are distinct and necessarily seperate exercises.

    3) Using Beta software. Well this is allowed by most industry benchmarks and I believe makes good sense. When do you want your app server vendor to test performance early in their product life cycle or late? I believe the earlier the better. Oh by the ECperf rules if all products stated in the benchmark report within 3 months vendors would have to withdraw their results.

    4)The comptetitive nature of the J2EE and software market will probably preclude the exact apples to apples comparison you are looking for, we will have to see what happens here.

    5) No EJB 2.0 wow where did this rumor start ?
    I can't talk about timetables but I can assure you we are already working on a J2EE 1.3 version of the benchmark!


    Hope this helps

    regards
    Tom Daly
  20. What I would like to see...[ Go to top ]

    O.k - so the BEA ECPerf BBops/min are about 50% higher than IBMs... Based on what we (CSIRO) know about the relative AppServer performances of these two products (well WebLogic 6.1 to be exact) on a level playing field I'd be looking at things other than the AppServers themselves to explain these results. As I pointed out previously the use of different JVMs is significant, but I'd ignored the h/w, which is possibly the most important factor.

    The BEA h/w is about 35% faster than the IBM h/w :-) This only leaves the JVM 15% to pick up - not impossible.

    More observations to come on

    http://www.cmis.csiro.au/adsat/j2eev2.htm

    Regards,

    Paul.
  21. What I would like to see...[ Go to top ]

    Just to add to Tom's posting, the application used in Ecperf is also extremely realistic (the benchmark is now over 2 years in the making!). Carefully chosen and written, considering several intricacies that typically come into play in enterprise apps. Involving several good design considerations (like for example, optimizing on DB updates using is-dirty checks, in BMP!).

    Apart from being a good benchmark to compare EJB containers, it is also a realistic scenario! If you see 16000 BBOPs, apart from other ops, this involves 8000 new-orders per minute (including system processing of the manufacturing for this order). Now how many actual production applications would there be that process that many orders per minute!! And that can be serviced by just 3 4-cpu servers (including the Db server)!

    This initiative by JCP is tremendous. And a great fillip to the technology!

    Ramesh
    - Team Pramati
  22. What I would like to see...[ Go to top ]

    Right to the point Ray Harrison...bravo! I agree 100%

    I want to see results on Sun 280 (2CPU, 2GB RAM), and Sun 3800/4800 machines (4, 6, 8, 12 CPU with 4, 6, 8, 12GB RAM). Then we can talk.

    Cheers
  23. Bravo[ Go to top ]

    But, I'd like to know if T3 or IIOP was the protocol used >in the test.


    What does it matter?

    From the Full Discloser:

    7.5.4 Disclose Driver To SUT Protocol Used
    The protocol used by the Driver to communicate with the SUT (e.g. RMI/IIOP) must be disclosed.
    The RMI over t3 protocol was used during the tests.

    mbg
  24. Bravo[ Go to top ]

    Posted by Mark Griffith 2002-02-20 19:20:28.43.
    >But, I'd like to know if T3 or IIOP was the protocol used >in the test.

    What does it matter?

    From the Full Discloser:

    7.5.4 Disclose Driver To SUT Protocol Used
    The protocol used by the Driver to communicate with the SUT (e.g. RMI/IIOP) must be disclosed.
    The RMI over t3 protocol was used during the tests.

    mbg

    I matters because if you are integrating with a CORBA Application than you will need to use RMI over IIOP and the WebLogic t3 protocol is much faster than their RMI-IIOP implementation. Therefore, the people using CORBA would probably be really interested in the differences between these numbers.
  25. Bravo[ Go to top ]



    >> Different JDK. Fair enough, we use our own JDK on W2K also.
    >> But, I'd like to know if T3 or IIOP was the protocol used
    >> in the test.

    Interesting that the topic of JVM has come up a few times in this thread. This page details Websphere's JVM support (you have to scroll way down to where it says "Java"). Unless it is out of date, it somewhat limited JVM support. For example, note that on Windows, only the IBM JVM is supported (and, ironically, it has the largest No. of JVM's available).

    You may think that you dont particularly care - but this has implications for *client* JVM also. If you are using the IBM JVM on the server, you must also use it on the client! This prevents you from using something as simple and powerful as JavaWebStart! (Now, its important to note that just about every appserver does not support mixed client / server JVM's - not just Websphere). Essentially though, the limitiation on the server affects the options you have on the client (also, the IBM JVM is not free).
    (Before anyone gets carried away - I *know* that its possible to eventually (its tedious) hack the client to use the Sun JVM - but in short, its not supported by IBM)

    On the flip side, Weblogic supports a raft of JVM's and platforms.

    On a different note, I have seen quite a few complaints about Weblogic's proprietary T3 which I think are somehat out of proportion: It is important to note that the only thing about T3 that permeates your design is the connection URL "t3://server:port" - the rest is hidden behind the RMI API. As far as interoperability is concerned an EJB in Weblogic can be simulatanously accessed by T3, IIOP, HTTP t3s, iiops, https - so there are no limitations to using T3 in some areas if you want to. Soon DCOM will be added to the list of supported protocols.
    (Sadly, until the Sun JDK ORB gets its act together, using RMI-IIOP will be problematic due to its interoperability problems (security and the like)).


    With the release of the ECPerf figures, it is evident that the performance of Websphere and Weblogic show are in the same league. Interestingly, however, it is difficult to find too many non-IBM employees that have anything good to say about Websphere - often its the opposite. Conversely, you dont find too many BEA employees having to defend Weblogic on this or any other site. I just hope that IBM can turn out a more appealing appserver (up to date, easy to use, open - not tied into other IBM products)


    Late last year, I spent quite a lot of time comparing Weblogic to Websphere - and there were quite a few differences - more than I expected. A lot of the differences were no-ones surprise - Websphere has consistently lagged behind the standards and features for long enough that its an established trend. While they have shown signs of reversing this trend with their marketing release of WAS5 alpha, the proof will be in the pudding of the final GA release date (I expect some time away).
    And while we wait, it is still the case that their existing *production* version, WAS4.0, (despite the publicity it received) supports almost none of the major J2EE1.3 features (CMP2, MDB's, EJBQL, JMS, Local Interfaces, Home Methods). Instead it supports "Message Beans" - a non-standard MDB-like session bean, and EJBQL syntax in non-standard DD's.

    While IBM are not the only ones that can be accused of marketing releases - vis Weblogic 7.0 Preview (alpha) - BEA have a slightly more credible story in that at least their 4-month old version 6.1 did support CMP2, MDB's, EJBQL, etc etc). The only important things missing from WLS6.1 was CSIv2 and OTS (ignoring JSP/Servlet stuff) otherwise, it was pretty close to J2EE1.3 certification.

    There were some other surprising shortcomings in Websphere that came up during this comparison:

    1) There is no secure CORBA client access to Websphere EJB's - not even if you are using IBM's own ORB. (this means that security EJB gives you is worthless - all beans must have no security configured). (Interestingly, Weblogic was the only one that had a decent story here - even the orb-vendors offerrings either didnt support security, or it was a pain to configure.)

    2) The developer version ONLY uses OS-security (e.g. you cannot use an LDAP). In order to do so, you need the production version - which is a real pain to use for dev.

    3) The application assembly tool was extremely buggy. The deployment verification process would detect hundreds of hilarious errors - none of which existed. Also, the stub generation would generate code that had compilation errors.

    4) Installing the production version was really painfull - it took days instead of minutes. (things like hard-coded paths in scripts - install DB2 out of its default directory and nothing would work)

    5) There were also some other issues related to clustering which I never got around to confirming (so they might be wrong). It would seem that clustering (work load management) isnt available to stand-alone java clients. In order to benefit from clustering, it seemd that you had to use the J2EE-Application-Client.

    The big plus that websphere has traditionally had over almost all other application servers has been the quality of its administration - however this "lead" has disappeared. More than just a good Swing GUI - thats bog-standard these days - it has long had support for centralised management/deployment to a cluster as well as the ability to remotely start/stop server instances.
    However, as long ago as WLS6.0, that difference has rapidly diminished. While WLS6.1's implementation wasnt as polished Webspheres, all of the required administration capabilities were present in WLS6.1 - and given its support for JMX, it is arguably better (as we will now see in 7.0).

    From a technical standpoint, it was very difficult to find much about Weblogic had a serious shortcoming. From a non-technical standpoint, it is the leading appserver - so there are no problems with 3rd party vendor support - its usually the first they support.

    While I dont necessarily have a lot of faith in the numbers that show Weblogic as 50% faster, it is in the same ballpark and that will be good enough for most people. Apart from price, there arent too many compelling reasons to look away from weblogic, unless you want to go open-source...


  26. App servers are merely tools to facilitate the automation of information processing and delivery.

    Having said that. We, as engineers, should pride ourselves on being able to select the ideal, most productive, best performing tool for a given job. We are all in agreement that J2EE app servers are superior to other alternatives. We agree that the benefits for simplified development, management and interoperability are revolutionary. To realize these benefits, we should strive to extend that consensus to a very few products. Such a narrower consensus can only be reached by considering the merits of the product and the abstract needs of businesses and ignoring the following:
    employer loyalties,
    selfish interests,
    anti-capitalistic sympathies
    or on hyper-capitalistic greed (ie. "How much consulting money will choosing this product generate?" ).

    To do otherwise only breeds general contempt for and confusion about the technology and tools -ultimately delaying their adoption. Recognize that these ecperf numbers puts us one step closer to achieving that much-needed consensus!
    Matt
  27. <matt>
    Recognize that these ecperf numbers puts us one step closer to achieving that much-needed consensus!
    </matt>

    Consensus? What consensus are we reaching here? Speak to this please.
    Cheers
    Ray
  28. As a software engineer, I do, in fact, take great pride in being able to select the ideal, most productive, best performing tool for a given job. However, I do not like nor do I agree with the concept, as I think you are perhaps trying to say (correct me if I'm wrong), that there should really only be a very few products to choose from as the key players. Here's my tool set of J2EE servers:

    WebLogic
    Orion
    JBoss
    EA Server
    Web Sphere
    OC4J (Orion - modified)

    They have their own sets of pluses and minuses and they are all continually evolving. I like it that way. I like competition and I don't like a narrow field -it makes no sense whatsoever. The ecperf numbers don't make it more likely that I will choose BEA over any of the others listed above. It wouldn't make sense for me or my clients. Especially, like you say, considering the merits of the various products and the abstract needs of businesses.

    Cheers
    Ray
  29. Ray,
    I would have agreed with that statement 1-2 years ago. Now, we are in a situation where app server innovation is not the bottleneck to the growth of J2EE. What we need now is for a few leaders to emerge and make the J2EE frontier safe for settlers. Confusion, due to too many products, and failures, due to poorly designed/understood products, will keep the settlers back east (using M$, Corba, Cobol, Pl/Sql, etc.)

    Innovation today is needed to create more packaged-apps based on j2EE and to find ways to improve productivity for developers and administrators.

    my $0.02
    Matt
  30. Matt -
    I don't actually see this confusion that you talk about - I don't know about your particular industry, but I work in two areas: Telecom and Insurance. There's no fear and there's no confusion. Once company I know is moving from their Cobol/Mainframe platform to J2EE - they've budgeted for 3 years of work. They aren't concerned about the number of products - and they aren't using BEA, IBM or Oracle for their app server.

    I would ask you to list poorly designed or understood products, I'm not quite sure what you are talking about - j2ee app servers or j2ee products?

    I think that since more app server products are complying with the j2ee spec (and I expect that the next 6 months will see all of the players there with production releases - by players I mean BEA, IBM, Oracle, EA Server, Orion, Pramati, HP, (don't know about JBoss), InQMy - to name a few) this fact will drive innovation in packaged products - it will be much easier to write for a range of app servers and price ranges. Fewer app servers would stifle innovation and competition would only be a token activity. Innovation needs a bigger canvas than what you seem to be talking about.

    Back to ecperf numbers: they can show certain things about a given app server on a given platform. That's about it (of course you can make a few abstractions from that information). And what are you going to do when a Pramati or Orion or JBoss comes out and smokes BEA on ecperf numbers? It wouldn't be that difficult to do. What conclusions would we derive from those numbers?

    Cheers
    Ray
  31. Ray: "Back to ecperf numbers: they can show certain things about a given app server on a given platform. That's about it (of course you can make a few abstractions from that information)."

    That's what ECPerf is for, and that's why we get a raw score and an effectiveness per dollar and information about the platform and configuration. Sounds like ECPerf is working beautifully.

    Ray: "And what are you going to do when a Pramati or Orion or JBoss comes out and smokes BEA on ecperf numbers?"

    Celebrate? Post an article on TheServerSide? ;-)

    Ray: "It wouldn't be that difficult to do."

    No one is stopping other companies from posting ECPerf numbers. I don't know about Pramati, but Orion and JBoss (both of which I'm familiar with) are not about to "smoke" those numbers, and I'd love to be the first to eat those words!

    If it weren't for ECPerf, the next release of Weblogic would not be as fast as it is. Period. If Borland or Orion or Oracle or IBM or whoever gets higher numbers, than (a) it is good for their customers and (b) Weblogic will have to improve and that will be good for their customers.

    Peace,

    Cameron Purdy
    Tangosol, Inc.
    http://www.tangosol.com/
  32. Hi Cameron -
    Thanks for your comments - and I agree with you for the most part.


    >>
    Cameron
    That's what ECPerf is for, and that's why we get a raw score and an effectiveness per dollar and information about the platform and configuration. Sounds like ECPerf is working beautifully.
    >>

    I didn't say it wasn't working beautifully. In fact I agree with you 100%. I made my statement because it seems others want to read more into it than there is - which I personally think isn't the most effective way to interpret the numbers.

    I see no reason from experience that Orion and certainly Oracle couldn't make you "eat [your] words" - I hope they do! I would expect that Oracle at any rate will eventually do so. I don't know about JBoss, I'm only familiar with it on a smaller scale. In fact, I expect that a number of vendors will beat BEA - a market leader is there to be beaten. And I agree with you - I like the concept because it creates better products all the way around. Others have intimated that the battle is over and now everyone should go home. That's what I disagree with in this thread - I say it has only just begun.


    Cheers
    Ray
  33. Hi Ray,
    "In fact, I expect that a number of vendors will beat BEA - a market leader is there to be beaten.... because it creates better products all the way around. "

    Competition is good between vendors but having products with short shelf-lives or small markets is even worse(It doesn't look like a mature technology.)

    Still I might agree with you if we were just talking about developing and porting J2EE code from app server to app server but the problem is a lot bigger. For each server supported an ISV has to address:
    Developer training
    Tools Support
    Administrative training
    3rd party product integration
    OS/JVM/DB support
    Scaling strategies
    Differences in non-standard functionality
    Ongoing support and upgrade engineering
    etc.

    Another negative is the project failures/setbacks that occur when trying to use the weaker products. This drags down the reputation of the technology.

    Our goal should be to try to accelerate the adoption of J2EE and MAXIMIZE the positive impact of J2EE on the economy -not retard adoption and perpetuate complexity through an ever-changing and mind-numbing array of options.

    Matt

  34. Hi Matt -
    I'm afraid that I can only say again that I disagree with what you say - but I've been doing it all thread. My own experiences don't coincide with your world view. I'll end it right here.

    Cheers
    Ray
  35. One good thing about this technology is that the levels of compliance and compatibility is extremely high here. For the first time ever, well written apps can easily be 'ported' (actually a misnomer in the case of J2EE!) onto other app servers. Other than deal with deployment, there shouldnt be any app change needed! With most vendors removing vendor lockins (proprietary APIs & extensions), it becomes easier now. (There is a lot of tool support, like our Pramati Studio, to abstract out server deployments even!)

    SO one need not worry abt the 'longevity' of any vendor.. as switching app srvrs is pretty easy now.

    Cheers,
    Ramesh

    ps: The vendors around now are not exactly with short shelf-lives. Most players with certified servers have been around since EJB1.0 days. (And have survived the tough times of 2001!)
  36. Well said, Ramesh!
  37. Most application ISV's are finding it extremely expensive to support more than 1-2 app servers. Also, many enterprises are also seeing economies-of-scale from standardizing on a given app server.
    Ray,
    Are you and Ramesh not seeing this?
    Matt
  38. Matt -
    I can't speak for Ramesh nor do I care one little bit what ISVs spend or don't spend - if there is a market they will spend, otherwise they won't. The organizations I work with (from very small to very large) who use j2ee application servers like having a choice. A choice between BEA and IBM isn't a choice. It depends on their (the organization's) development staff, their budget, their requirements. BEA isn't a one-size fits all solution in any way, shape or form. And neither is IBM. And neither are they always the best for a given task. My clients use these app servers as tools - and they choose the best one for the job at hand. In my daily work, I see a lot of scenarios out there.

    I've often thought that the market can be a wonderful thing - if one ISV finds it too hard or expensive to support more than BEA or IBM - you can bet another ISV will fill the void if there is a market for the product. Pretty simple.

    So - to answer your question - no - I don't see what you see.

    Cheers
    Ray

  39. Excellent points Ray. The value of J2EE is in having a wide range of servers to choose from. If it comes down to 1 or 2 we will all suffer. If there wasn't a lot of competition, you can be sure that Weblogic, Websphere and others would be priced much differently then they are....


  40. Ray,

    Apart from price, for what reasons are you seeing your clients steering away from the "big-2"? (you can illustrate by example)

    What features are they going for that other appservers provide? Do you see this a lot?

    Cheers,
    Nick
  41. Hi Nick -
    Sure - apart from price (price does play a large factor) I see the following:

    * Overkill - client doesn't want or need every little bell/whistle (and even the support) that comes with the product.

    * Client's existing architecture - for instance one of my clients is an old powerbuilder/sybase shop - you can publish powerbuilder objects to EA Server - no need to look elsewhere for a solution. Also I have a client who is all Oracle - they chose Oracle for the solution there.

    * The development staff likes a particular product - like Orion or JBoss - or they started out developing on these products as proof-of-concept - and were pleased enough with the performance to go production with them and have been happy.

    * Sort of goes with the above - the availability of the product: EA Server is free for development use, Orion, JBoss, et al - are much easier to get and start development immediately (and then continues as in the above point). BEA costs money - don't know about IBM.

    * Price - I know I am not supposed to include this, but it does play a large role in decision making even in very large companies. Not everyone that I work with needs the 24x7-ultra-availability-mega-clustering solution (if they do they go straight for the BEA/IBM packages) - in fact many don't. So they will look to other solutions.


    Just my observations....


    Cheers
    Ray

  42. Also Nick -
    I realized that I didn't answer your questions fully:

    It isn't so much that folks are finding things in other app servers that they don't in BEA/IBM - its just that they don't always need as much. Do I see this behaviour a lot? Not enough that the so-called big 2 need to worry aobut their market share - but enough to let me know that BEA and IBM are not the only game in town and that my clients like it that way, fine as BEA/IBM are as products.


    Cheers
    Ray
  43. "Most application ISV's are finding it extremely expensive to support more than 1-2 app servers. [...]
    Are you and Ramesh not seeing this? "

    ISVs generally would like to retain the option of not being locked in onto a single vendor (one key reason being to accomodate customer preferences). And there is no better way to ensure this than to work with two or more app servers.

    The best scenario though is where at any point they can quite easily plug-and-play any app-server! But this is easier said than done today! The spirit behind J2EE of pure compatibility is still not fully met yet. When the server vendors avoid lockins, then the only issues dealing with multiple vendors will be to deal with deployment mappings. Here, there are several tools in the mkt that abstract this out completely.

    One good thing though is even if there are any differences between app-servers, it is alomost always a super-set of the standard specs. So an app that doeesnt use ANY of the proprietary extensions of the app-servers, will most likely work on another compliant app-srvr without any difficulty. We make this more possible by ensuring that Pramati Server is pure-standards! And this, our ISV partners like.

    Cheers,
    Ramesh
  44. "No one is stopping other companies from posting ECPerf numbers. I don't know about Pramati, but Orion and JBoss (both of which I'm familiar with) are not about to "smoke" those numbers, and I'd love to be the first to eat those words!"

    If I understand how Sun thinks, JBoss will not be allowed to post any ecperf results, because it is not certified.

    But that doesn't stop Marc and his friends from creating an equivalent benchmark and publishing the results on their web site. Certainly this will have an impact, due to the inevitable flame fight on TheServerSide. But I don't expect JBoss to be as fast as weblogic. (That's my opinion and please let's not start a discussion on this.)

    Guglielmo
  45. I don't believe that certification and ecperf are that closely tied - if at all. As I understand it, a vendor (certification doesn't seem to be a qualifiying factor - or at least I haven't found it in the specs with a quick look) can publish their results as long as they conform to the ecperf rules about full disclosure, etc.

    Anyone can download and run the tests for internal purposes, of course. And actually - that's where I think these tests are the most useful.


    Cheers
    Ray
  46. Ecperf rules require CTS certification. But then Ecperf 1.0 (& 1.1) being EJB1.1 based, do not require J2EE 1.3.

    Cheers,
    Ramesh
  47. Ramesh:
    "Ecperf rules require CTS certification. "

    Yep - sorry for my mis-statements earlier on that point - I just read the spec more closely.

  48. It is simply not possible to compare the results of different vendors in any useful way when they are using different hardware, different databases, different JVMs, etc.
  49. Thanks to all the folks out here..

    Atlast the ecperf has picked up !!

    Thanks to HP . Nice to see the products from different vendors working together to deliver a cost effective solution... :-)

    I believe it's not time to fight or argue over the different H/W & S/W combinations used by these folks.Who knows some dotcom sponsored by M$ might publish similar results where the price will be rock bottom low.And once again we will start bashing out !!!

    Tell me if all the AppServers are tested on a single
    JVM - O/S - H/W stack will they outperform each other to a very large extent.I don't believe so. Given the situation where the AppServers performance,stability etc are pretty much comparable..only the price of the AppServer will greatly influence the final result !!! ( JBoss will be the ultimate winner ).

    My sincere suggestions to the Ecperf team ..

    a) Technical Specs & Commercial Specs shouldn't be combined together. I believe ECPerf focuses more on what ROI(Return of Investment) do i get from the product stack. So Sun should focus on only conformance !!! & nothing more.. Hence ECPerf shouldn't be a part of the TestSuite !!!
     
    b) There can be a independant vendor ( theserverside ) or someone who can come out(with or without sponsors ) with possible combinations of different J2EE stacks. Any one can look & choose the right stack for their business.
    e.g
    Sun Machines + Weblogic + Oracle
    Intel/AMD + JBoss + PostgreSQL/Oracle/DB2

    Let me put in this way. If a company has only Microsoft products then it sounds fine for them to choose a Windows2000 based stack !!!

    This is when i believe that J2EE is in the right success track :-)


  50. Why did WebLogic not relese this test for 6.1? A beta product does not help me now. WebSphere 5 is due out soon, I'm sure there will be some new results for that. It will be a back and forth fight.

    Where is everyone else?
  51. A couple of points to keep in mind:
    1. Performance and price performance is just one criteria for comparing appservers. (And in my opinion,not the most important -by far.)

    2. Not only does ecPerf minimize the noise and reflect the true superiority of WebLogic from a performance perspective it is consistent with it being superior in all these other respects:
    a. Market leadership
    b. Spec implementation leadership.
    c. Number of ISVs and Partners.
    d. Usability and Administration.
    e. Robustness (clustering and failover capabilities)
    f. Industry awards.
    g. Overall completeness/Purity of j2ee implementation.

    3. Its really becoming an IQ test as to which Product is truly superior. (This might actually mean that theserverside will soon become theweblogicserverside ;-)

    Matt
  52. <quote>
    3. Its really becoming an IQ test as to which Product is truly superior. (This might actually mean that theserverside will soon become theweblogicserverside ;-)
    </quote>

    Matt,
    No offense, but the ECPerf is hardly an IQ test as to the supierority of a product - including BEA. No results published for ECPerf (including BEA) have minimized the noise for anyone. That's actually why we are discussing how ECPerf could actually be used to do just that - minimize the noise and go beyond vendors stroking themselves. I'm glad vendors are at least testing the waters, but to me, the information is far from useful and doesn't do anything specified in your post (in my opinion).

    Cheers
    Ray
  53. Ray,
    You misunderstood my point. to clarify, performance is one of many aspects of quality/value and its not surprising that weblogic has smoked websphere in this respect as well.
    matt
  54. Hi Matt -
    I actually got the point you were trying to make, but I disagree with it. Ignoring your marketing talking points on BEA (item 2), I don't believe that anyone can reasonably make those assumptions (1 or 3). I'm saying that it is actually difficult to make the assumption that BEA "smoked" IBM (or IBM smoked Borland - and the next vendor that smokes BEA) in any sort of real sense because of the nature of how the test is performed. Show me a test of the two app servers on the same platforms with the same database engines, and then you can tell me that one smoked the other.

    Cheers
    Ray
  55. Hi Ray,
    None of the points I made are based on marketing (although they do sound good!) They are actually based on data and/or accepted principles.

    for example:
    Developer/Administrative productivity is normally regarded as the most significant component of TCO.

    Gartner, Giga, IDC etc. all recognize WebLogic as the market leader. (This is important if you want to find developers and 3rd parties supporting the product.)

    Websphere currently uses MQSeries for JMS. Based on J2EE 1.3's requirement to provide a JMS implementation in order to be certified, IBM is having to rewrite MQseries in Java.
    (So this will be an unproven 1.0 implementation.)

    One of the best ways to validate my "marketing" points is the fact that Websphere's CMP implementation and ( in certain versions of the product the Configuration repository) is tied to DB2. So what if we ran the ECPerf comparison you are asking for. Would WebLogic be forced to use DB2 (an arguably inferior product to Oracle) or would WebSphere be disqualified?

    I can provide additional backup on any of these points if you are interested. Then you will see why #3 is true as well.

    Matt
  56. Matt -
    Thanks for your posts - however, I am not interested in BEA or IBM in particular. I'm also not terribly interested in Gartner, et al. either. Nor are my clients, actually. What they *are* interested in is the true total cost of ownership, as you like to say, and I am saying that the current set of results (and let me re-iterate that I am happy that vendors are trying the tests out) doesn't really help anyone make those decisions. Market leadership in j2ee app servers doesn't lead to lower cost of ownership as it might with other items - a good j2ee programmer can work with any app server with relative ease. Finding developers isn't a problem. Components written to the j2ee spec (should) work on any j2ee compliant app server regardless of its position in the market hierarchy.

    Cheers
    Ray

  57. Ray -

    No one measure can give a good picture of the total cost of ownership. ECPerf it is not irrelevant; it is just one of the measures to be taken into consideration. A lot of factors are based on the needs, constraints and existing infrastructure of the client. Can you tell me one single measure that (even theoretically) indicates a total cost of ownership that fits all scenarios?


    " Components written to the j2ee spec (should) work on any j2ee compliant app server regardless of its position in the market hierarchy. "


    Guess you haven't really tried to migrate a full fledged J2EE enterprise app across two vendors .... :-). And even if it was all that easy ..... One of the primary factors in getting a good performance is tuning the app and database servers. This requires product specific knowledge and experience. The concept of a global pool of J2EE programmers fits only low end requirements.

    Samrat
  58. Hi Samrat -
    Thanks for your comments. However, I never meant to say anything indicating that I thought there was one indicator for determining total cost of ownership - far from it. Sorry if I misled you or I wasn't clear. And while you might think that test-bed ECPerf testing is academic - if you are going to do ECperf testing at all and an organization deciding on app servers is going to evaluate those results at all - test-bed scenarios of the app servers in question are, to me, the best way to help evaluate the results.

    Luckily, I have moved full fledged j2ee applications across vendors... :-) and I found having an expert in j2ee was *at least* as useful as some app server vendor expert. I think if you deploy poorly designed ejb code (for instance), you can tune the app server until the cows come home and it won't do you any good. Also, I didn't mean to construe a global pool of j2ee programmers coming in to save the day - just that *good* j2ee programmers are worth a lot to any project - including the migration of applications between vendors.

    Just my experience

    Ray

  59. "None of the points I made are based on marketing (although they do sound good!) They are actually based on data and/or accepted principles."

    This statement makes you wonder what marketing is usually based on ;-)
  60. Touche! Let me try again...
    "None of the points I made are based [merely] on marketing (or Hyperbole) (although they do sound good!) They are actually based on data and/or accepted principles."

    This statement makes you wonder what marketing is usually based on ;-)
  61. Having architected many J2EE applications on different app servers (Weblogic 4.x/5.1/6.1, NAS40, iPlanet6.0, Websphere 3.x) for different firms, it has become obvious that an app is server is a very important part, but yet only a part of the entire solution.

    A firm does not opt for an app server by looking at it in isolation. Its a piece in the puzzle. If I opt for an IBM solution, then I would the best that combination has to offer. If I go for a Weblogic, I would again go for the best combination there is. What I am really interested is ....which is the best among these *combinations*? The complete solution....the big picture.

    That is why I believe demanding a comparison test against an uniform platform is simply academic. It has no significance in real world. ECPerf is a pretty good measure of the entire solution that a firm has to offer and that is indeed a lot more relevant.

    The choice of an app server depends on many factors including :
    1. Criticality of the application. A critical application requires high uptime, robustness etc. and customer support from the vendor is a big thing here.
    2. Existing applications. Are we starting from scratch here or do we already have a data warehouse on Oracle. What has been the primary platform till date? Does my firm have the skills to administer a Solaris or a HP or an AIX box?
    3. Database intensive applications. In DSS systems, the time taken by an app server is negligible in comparison to that taken by the database operations. Does my app server allow me to leverage the best features of the underlying data source? (e.g. Oracle SQL hints)
    4. The cheapest solution is not necessarily the best....
    A free app server does not make the entire solution free. As a matter of fact the biggest cost for a large and critical application is developing and maintaining the application.

    We need to keep in mind current constraints and have flexibility for the future. What is database vendor A comes up with a new feature in its next rev and it is just the thing I need. Is my app server compatible with vendor A? Or would I be locked in?

    Let's look at the solution in totality,
    Samrat
  62. Websphere's CMP implementation and (in certain versions

    > of the product the Configuration repository) is tied to
    > DB2.

    This is false and has been for some time. WebSphere's CMP implementation supports DB2, Oracle, Sybase, Informix and MS SQL Server. There are also numerous customers running their AE config repository in production on Oracle, even on the previous WS release 3.5.

    > So what if we ran the ECPerf comparison you are asking for.
    > Would WebLogic be forced to use DB2 (an arguably inferior
    > product to Oracle) or would WebSphere be disqualified?

    Of course not (although I'm not going to step into an argument here on Oracle vs. DB2). Quite a few WebSphere customers use Oracle. Quite a few use DB2. I'd guess there are some that use both, along with the other DBs.

    Randy
    Views are my own and not necessarily IBM's
  63. My initial impression is that the difference in the results reflects the fact that IBM's hardware is behind Intel's in both price and performance. But I don't think we needed to see an ecperf benchmark to know that.

    I just don't see how two mature app servers can be that different inside ...

    Guglielmo
  64. Guglielmo,
    The difference between the products is like night and day!
    A few differences:
    Performance (WebLogic is better but especially with EJB's and Clustering)
    Administration (WebLogic allows servers and Apps to be managed centrally without being a single point of failure)
    Development (WebSphere has consistently been very late in delivering implementations of the spec. Websphere CMP depends on DB2 exclusively. etc.)
    JMS (Websphere depends on separate MQseries product for JMS implementation and will now need to rebuild in Java.)

    let me know if you need other examples,
    Matt
  65. The difference between the products is like night and day!


    Exaggeration.

    > A few differences:
    > Performance (WebLogic is better but especially with EJB's and Clustering)

    Where's the proof? ECPerf results just came out and being digested. As you can see from this thread, it's far from truth-telling yet.

    > Administration (WebLogic allows servers and Apps to be managed centrally without being a single point of failure)

    ??

    > Development (WebSphere has consistently been very late in delivering implementations of the spec. Websphere CMP depends on DB2 exclusively. etc.)

    Not true anymore. WebSphere just beat WebLogic on J2EE 1.3 certification, about a month eariler.

    > JMS (Websphere depends on separate MQseries product for JMS implementation and will now need to rebuild in Java.)

    Not true anymore. 5.0 will include JMS implementation.
  66. "The difference between the products is like night and day!
    A few differences:
    Performance (WebLogic is better but especially with EJB's and Clustering)
    ...
     let me know if you need other examples, "

    Actually I was only talking about performance. This thread is about ecperf which is a performance benchmark.

    What I would really like to see is a point-by-point comparison of architectural decisions for WebSphere and WebLogic.

    If you are going to write an affordable, fast and scalable ejb server, my guess is that these are the important things to do:

    1. Choose the fastest jdk you can find.
    2. Port it to the fastest platforms.
    3. Implement pooling of resources, specifically jdbc connections, threads, beans, etc ...
    4. Implement TCP/IP multiplexing.
    5. Implement a coherent cache (so you can scale on a cluster without overloading the database). I don't know if the ecperf benchmark benefits from a coherent cache, but if it doesn't we probably need a benchmark that does.

    Now, I would expect that WebSphere and WebLogic both have all these features. Hence my observation that on the same hardware they should perform similarly.

    I actually expect that WebLogic will do slightly better than WebSphere on smaller hardware, but that if you are willing to pay through the nose WebSphere will eventually take advantage of the custom hardware of parallel sysplex (this would be cool to see, but maybe they will never do it), and achieve really high throughput and really low latency for really nasty workloads.

    Guglielmo


  67. "5. Implement a coherent cache (so you can scale on a cluster without overloading the database). I don't know if the ecperf benchmark benefits from a coherent cache, but if it doesn't we probably need a benchmark that does."

    I've heard that a few app server vendors are considering this ;-)

    Peace,

    Cameron Purdy, Tangosol Inc., http://www.tangosol.com/
  68. ECPerf- apples to apples?[ Go to top ]

    Ecperf is certainly allows apples-to-apples comparison from a Total cost of ownership standpoint. While app servers may want to compete in isolation (meaning using head-on comparisons), the customer is more interested in the total solution. And any comparisons HAVE to be at that level. And this is the spirt behind the benchmark.

    Coming to the differences, in a competetive market, as the market forces determine the pricing, the hardware pricing WILL absorb the benifits from a piece of hardware. Performance being an important aspect, any hardware that performs better will be priced higher (by and large!). And when there are different classes of hardware (say extremely fault-tolerant), the app-server vendors will certainly have numbers on all classes of hardwares. So within a class, the comparisons will still be possible!

    The benchmark further levels the field by clearly stating all configuration and deployment settings that is normally specified by the deployer/user. So the users see the same view of the application on any ecperf run. App server Vendor specific optimizations would be strictly internal to the server, not altering the app semantics seen by the user.

    The benchmarks, while certainly may serve short term one-upmanship mktg goals, the long term utility is for Users to compare results based on their reqts and make a determination as to the best combination of hardware, JVM and app-server for their solution needs.

    - Ramesh
  69. ECPerf- apples to apples?[ Go to top ]

    Hello,

    IMO the ECPerf-results are good for determine the facts what performance is really possible and in combination to that what are the TCOs of the vendor's scenario. Neither less not more. That means BEA is the new World Record holder, but not means BEA is the fastest Appserver. Only under the tuned conditions, i.e. VM, own JDBC-Driver, own proprietary protocol (T3) etc. they prove to be best for the demands given by the ECPerf-spec.

    So what did it mean to me as a developer? Not so much, only the suggestion, that BEA and former IBM has done a lot of work and tests for reaching this goal, in order to get rid of the attacks of M$ and Oracle.

    So for me this is not much more than pure marketing!

    Just my 2 cents

    Michael Stachel
  70. ECPerf- apples to apples?[ Go to top ]


    Marketing? Ok, in a way, but it also gives you an impression that they're really trying to do something to increase performance.

    I'm not so much interested of pure performance as of scaleability and clusterable state management.

    Of course then there are other little things like security, availability etc...

  71. "WebSphere will eventually take advantage of the custom hardware of parallel sysplex (this would be cool to see, but maybe they will never do it), and achieve really high throughput and really low latency for really nasty workloads. "

    Websphere for zOS and OS/390 already does this. It is designed from ground up to take advantage of the unique hw/sw features of the 390/zSeries platform such as Parallel Sysplex, goal oriented Work Load Management and all the other characteristics expected of a true OS/390 or zOS application. In your words, it is built to handle the "really nasty workloads", with the highest reliability, security and availability.
  72. "Websphere for zOS and OS/390 already does this. It is designed from ground up to take advantage of the unique hw/sw features of the 390/zSeries platform such as Parallel Sysplex"

    Tom,

    can you expand or provide links to this?

    Guglielmo
  73. I found this PDF manual which gives technical details of how WebSphere 4.01 (I believe) uses parallel sysplex (among other things):

    ftp://ftp.software.ibm.com/software/webserver/appserv/zos_os390/v401/bosp1mst.pdf

    On p. 227 it specifies how to set up data sharing, and basically it says that data sharing is done through DB2. This means that to share entity beans you have to call ejbStore/ejbLoad, then DB2 has to execute sql (or a stored procedure), which eventually sends/reads the data to the "coupling facility" (shared cache). The last step is fast because it is synchronous (no context switch), but the latency and throughput would probably be much better if WebSphere serialized the entity beans and sent them directly to the coupling facility.

    That's what I meant when I said "they may never do this".

    I wonder how well Oracle's Real Application Clusters performs against DB2.

    Guglielmo
  74. Thanks Guglielmo!

    Looks like DB2 is also a websphere dependency on Z/os and OS/390!!

    Should the ecperf spec dictate DB, JVM, OS and so on just to make the playing field level for IBM?

    an apples to apples comparison is unrealistic. We should realize that being able to choose the best jvm and the best database and a broader range of hardware/os platforms IS a valid and important take-away from these results.
    Matt




  75. "Looks like DB2 is also a websphere dependency on Z/os and OS/390!!"

    I think any technical person reading understands that this is not the case.

    Either WebSphere or WebLogic can use DB2, and on s390 DB2 has good data sharing. I am assuming that WebLogic is supported on s390?

    Guglielmo
  76. "I think any technical person reading understands that this is not the case. "

    Certainly weblogic on OS/390 can use DB2 but the issue is whether websphere can run without it:
    "The following z/OS or OS/390 elements, features, and components must be
    installed, enabled, and configured.
    ... (about a half-dozen other dependencies)
    DB2 Version 7.1.
    ... (about a half-dozen other dependencies)"

    Maybe I'm not technical or not able to read but this looks pretty bad. (esp. since its out of the same IBM doc you mentioned above.)
    Matt


  77. WebSphere 4.0 for the Mainframe is a different product, it is not the same WebSphere 4.0.x Advanced Edition. It was developed by differnent development teams. They solve different problems.

    WebSphere 4.0.x Advanced Edition can run on the Mainframe on a LINUX partition, but it not the WebSphere 4.0 for OS/390.

    WebSphere 4.0.x for distributed platforms can access different databases with different JDBC Drivers. It can do CMP code with different databases. If you host the application server on a distributed platform and access DB2 data on the Mainframe, then it uses JDBC. DB2's app driver (type 2) performs better than their type 4 driver.

    WebSphere 4 for the Mainframe is meant for hosting applications on the mainframe. Then it takes advantage of all the benefits of the Mainframe and interacts with CICS transactions, IMS, etc...

    Most typical deployments are hosting EJB wrappers or Connectors for things like CICS and have WebSphere 4.0.x on a distributed platform access data through a single Remote call. The Distributed Platformed WebSphere would interact with databases sitting on a distributed platform using J2EE compliant ways. This way the distributed platform does not have to worry about COBOL record layouts or Mainframe details.

  78. Matt,
    BEA deserves much praise for achieving the numbers they did. You are entitled to having a favorite product, but your post contains a great deal of misinformation. Hopefully it was not intentional. Websphere has not depended on DB2 for CMP or anything else as long as I have worked with it (over 2 years). It does not rely on MQSeries for JMS either, it just doesn't ship with a JMS implementation. It will work with any JMS implementation, although if you use it with the MQSeries JMS you can include messaging in a transaction that includes ORacle and Sybase. It also allows you to send/receive messages from other applications, something you can not do with the BEA JMS implementation.
    You should refrain from commenting on products that you are not familiar with.
  79. Correct me if I am wrong,
    on page 22, section 7.5.4 "Disclose Driver To SUT Protocol Used"
    "The RMI over t3 protocol was used during the tests."
    I cannot believe this is a fair test result. BEA should test with RMI-IIOP not their own T3.
    With EJB2.0, more focus is on EJB-CORBA interoperability, which relies on RMI-IIOP.
    There is big difference in performance from T3 to IIOP.
  80. Sheng Song:

    You can use any protocol as long as it is supported by the EJB Container ... see the following excerpt from the spec:

    4.12.2 The Driver communicates with the SUT using any standard protocol supported by the EJB Container,
    such as RMI/JRMP, RMI/IIOP etc.

  81. "4.12.2 The Driver communicates with the SUT using any standard protocol supported by the EJB Container,
    such as RMI/JRMP, RMI/IIOP etc."

    T3 is a standard?

  82. yes ... looks like you've been living under a rock all these years! :-)


  83. >> There is big difference in performance from T3 to IIOP

    I think that this may be less so now with WLS 7.0 than with 6.1 - I know they have done particular work in this area.

    We will have to see the performance results though when 7.0 is publically available.



  84. >> ...although if you use it with the MQSeries JMS you can
    >> include messaging in a transaction that includes ORacle and Sybase

    Shouldnt that be possible with any JMS implementation - so long as it is XA compliant? I didnt think that MQJMS had anything special here.


    While Websphere is (and should be) JMS vendor agnostic, WAS4.0 does rely on MQ for its non-standard "Message Bean" implementation (unless I am mistaken).
    Even though it supports other vendors - it does provide particular support for SonicMQ (automatically stores Sonic Administered Objects in WAS JNDI).


    I dont ever remember Webpshere CMP being dependant on DB2 - though I do remember that:
    a) it was a long time before WAS even supported CMP1.1 ()July 2001 !) - and
    b) *not* using DB2 for the administration database was a real pain (it certainly wasnt a 2-minute operation to use oracle - its not even a 2 minute operation using DB2. In the end we gave up trying with Oracle). Perhaps we needed a degeree in IBM product usability... ;-)


    While I think that some criticise Websphere unfairly at times - it is difficult to find avid fans of Websphere (other than IBM employees of course ;-)

  85. Hi John,
    check your IBM website it EXPLICITLY states this dependency on DB2:
    http://www7b.boulder.ibm.com/wsdd/downloads/T4Dfactsheet.html

    Also, Tom from IBM states in the WST4D forum that CMP 2.0 was not designed using JDBC and therefore might not work with oracle. Again, check your facts here:
    http://www7.software.ibm.com/w5ptforum.nsf/2b6f16d66fa7aa6e852566f9006476ba/
    edc48f21069ed88587256b35000b3fd4?OpenDocument&Highlight=0,oracle


    Matt





  86. Matt,
    I hesitate before jumping in this thread at this point but the DB2 thing with TD is a point in time statement about WAS 5.0 TD, not WAS 5.0 GA. WAS 5.0 GA will support the normal databases supported with CMP 2.0

    Thanks
    Billy
  87. Matt,

    Your statements are applicable only to the WS 5.0 Technology for Developers release, not to production-use WebSphere product releases including previous releases. WebSphere 4.0 and even WebSphere 3.5 support CMP to multiple databases, including Oracle. Because WS 5.0 TD is intended for developer use as a technology learning vehicle, and to allow it to be delivered to the market in an expedited manner, the O/S platform and database coverage was limited to Windows and DB2. The full WebSphere 5.0 product will remove these restrictions.

    Randy
    Views are my own and not necessarily IBM's
  88. Billy et al,
    Software is definitely a point-in-time thing -that's why they invented version numbers ;-)

    Seems like if the 1.0 CMP code was working so well in 4.0 why would they turning it off now? After all, once you're the first to be 1.3 compliant what's a little thing like support for Oracle?

    BTW, when is 5.0 going to GA?
    Matt
  89. Matt,
    OK, now I know what you mean. The Websphere 5.0 PREVIEW only works with DB2. That is not a limitation of Websphere 3.5/4.0. I am sure 5.0 will work with Oracle, Sybase, SQLServer and others DBs.
    After all, the Weblogic 7.0 Preview only runs on W2K but I am reasonably sure they plan to support other plaforms and databases. :)
    In any case, this thread is not about Websphere database support, it is about BEA's ECPerf numbers.
    JH

  90. Ha ha, is there an echo in here? Or have all the IBM employees logged onto theserverside?

    Just kidding guys... ;-)

  91. I think that its great that some of the IBM guys hang out here. They offer great info. It would be great if all of the app server vendors had people here (like Billy, Cedric, etc)
  92. Some remarks,

    1) Suppose I want to write a J2EE-app which integrates with an existing Tibco Messaging infrastructure. I don't give a damn whether or not my WebLogic server comes with a Message Oriented Middleware (MOM) implementation. I will use MOM to integrate with some (Legacy) EIS. I need a JMS interface/driver for my App Server, in the same way I need some JDBC interface/driver to access an RDB. I don't mind if my DB is from another vendor.

    Things differ of course if you use MOM to connect two J2EE App Servers, for instance.

    2) I am surprised to see that "WebSphere has consistently been very late in delivering implementations of the spec". I guess Mr. Gunter does not talk about J2EE 1.3. Two months ago, there were 2 (two) vendors who had a J2EE 1.3 compliant App Server available. One of theme was IBM. The other wasn't WebLogic.

    3) Moreover, I see that Mr. Gunter is convinced that WebSphere CMP depends on DB2 exclusively. Sometimes it is better to do some elementary checks before issuing such nonsense (ever heard of Oracle? SQL Server? Sybase? Informix? At least each of these RDBs are supported for mapping CMPs to). Unless Mr. Gunter's note was written a few years ago).


    Given the quality of the examples, I don't need any more of them.


  93. Bruno,

    Most of the points you raise have already been treated earlier in this thread - arguably in a less inflammatory manner.

    1) A bundled JMS is very useful for messaging within an application. Iona ship SonicMQ, Borland ship SonicMQ, BEA ship their own, Websphere ships... what? I havent found too much info on this new WebsphereJMS Lite...

    2) Websphere *have* been consistantly late in delivering implementations of standards. Its an established trend - seen over the last 2 years. An alpha release of WAS 5.0 doesnt change that - though it does signal that IBM have finally gotten the message that their customers would prefer they expended some effort keeping up. The proof will be in when a production version WAS5 is relased. I will also be keen also to see when JDK1.4 is supported by Websphere - I suspect late in 2002 if in 2002 at all (they are still on jdk1.3.0).

    3) WAS 5 alpha, released in December 2001, does only support DB2 (as well as only run on Windows). It is the first release of any kind that I have seen tied to a particular database...

    I dearly hope that they are catching up - I am just not about to assume they have just yet.

  94. I don't consider Bruno's arguments to be inflammatory. The posting he was responding to was inaccurate, deliberately misleading and deserved a much stronger response than anyone gave.
  95. John: "I don't consider Bruno's arguments to be inflammatory."

    Actually, I consider most of the posts to be inflammatory. If they weren't, no one would be interested in reading them ;-)

    I just wish I could tell where all of you worked so I could understand the thinking behind the posts. For example, do you work for IBM? Having to ask that is rude, but if you work for IBM and post something non-neutral on the topic without disclosing whom you work for, that too is rude. Maybe the registration here could require company name, and that could be displayed with the user name ... hint hint Floyd!

    Until then, I'll just keep posting with a footer, just in case.

    Peace,

    Cameron Purdy
    Tangosol, Inc.
    http://www.tangosol.com/
    "This space for rent: Your witty saying or aphormism could be here."

  96. "I just wish I could tell where all of you worked so I could understand the thinking behind the posts. For example, do you work for IBM?"

    Good point Cameron!

    I've often seen that the only folks who publicly defend WebSphere are the IBM employees ... it sure is hard to find a non-IBMer who likes WebSphere! :-)
  97. I just wish I could tell where all of you worked so I could

    >> understand the thinking behind the posts. For example, do
    >> you work for IBM? Having to ask that is rude, but if you
    >> work for IBM and post something non-neutral on the topic
    >> without disclosing whom you work for, that too is rude.
    >> Maybe the registration here could require company name, and
    >> that could be displayed with the user name ... hint hint
    Floyd!

    Here here. I totally agree.

  98. I was told it would be out of beta and public in April, but you can get a developer copy of the BEA web site.
  99. Greetings,

    First of all, congratulations to BEA team (and to all WebLogic fans too).

    So now, BEA has provided cheapest J2EE solution. Unfortunately, ECperf does not indicate what App Server is superior. It only displays price of the performed work. As long as App Server can perform certain amount of work, it is only matter of hardware and software price used during the test to display better results. Probably, if some App Server vendors will decrease price of its product ($0.99 per CPU or even give you cash back) plus use self-assembled server then they can be one of the leaders. In order to find out what App Server is better then hardware, software and DB must be the same in all cases plus same application must be used without any proprietary code or even proprietary configuration.

    By the way, I am wondering where is the Oracle who was claiming that its App Server three times faster then any other.

    Best regards,

    Taras
  100. "By the way, I am wondering where is the Oracle who was claiming that its App Server three times faster then any other."

    I would also be very interested in an actual ECPerf version of iAS 2.0-Orion. Knowing that Oracle is drastically cutting prices on their application server, they could have a nice $/BBop figure, but I'd be surprised if their performance reaches 46.000 BBops (3 times the fastest AS so far). Just like I'd be surprised if noone breaks their 'unbreakable' solution. Now wait! They already broke it after 2 months ? :))

    But I too think that ECPerf is a very nice step forward towards objectivity (in this world of hype and tout) but still, much progress can be made wrt the non-business requirements (*ability words).
  101. I would like to ask the BEA guys why the test used Oracle 8.1.7 and not Oracle 9, especially when Oracle claims that 9 is so much faster. Does this mean that 8.1.7 is faster, better supported, what was the reason?

    Cheers,
    Simon Reavely
  102. I have noticed that every time a new performance report comes out, there is all sorts of clamoring about the validity of the report, what hardware was used, what JVM was used, blah blah blah.

    While I know that there are many companies out there where speed it THE critical issue when it comes to an app server. For many of us, though, speed is only one of the issues - and maybe not THE issue. And I would bet a lot of companies do not need THE fastest server out there, just one that is "fast enough".

    For my money, if two servers are both fast enough for me (and most commercial app servers fit this requirement), then the more important feature is ease of use. If an app server is SCREAMINGLY fast, but is a pain to deploy and admin, to hell with it. Why would I want to save my users 1/10 sec per transaction that they will never notice it anyway, but cause myself massive anguish any time I want to deploy?!? Also, is paying thousands of dollars per license (per cpu/per machine) for an app server that is MUCH more powerful than I need worth it when I can get exacltly what I need for much less or (GASP!!!) for free?
  103. I'm glad you brought that up! In fact, the theme of our eWorld conference next week is Radically Simplified, Radically Unified. In Alfred Chuang's keynote address where he details Cajun, the future of application infrastructure, and all of BEA's announcements will be webcast on Monday.

    Usability and ease of use are incredibly important and that is a major thread that runs through all of our technology that will be discussed next week, especially Cajun.

    I definitely encourage everyone to check out what's going to happen next week.

    Tyler Jewell
    Director, Technical Evangelism
    BEA Systems, Inc.
  104. What JVM does BEA Ship with?[ Go to top ]

    JRocket? If not why did they used it for their benchmark? How about hte hardware configuration. Is it a real world production evironment or something optimized to deliver ositive test results?

    I don't know, looks like BEA cooked the books to me. Another Enron for the Benchmark world? :)
  105. What JVM does BEA Ship with?[ Go to top ]

    Dean Henkel:

    As long as the VM used in the benchmark is a supported configuration, why should anybody care? I suggest you read up on the ECperf review process before coming to any conclusions.

    I don't quite understand this ... earlier, the J2EE community was clamoring for an independent benchmark and when the vendors start publishing results, most folks are ready to trash them! :-(
  106. What JVM does BEA Ship with?[ Go to top ]

    George -
    I don't think people are particulary interested in trashing the results - in fact, I dont question the results at all. The problem is that the more results people see, the less effective they seem, and the more people (myself for instance) want to see more useful testing scenarios using ECPerf. Its certainly a good first effort, and its great that BEA, IBM, and even Borland (regardless of their recent retraction) put out results. But items like a common test bed, etc would make the results much more useful. I think people are interested in improving the process rather than trashing them. The BEA results are nice - they (and the others too, not just picking on BEA) just don't offer much in the long run. There were comments earlier that BEA might test on other platforms - that would be great! It gives a somewhat more realistic picture.

    Cheers
    Ray
  107. What JVM does BEA Ship with?[ Go to top ]

    Note:

    BEA Systems, Inc. (Nasdaq: BEAS - news), the world's leading application infrastructure software company, today announced the stock acquisition of Appeal Virtual Machines AB, a leading Java Virtual Machine (JVM) software firm

    The JRocket JVM

    http://biz.yahoo.com/prnews/020225/sfm110_1.html
  108. What JVM does BEA Ship with?[ Go to top ]

    Jay,

    The latest revs of WLS 6.1 on Unix do not ship with a JVM. They update all their shell scripts with whatever JVM you install the server under. As for supported JVM's they list
    1.3.1-b24 as their only supported JVM. I know that in the past our SE has said that BEA will support just about any JVM, but it always gave me an uneasy feeling. Sounds like the landscape is changing with this acquisition.

    ---
    Andrew Spyker
    Software Engineer
    http://spyker.dyndns.org
  109. What JVM does BEA Ship with?[ Go to top ]

    Andrew: "The latest revs of WLS 6.1 on Unix do not ship with a JVM. They update all their shell scripts with whatever JVM you install the server under. As for supported JVM's they list 1.3.1-b24 as their only supported JVM. I know that in the past our SE has said that BEA will support just about any JVM, but it always gave me an uneasy feeling. Sounds like the landscape is changing with this acquisition."

    There are two ways to look at it:

    1. Weblogic wants to move away from supporting all those JVMs.

    2. Weblogic needed its own JVM because it couldn't get proper support from "the JVM vendors", and this provides both leverage and insurance.

    I have reason to believe that it's all about number two, and not at all about number one. Hopefully JRockit will continue to be available separately, and I expect that the pace of improvement (particularly in terms of reliability) will continue. The up-side is that JRockit coming from BEA has a lot more visibility and corporations will be a lot more willing to depend on it.

    Peace,

    Cameron Purdy
    Tangosol Inc.
    http://www.tangosol.com/
  110. What JVM does BEA Ship with?[ Go to top ]



    I guess that while Weblogic has a selling point in being highly platform agnostic (JVM and OS), ECPerf (or at least the discussion in this thread) has shown that even though they have topped the numbers, when it comes to raw performance they are somewhat vulnerable. When it comes to performance, they do have a dependacy on the raw performance of the underlying virtual machine, the operating system AND the hardware - none of which are under their control.

    By aquiring JRockit, they remove *one* of those external dependancies.

    They will need to be in a postion where they are ready to compete (if indeed it becomes necessary) with IBM in providing a highly optimised, single-vendor platform (like Websphere for z390).
    Obviously BEA are not about to branch out into operating systems and hardware - but having their own JVM will bring a good return for their investment.

  111. Thursday February 21, 6:02 am Eastern Time

    Press Release

    SOURCE: BEA Systems, Inc.

    BEA WebLogic Server Achieves Record Performance and Price/Performance In Independent Benchmark

    BEA Platform Delivers 47 Percent More Processing Power at Significantly Lower Costs Than the Nearest Competitor

    SAN JOSE, Calif., Feb. 21 /PRNewswire-FirstCall/ -- BEA Systems, Inc. (Nasdaq: BEAS - news), one of the world's leading application infrastructure software companies, announced that BEA WebLogic Server(TM), the #1 market-share application server according to all major industry analyst firms, has set all-time world records for both performance and price/performance in the independent ECperf benchmark. ECperf is an industry standard benchmark that measures the scalability and performance of Java 2 Enterprise Edition (J2EE) servers.

    These independent benchmark results prove that BEA WebLogic Server is significantly faster and more cost-effective than the closest competitor with 47 percent more performance at 25 percent lower price per transaction. Actual results demonstrate that BEA WebLogic Server can process 15,180.43 BBops/min@Std at $20/BBops/min@Std. Comparatively, the closest competitor results show 10,316.13 BBops/min@Std at $27/BBops/min@Std for a similarly priced system.

    ``The results are clear: BEA WebLogic wins hands-down both in terms of value and raw power,'' said Scott Dietzen, chief
    technology officer, BEA Systems. ``The value to customers is twofold. First, these numbers definitively show that it takes less hardware to power WebLogic than the competition, dramatically reducing the total cost of operations. Second, for customers deploying large systems and applications, BEA WebLogic Server is the fastest performing, most scalable application server on the market.''

    The ECperf benchmark simulates actual business conditions to help potential buyers understand how application servers perform in demanding enterprise environments such as manufacturing, supply chain management, and order/inventory. The rigorous ECperf tests are conducted by the vendors themselves, yet provide an independent and auditable measure of the performance and cost of application servers. ECperf was designed to provide a credible third-party test and help customers select the best application server that meets their business and IT needs and criteria. ECperf stresses the ability of application servers to handle the complexities of large Enterprise Java Beans (EJBs).

    BEA's results were achieved in joint collaboration with HP. BEA WebLogic Server ran on two clustered HP LT6000R
    NetServers running Microsoft Windows 2000 operating system and Oracle 8.1.7 databases on a separate tier. Each HP
    NetServer ran four Intel 900MHz Pentium III Xeon processors. The Virtual Machine for Java(TM) was JRockit. ECperf is a
    trademark of Sun Microsystems Inc. ECperf was prototyped and built in conjunction with Java 2 Platform, Enterprise Edition server vendors and is being developed under the Java Community Process. Competitive numbers shown reflect results published on http://ecperf.theserverside.com as of Feb. 21, 2002. For the latest ECperf results, visit http://ecperf.theserverside.com/ecperf/ .

    About BEA

    BEA Systems, Inc. is one of the world's leading application infrastructure software companies, with more than 11,800 customers around the world, including the majority of the Fortune Global 500. BEA and its WebLogic® brand are among the most trusted names in e-business. Businesses built on the award-winning BEA WebLogic E-Business Platform(TM) are reliable, highly scalable, and poised to bring new services to market quickly. BEA's e-business platform is the de facto
    standard for more than 2,100 system integrators, independent software vendors (ISVs) and application service providers (ASPs) to provide complete solutions that fast-track and future-proof e-businesses for high growth and profitability. Headquartered in San Jose, Calif., BEA has 96 offices in 34 countries and is on the Web at www.bea.com.

    NOTE: BEA, BEA Tuxedo, and WebLogic, BEA WebLogic Server, BEA WebLogic Commerce Server, BEA WebLogic Personalization Server, BEA WebLogic Process Integrator, BEA WebLogic M-Commerce Solution, BEA WebLogic Collaborate, and BEA WebLogic E-Business Platform are trademarks or registered
    trademarks of BEA Systems, Inc. All other company and product names may be trademarks of the company with which they are associated.

    BEA will host its 7th annual Developer Conference in San Diego, Feb. 24-28. For details: www.bea.com/events/eworld/2002/sandiego.

    SOURCE: BEA Systems, Inc.
  112. These figures are too narrow. How can you summarise something as large as J2EE into a couple of numbers. Any statistician would have a field day with the credibility of these figures.

    What is really needed are sets of simplier figures that test specific features, not based on a complex system. Trust application developers to draw their own conclusion by looking at the performance of their own required features.


  113. Two things I thought were interesting which I don't believe have been commented on.

    Why did BEA use the oracle thin JDBC driver, instead of the native driver?

    Also, Websphere deployed only CMP beans, whereas weblogic deployed BMP and CMP - it would be interesting to see how weblogic would do with only CMP beans.

    Any comments?

    -Scott
  114. The thin driver has better performance in many cases.
  115. wow ... look at all the IBM engineers on this thread. I'm sure you folks have better things to do than desperately trying to dig up some dirt here :-)

    Scott Gilpin, re: the comment about thin driver, cmp vs bmp ... what does it matter? All these combinations are perfectly legal by the ECperf spec so BEA is free to use what they feel is appropriate.

    I hope you folks get over it ... an apples-apples comparison is just about impossible (fancy getting IBM to run on Sun hardware + Sun JDK + Oracle DB). BEA is probably the most vendor agnostic and therefore has the best chance of publishing results across a wide variety of platforms.
  116. Right on George! The benefits of that agnosticity, flexibility, and platform support will and SHOULD show up in the numbers.

    The only reason the test seems unfair is because websphere is very limited in the options it provides:
    No high-performance 3-tier protocol (ie.t3)
    Mainframe version doesn't work without DB2
    Older versions were not J2EE certified.
    Newer certified versions don't support CMP with Oracle.

    I can imagine that the websphere-like product that runs on the mainframe may be so tightly intertwined with the OS that it produces incrementally better performance than WebLogic on the mainframe. And when and if that happens we shouldn't claim IBM cheated or that the results are invalid except for the obvious caveat that it is a completely different product.

    Matt

  117. Try selling J2EE without Mainframe integration and Mainframe solutions to a Mainframe shop. IBM has multiple products to serve different customer needs.

    BEA has to be vendor agnostic in order to survive. They don't sell their own database, OS, or hardware.

    Microsoft has surivied on a single platform, and BEA is surviving now on openingness. There are benefits and downfalls to both. WebSphere supports Soalris, AIX, RedHat, OS390, ZOS, Oracle, DB2, Sybase, SQL Server, etc... You guys can keep saying WebSphere is tightly integrated with DB2 and AIX, but that is not the case.

    The preview of 5.0 is just a preview, just like WLS 7.0 preview only ran on limited platforms, the 5.0 preview is being previewed with DB2. However, I used it with Oracle and it works.

    (Views are my own and not necessarily IBMs)
  118. "BEA has to be vendor agnostic in order to survive. They don't sell their own database, OS, or hardware"

    I don't have to say websphere is tied to DB2; IBM documentation says it. Also, if your not tightly integrated with DB2, AIX, OS/390 then what's the point of NOT being agnostic?
  119. "BEA has to be vendor agnostic in order to survive. They don't sell their own database, OS, or hardware."

    Actually they have to be vendor agnostic because they probably don't have the money to optimize for each platform separately. Otherwise they ought to do it.

    I hope that IBM will continue to sell special versions of software that perform extra well on IBM hardware. Far from being a weakness, this is exactly what Sun wants you do with J2EE. Comply with the J2EE the spec, but optimize the implementation.

    Guglielmo
  120. Exactly right Guglielmo.
    Personally, I'd be suprised if all of the mediocre IBM products together provides a better solution than a best-of-breed approach -especially if there is no special integration as Roland claims.

    I can't wait to see those results.

    (Uh oh!,I almost forgot, we already have them!)
  121. Matt,
    Fun stuff, eh? And yes, we do actually have other stuff to do ;-)

    Nobody claims cheating or invalid results.

    The only comments I've made on this thread are really highlighting some of the tricks (and tricks is not a bad word) used to make the numbers. Things of genuine interest to readers I believe.

    The two JVM instance is an interesting tweak. Which is faster, one instance on 4 CPUs versus two instances in a 2x2 CPU pinned split. The two instance is probably a good bet to be x percent faster because of less contention for JVM stuff on anybodies product.

    The question is whether in a test like this, it's worth paying more dollars for RAM for this performance delta over the single instance approach. More dollars means a higher cost per bop. But, two instances means faster overall numbers. So, while even with WAS, two instances is slighter faster than one, it's a cost/performance question.

    For us, we used 1 instance so far. We may decide to use two instances if the performance delta weighed against the RAM cost made sense.

    So, why do I post, this is why. I believe this is 'interesting' for WAS or WebLogic users alike. I'm not interested in poking holes or saying vendor x is crap or making marketing claims about a product, whether ours or not. I'd enjoy a discussion over points like these on a moderated topic which I believe would yield great info for J2EE users. But, the marketing/wild posts etc detract from the usefulless of the whole thing here.

    Now, your limited options statement is a good example of what I dislike. It's like me making the following statement:

    "newer certified versions of WebLogic don't support Solaris?"

    But, thats not what I do because it gives a blatantly false impression on the real situation to casual readers. Of course, WebLogic 7.0 will run on Solaris, the previous statement, while factual is obviously deliberately designed to mislead in its omission of the 'truth' thats whilst the current certified WebLogic only runs on NT, the production version will, of course, run on Solaris.

    Now, if someone wants to have a decent discussion on ECPerf tweaks etc, then by all means and I'll contribute. It's not cheating, or claiming numbers aren't valid, it's discussing the various tricks used to get the 'best' number for the benchmark. But if this is what the threads turn in to, then frankly, you're not doing yourself any favours either.

    Thanks
    Billy
  122. The two JVM instance is an interesting tweak. Which is

    > faster, one instance on 4 CPUs versus two instances in a 2x2
    > CPU pinned split. The two instance is probably a good bet to
    > be x percent faster because of less contention for JVM stuff
    > on anybodies product.

    > The question is whether in a test like this, it's worth
    > paying more dollars for RAM for this performance delta over
    > the single instance approach. More dollars means a higher
    > cost per bop. But, two instances means faster overall
    > numbers. So, while even with WAS, two instances is slighter > faster than one, it's a cost/performance question.

    Can you explain this a bit more? We've seen recommendations from WLS on running once instance per single CPU, but we've also been told by WLS to run one instance per two CPU's in a processor set. I can understand if you allocate a single node of the server to a single part of your requests, but if all nodes are equal, how does it best make sense to work across available SMP?

    Without too much knowledge I would guess it would make the most sense to allocate once instance per CPU, so you don't need to worry about shared memory contention issues and scheduling between CPU's. Therefore, why didn't they run four instance per box?

    Thanks ..

    PS. I agree wholeheartedly. I've learned quite a bit from BEA's submission already. There are tuning parameters in this document that haven't been suggested blatantly in other online documentation (JDBC prep. statement cache size and servlet change modification checking disablement). There are more benefits from these tests than just being able to see one vendor trash another. We all win.

    Finally, I haven't seen it yet, but is there a place where you can download the source to the ECPerf submissions? Or can I assume that all ECPerf application code stayed the same with the packaging changes added from the FDA zip?
  123. Why not 4 instances.[ Go to top ]

    4 instances would requires 4 times the RAM which costs more, plus, it a diminishing return game at work here.

    Typically, an operating system loads an application and then if another process uses the same application then the application code is only loaded once in to RAM and shared between the processes using that application. The application data is of course not shared. So, we use memory for the application code once and then n times the application data.

    Java only uses a very small program. All the JDK byte code, app server, drivers, your byte code and all your data is data as far as the OS is concerned and not shareable.

    So, the OS gets to share maybe 20K out of a 1.5GB process. So, running 4 instances would require 6GB of RAM. ECPerf is cost senstive so it's a trade off.

    Billy
  124. Why not 4 instances.[ Go to top ]

    4 instances would requires 4 times the RAM which costs more,

    > plus, it a diminishing return game at work here.

    Totally understood. What kind of performance issues present themselves when you compare (one a 4 CPU box):

    1 instance running across 4x1 cpus with all of memory
    2 instances running across 2x2 cpus with 1/2 of memory
    4 instances running across 1x4 cpus with 1/4 of memory

    Ram is cheap right?

    One instance across 4 cpu's have the penalty of accessing memory across the 4 cpu's even though the threads are only running on a single cpu (ie. thread 1 and thread 2 are on CPU 1 and 2 respectively). Wouldn't this mean there would be contention for the memory across the CPU's?

    Would it not be smarter (as long as you're not considering the RAM price) to have each CPU manage its own memory space and then the CPU's won't have contention for the same memory?

    Of course this is assuming the OS/Hardware understands that not only the process but the heap is pinned to a CPU?
  125. Why not 4 instances.[ Go to top ]

    Billy: "4 instances would requires 4 times the RAM which costs more, plus, it a diminishing return game at work here."

    I don't agree with this assertion. If the OS uses 150 MB and the Java bin uses 10 MB and the app server uses 30 MB + 5 MB per exec thread + .25 MB per session then I can have 1 or 4 app server instances as follows:

    1 instance: 150 + 10 + 30 + 5 * 60 + .25 * 1000 = 740
    4 instance: 150 + 4 * (10 + 30 + 5 * 15 + .25 * 250) = 830

    My numbers are all made up and would vary from app to app and server to server, but you get my drift -- it isn't 4x for four servers but it might be 2x.

    The other numbers are the overhead of clustering (TCP/IP traffic and/or additional file IO) ... etc.

    As for my opinion on which is faster ... it's Weblogic with a good lead from my experience, but that's what ECPerf is for -- to get some blood thawing in this game. I'm rooting for you big blue, let's get some better numbers!

    I think we're seeing how the "consumer" and the Java market will win in the end.

    Peace,

    Cameron Purdy, Tangosol Inc., http://www.tangosol.com/
  126. Why not 4 instances.[ Go to top ]

    Cameron,
    For ECPerf, you're looking at a large JVM size, well over a gigabyte. If you look at ours or their number, the heap size for the JVM is like 1.5GB. I noticed a -Xns:512m on jrockit also but I don't know what thats for.

    Given this is so large, it kind of rounds off the OS usage if you see what I mean. You're sharing maybe 10MB per instance with a JVM size of 1.5Gb.

    So, 1 instance = 150MB + 1.5Gb = 1.65Gb
    2 instance = 150MB + 2 x 1.5Gb = 3.15Gb
    4 instance = 150MB + 4 x 1.5GB = 6.15Gb

    I'm not subtracting 10Mb from each instance because lets face it, 10MB on a 1536Mb JVM isn't really playing in to this.

    Billy
  127. Why not 4 instances.[ Go to top ]

    Hi Billy,

    Billy: "Given this is so large, it kind of rounds off the OS usage if you see what I mean. You're sharing maybe 10MB per instance with a JVM size of 1.5Gb."

    With 4x server instances on one box, each one has to support a smaller number of concurrent requests and a smaller number of sessions. Memory usage, depending on the app, is largely a function of how many concurrent requests are being processed and how many sessions are being managed. I can explain in more detail if you would like.

    Peace,

    Cameron Purdy
    Tangosol Inc.
    http://www.tangosol.com/
  128. Why not 4 instances.[ Go to top ]

    Cameron:
    "With 4x server instances on one box, each one has to support a smaller number of concurrent requests and a smaller number of sessions. Memory usage, depending on the app, is largely a function of how many concurrent requests are being processed and how many sessions are being managed. I can explain in more detail if you would like. "

    Please do so. That is kinda what I meant by splitting a four cpu box into four nodes. Just because you need 1.5 gigs of mem doesn't mean you need 1.5x4 gigs of mem for 4 nodes unless you have a large ammount of cached data that isn't specific to a given client session.

    Given you split the memory between nodes, which would process requests faster? I've got to imagine it has something to do with contention issues.

    Thanks ..

    Andrew Spyker
    Software Engineer
    http://spyker.dyndns.org
  129. Why not 4 instances.[ Go to top ]

    I think putting out these "preview" releases to get J2EE 1.3 certification and mess with the benchmarks is rather shady. And they've all been doing it so I'm not just targeting BEA. You'd think more integrity would be demonstrated in an Enterprise sector.
  130. Here we go again ...[ Go to top ]

    What's wrong with preview releases?

    btw, there are 100 msgs on this thread already ... and a fair number of them are from the IBM trolls. Is this their day job? Hey IBMers, you folks hiring? There are lots of folks who would love to sit around and get paid to do nothing but post meaningless drivel on the various message boards.
  131. Here we go again ...[ Go to top ]

    <quote>
    btw, there are 100 msgs on this thread already ... and a fair number of them are from the IBM trolls. Is this their day job? Hey IBMers, you folks hiring? There are lots of folks who would love to sit around and get paid to do nothing but post meaningless drivel on the various message boards.
    </quote>

    Actually, I would classify this to be the most meaningless and inflammatory post that I've read so far in this thread.

    As long as the discussions are relevant and informative, does it really matter whether or not they work for IBM, BEA, Microsoft, or [insert company here]?

    Most of the people that post on this site tend to favor one product or another. Who cares? I don't see anyone trying to hide the fact that they work for IBM or BEA. Intelligent debate over the merits of WLS, WAS, and ECPerf is welcome as far as I'm concerned.

    Leave the meaningless drivel flame posts to other sites whose members appreciate that sort of thing.
     
    -- jason

    BTW, JBoss rules the world! Bow down before JBoss! :-)

  132. Why not 4 instances.[ Go to top ]

    Hi Andrew,

    Cameron: "With 4x server instances on one box, each one has to support a smaller number of concurrent requests and a smaller number of sessions. Memory usage, depending on the app, is largely a function of how many concurrent requests are being processed and how many sessions are being managed. I can explain in more detail if you would like. "

    Andrew: "Please do so."

    Let's take a typical four-way server, the Sun e4x0. This server typically is purchased with 4GB of memory and 4 processors, although sometimes you'll see it with 4GB of memory and maybe 2 processors. I'll pick this server because it's a very popular model for running Weblogic on (and humorously enough it's also very popular for running Websphere on ... but try getting a build of Websphere that runs on Solaris without running the gauntlet with IBM sales).

    So I could set up an app server to use about 2GB for the heap (typical max for 32-bit JVM implementations) which means that if I only run one instance and the OS, then I'm wasting a good deal of memory. So I'm probably going to run at least two instances. Let's say I give them 1.5GB each for a heap, so now I'm using "most" of the 4GB (particularly since Java uses a lot more memory than just the heap that Java objects are allocated from, for example if you are running a native JDBC driver like Oracle OCI).

    So my app is running along serving up pages (maybe JSP) with some tags and some string concatenation somewhere and the JVM is allocating some objects and cleaning most up (the new generation heap) but eventually it decides to do a full GC and ... ... ... ... ... ... it takes about 20 seconds.

    So from an actual production deployment perspective (which may have nothing to do with ECPerf) you might want to run 3 or 4 instances on the box so you can _effectively_ use the memory that the box has. You might also pin each to a CPU, although YMMV.

    Andrew: "That is kinda what I meant by splitting a four cpu box into four nodes. Just because you need 1.5 gigs of mem doesn't mean you need 1.5x4 gigs of mem for 4 nodes unless you have a large ammount of cached data that isn't specific to a given client session."

    That's part of it. There's user session data too ... if you have more server instances, you can split that memory requirement.

    However, memory isn't often a limiting factor for the running of an application, although it may limit how much you can cache (we've got particular experience in this area, if you hadn't already heard of our Coherence product that caches and shares data transparently and in sync among all cluster members).

    Andrew: "Given you split the memory between nodes, which would process requests faster? I've got to imagine it has something to do with contention issues."

    Surprisingly, all other things equal, there isn't a huge difference between 2 servers with 60 threads and 4 servers with 30 threads. Context switches are expensive, but only in the sense that an assembly programmer calls expensive. One JDBC operation is probably 10,000 more expensive than a context switch ... so that's not a big deal. Pinning processes to CPUs can also help (sometimes ... a little ... if load is even among servers) because the switches are "light" because the process itself will get switched less, which reduces CPU cache loading among other things.

    The problem is that having 2 (or more) servers instead of 1 often introduces its own set of issues, like keeping things synced between multiple servers or replicating sessions to handle failover. Since most high-end apps require those QOS features, you've already been forced to accept their overhead, so why not run a couple extra server instances if it makes sense.

    Look at two simple scenarios and select the better: One box, one server instance, 2GB heap, 30 second full GC pauses max, no failover. Or one box, two server instances, 1GB heap each, 10 second full GC pauses max, failover of one server instance to another.

    Peace,

    Cameron Purdy
    Tangosol, Inc.
    http://www.tangosol.com/
  133. Why not 4 instances.[ Go to top ]

    Cameron,
    On the money ;-) but don't forget any non session specific state that may be need to be cached.

    If the app was stateless and didn't use any data caches then of course, you're absolutely right.

    I think 'depending on the app' is the key to this line of thought in the thread.

    Billy
  134. Billy sez "Now, if someone wants to have a decent discussion on ECPerf tweaks ..."

    How about we discuss the "special" pricing tweaks that IBM used in it's ECperf submission?
  135. Well George,
    The thing you have to learn with benchmarks is that you try to get the best number within the rules because thats what your competitor does.

    If the rules say package pricing is allowable then fair enough, it's allowable and all the moaning in the world won't stop your competitors from doing it. If the ECPerf leads decide to disallow it then they can do it but until they do, it's legal.

    If you don't like it then just concentrate on the hardware/os combination and the performance number. Pricing is so variable in any case.

    Billy
  136. Billy fesses up "The thing you have to learn with benchmarks is that you try to get the best number within the rules because thats what your competitor does"

    RIGHT!!!!!

    Now apply this to your numerous postings and you'll understand why it is better for you to quit trying to find holes (or as you term them euphemistically "tweaks") in the BEA/HP submission. As is evident from the acceptance, the result is within the rules.

    I wonder how many engineers IBM has dedicated to monitoring theserverside.com on a full-time basis? :-)


  137. Billy,
    No offense was intended.
    (unfortunately, those limitations that I mentioned were necessary and relevant.Maybe we need an ecperf to evaluate openness/limitations/complexity/etc. -yet another thread.

    So, really what the current ecperf numbers represent is the performance benefits of a truly open solution vs. a single vendor solution. Which, in the end, resulted in the outcome most folks expected.
     
    Perhaps the websphere-like product for the mainframe will be integrated enough into the OS to surpass the performance of an open solution.
    On the other hand, it makes websphere (distributed) look like a lose-lose compromise between not being (as) open and not offering superior performance (or price/performance) even within a pure IBM environment.
     
    I am also interested in what tweeks/tricks were used but the whole point of objective tests such as ecperf is to clearly identify the best(and worst) products and stacks. If I am reading the ecperf results wrong, let me know -this thread should be a "no-spin zone"! (Perhaps we could even get Bill O'Reilly to moderate? ;-)
    Matt
  138. <George Art>
    Scott Gilpin, re: the comment about thin driver, cmp vs bmp ... what does it matter? All these combinations are perfectly legal by the ECperf spec so BEA is free to use what they feel is appropriate.
    </George Art>

    Of course they are perfectly legal - that's not what I was questioning.

    From a customer point of view, if I have two app servers, one in which I can write EB's using CMP, and one that I have to write EB's using BMP to get similar performance, writing BMP could potentially take longer. This could increase the cost of my development time, increasing overall BBops/$.

    As for the jdbc driver comment - I'm just surprised they didn't use the native driver. For most applications I've seen the native driver run 20-30% faster.
  139. Try using Oracle and Websphere?[ Go to top ]

     
    what if I want to use Oracle and WebSphere? Then WebSphere licensing cost would increase substantially ... what a rip-off!


    ====================================================
    Posted By: Billy Newport on January 11, 2002

    First, the customer joins the passport advantage program. This is free. They then order the software. If they order the number of licenses for WAS 4.02 Adv and DB/2 in the test then they get points for each license purchased. This points for this amount of licenses puts you in "band D" which qualifies you for a discount which brings the WAS price to the one in the report. You don't need to be a huge customer, a single person company will get these prices.
    =====================================================
     
  140. Congats BEA etc. etc. etc. But where on earth are the details of the results?! I've search high and low but can only find waffle. WHERE ARE THE RESULTS?