New ECPerf Results: BEA Posts Best Price/Performance Figure

Discussions

News: New ECPerf Results: BEA Posts Best Price/Performance Figure

  1. BEA Weblogic Server 7.0 Beta has posted the best Price/Performance figure at an astonishing $7 BBops/min@Std, an $11 improvement over their last results; however, their performance results were more than 9000 BBops lower than their previous results. This time around, Weblogic was run on a Dell PowerEdge 4600 machine using Oracle 9.0.1.1

    Read Official BEA Announcemnet

    Check out BEA's ECPerf Results

    Threaded Messages (19)

  2. I'm finishing up a product for delivery that is planned to run on Sun boxes with Solaris. My main focus is performance at this point. The fact WL has been releasing performance data on W2K indicates (strongly, to me at least) that this is a better choice.

    I also saw unofficial discussions of comparing performances on Sun vs Wintel and the results seemed to point to the same direction. I'd be more than happy if some one can convince me otherwise.

    Bill
  3. Bill,

    I made some general Java performance tests on Intel PIII 900Mhz and Intel P4 2GHz(Windows and Linux), Sparc (Solaris), OS/390 (CICS and USS) and Compaq Nonstop Himalaya to compare the different platforms. On all platforms I used JDK1.3 (except on Compaq I used 1.2.2).

    The result showed, that the performance nearly increases linear with the processor's clock rate. This means, that the Intel P4 2Ghz PC was a lot faster than for example the OS/390 or Compaq machine (which have lower clock rates).

    If you deside for a maschine, you have to look which kind of application you have. If you have a lot of database transactions or you have to integrate some legacy systems than maybe the result will be totally different. But for pure Java code, Intel seems to be the fastest and cheapest platform.

    But you also have to look what quality of service you need for your application. I think the quality of service is like this (my personal opinion):

    Windows < Unix < (OS/390 - Compaq NONSTOP)

    Mirko
  4. Mirko,

    That's what I saw from what I have and elsewhere as well. Sure, QoS is important. But I (doesn't everyone?) still need the numbers. We used to say UNIX performs better but more expensive. Now the balance is tipping further towards wintel - cheaper and faster. But someone might argue on "overall" cost of ownership. Hey, we are all shortsighted, if you will - we need the numbers today!

    Cheers!

    Bill
  5. I don't think that your conclusion is right.

    Bea is 5th out of 6 in terms of performance. Maybe it's the cheapest, but that wasn't your main focus.

    In my opinion it's a dirty but legal trick of Bea to get to the top. When you can't get the highest performance in the price/performance ratio, use a cheap system.

    Ingo.
  6. It is getting interesting :-)

    I wish to see some benchmarks with linux also..

    If some one comes up a combination of
    Linux + PostgreSQL + JBoss ,I think that will be the cheapest solution & no one can ever beat them in Price/Performance...
  7. This is good one, Murali! I am totally agreed with you. With any positive performance results the price of the operations will be $0.00!

    Best Regards,

    Taras
  8. $0.00?!!! Who giving out the free hardware? Can I get some? j/k

    Seriously though, even the all open-source route will still have hardware costs and possibly support costs. The total cost will obivously be less than the other setups though.


    Posted by Taras Zhugayevich 2002-03-28 10:45:35.922.
    This is good one, Murali! I am totally agreed with you. With any positive performance results the price of the operations will be $0.00!
  9. You absolutely right Chris. Of course hardware will have some cost. But you can build cheap one from parts. :)

    Regards,

    Taras
  10. No no not JBoss, it's too slow !!!!!!!!!!

    If you want a benchmark using freesoftware use JOnAS

    http://www.objectweb.org/jonas/

    JOnAS is twice faster than JBoss ( TheServerSide would not publish a benchmark about it ... funny not ?)

    http://www.cs.rice.edu/CS/Systems/DynaServer/perf_scalability_ejb.pdf

  11. >> In my opinion it's a dirty but legal trick of Bea to get to
    >> the top. When you can't get the highest performance in the
    >> price/performance ratio, use a cheap system

    I think that showing that you can get good (even better) perf/cost on cheaper hardware is exactly what people want to see. The ability to scale down as well as up is very important.

    On the other hand, using the same kit and lowering the SW/HW prices in order to get a better result *would* be dirty - but it would still be a win for the consumer.

    So long as getting good ECPerf results doesnt distract the Appserver vendors from the main game, I think that the performance competition can only be good.

    I think that with J2EE, Sun have made two very good decisions that have been successful in differentiating it from previous standards efforts; A solid certification process and now an honest performance comparison.

  12. Greetings,

    First of all: ECPerf results did not help me to choose App Server for my own environment. I can not afford system for $100K. For now I see only one result: WebLogic + DELL PowerEdge but where are the competitors? Where are the results from which I can choose one that satisfies most of my needs? Since, WebLogic posted their first results I have stopped reading ECPerf results just because it is mater of cheap tricks in order to became leader of the ratings. Maybe, ECPerf is not needed at all? Maybe, it is just waste of time and money?
    All these tests results seem so far from reality. In the real world load on your system in most cases will be unpredictable, you can try to catch up with it, but it is hard to predict what is going to be in future: too many variables involved. Why don't ECPerf’s Common (exactly same for all) Driver/Emulator provides stress level load (example: all ATM devices in the world connected to single System) and then let's see what kind of test results will be produced.
    Of course the ideal case would be if all ECPerf tests will run in the standard, same exactly for all environment but those tests will show only WHICH APP SERVER IS BETTER (which probably can be more helpful for me) but not WHOSE SOLUTION IS CHEAPER.

    This is question for ECPerf team:

    Why each vendor (Borland, IBM, WebLogic) uses their own Driver/Emulator? Would not be it more efficient to test all this systems using same Driver/Emulator.


    Thanks in advance,

    Taras
  13. Why each vendor (Borland, IBM, WebLogic) uses their own

    > Driver/ Emulator? Would not be it more efficient to test
    > all this systems using same Driver/Emulator.

    Am I missing the point or is everyone else missing the point?

    The point of ECPerf is that the vendor cannot touch the *application*. The application is *identical* for every run of ECPerf. Everything else is fair game. If a vendor can tweak the deployment or tweak a driver or tweak a JVM, or use better hardware, more power to them. What they are trying to show is that their product can run this *application* faster or cheaper than anyone else with their particular product/environment. If one vendor has more tricks up their sleeves than another, that is exactly the point (as long as these tricks are documented and can be used by customers in the same way).

    cc
  14. Greetings,

    I think that Driver/Emulator is a component that simulates clients or users. If I am correct it is a component that provides load (sends requests to do some work) for the tested platform. So, my point is to test provided solution (servers) indepently. Same load (big or small does not metter, probably bigger is better) must be provided in all cases.

    You can use tricks described by the vendor to tune up your server, but you can not tune up client's PCs.

    Best regards,

    Taras
  15. I disagree that it's a waste of time and money. Whilst who is faster or cheaper is basically a point in time statement as we've demonstrated, ECPerf is definitely forcing vendors to tune their application servers in order to be competitive. I can see this at IBM and I'm sure at BEA and other vendors, it's the same.

    They also highlight topology choices that are at least driven from a bang for buck scenario for various application servers. You can see this from studying the various FDAs on this site.

    The customer wins in this situation as the application servers are faster as a direct result of ECPerf.

    Billy
    (these are my own views and don't represent my employer, IBM).
  16.     I agree with what you have said here Bill. All us Technos can quibble about pricing, but pragmatic/conservative corporate America wants market leadership and the associated benefits.
        Being an IBM shareholder, I am pleased that IBM seems to be winning (at least as far as published results) at the high end $/BBop, even if it is largely due to superior bean tuning.
        IBM should shortly publish a high end 2-way (1 AppServer) number that will impress us. There is no trickery going on here. IBM owns the whole solution. This is a real customer value proposition (albeit not for all), not a trick.
        This will be an interesting battle for the Lion's share of the AppServer $. I don't think there is any question that it will be a fight between IBM and BEA. I would like to see IBM capture a larger share of the installed EJB base as opposed to seeing some financial accounting numbers where IBM is telling us 34% of $ in the AppServer market.
       The smaller folks will capture some of the technologist and innovater market, but will just not get any leverage with the mainstream market. Oracle is a joke and Sun has basically lost a year reorganizing.
                       GS
        
  17. <snip>
    Of course the ideal case would be if all ECPerf tests will run in the standard, same exactly for all environment but those tests will show only WHICH APP SERVER IS BETTER (which probably can be more helpful for me) but not WHOSE SOLUTION IS CHEAPER.
    </snip>

    ECperf is designed to neutralise any differences in hardware. The BBOPs is an indication of which is better and the price/BBOP an indication of which is cheaper. Allowing comparison across heterogenous configurations.

    Abt any optimizations specific to this 'application', it is strictly disallowed per teh benchmark rules. Now there have been some ambiguities that have been used to the vendor's advantage (like the data-stomping/data-corruption that happens as no concurrency control was used in their runs). Even these ambiguities have been fixed in 1.1 spec.

    On the whole, ECperf is certainly the best thing, next only to CTS, that happened to give a boost to the EJB technology!

    Cheers,
    Ramesh
    - Pramati
  18. Greetings,

    I agreed that you can start your project on the right foot based on the ECPerf results. But you can never predict how one or other Application Server will work in YOUR own environment where you do not have dual Xeon-CPU Server with 2GB of RAM. You can use advises how to tune up your server but that's it - you can not jump over hardware that you have. Therefore, in this case I will be more interesting in the App Server that shows best results in same environment, so I can predict which one will show best performances in my environment.

    Best Regards,

    Taras
  19. "Therefore, in this case I will be more interesting in the App Server that shows best results in same environment, so I can predict which one will show best performances in my environment. "

    This will be difficult. At best it may be possible to get it on a given platform. But any dependancies on other platforms (where a cetrain server may perform better or worse will still remain unknown). The best solution to your problem is to run the benchmark in your target hardware by yourself. The benchmark is supposed to be easy to setup and run (if it isnt, pls let any of us in the expert group know, we will fix it). And teh FDRs have some basic tuning info required by each server. Anyone can get a quick direct first-hand comparison.

    Cheers,
    Ramesh
    - Pramati
  20. Ramesh,

    I agree with you, that it would be the best to make the ECPerf tests in the customers environment. I really don't know Pramati, but I know that Borland ships a ECPerf test suite/environment with its app server, so you can run it without overhead and configuration "out of the box". I never saw this for BEA and IBM...this seems to me like they don't want a "real" comparison.

    Anyway, the tests makes it more transparent to me where the limits of the different app servers are and it is a challenge for all app server vendors.

    Mirko