Web Services Performance Kit compares BEA, IBM, SunONE

Discussions

News: Web Services Performance Kit compares BEA, IBM, SunONE

  1. PushToTest has released a Web Services Performance Kit, an open source benchmark to measure the performance of Web Services running on IBM WebSphere, BEA WebLogic and SunONE Application Server platforms. PushToTest has published initial Performance results on their website and has Sun One in the lead.

    Check out Intitial Web Services Performance Kit Benchamark Results.

    Press Release
    ----------------------
    The Web Services Performance Kit furthers PushTotest's mission to provide reliable, repeatable performance measurement tools for systems that drive Web Service-based business. PushToTest's products and technology is delivered under a free Apache-style open-source license. The Performance Kit delivers immediately useful tools and knowledge to enterprise information
    managers that are making Web Service infrastructure choices.

    The Web Services Performance Kit was developed by PushToTest to enable performance evaluations for systems supporting the SOAP protocol running on IBM WebSphere, BEA WebLogic and SunONE Application Server platforms. It features detailed step-by-step instructions to configure a Test Web Service on each server, plus a free open-source Web Service test utility called TestMaker to show the actual performance of each server. It is designed for
    software developers, QA analysts and IT managers that require performance data to support Web Service infrastructure purchase decisions. It also allows enterprises to test and fine-tune software applications under development.

    The benchmark fills the void for a simple test that shows Web Service throughput in today's shipping Web Service platforms. It models the stress placed on a Web Service application server from Web Service client applications making SOAP RPC-encoded requests to a Test Web Service (TWS) that simulates a database-driven Web Service.

    The Web Service Performance Kit is available immediately from PushToTest at no charge. Free downloads are available at
    http://www.PushToTest.com/ptt/saskit.

    -Frank

    Threaded Messages (32)

  2. BEA, IBM, SunONE Measurements[ Go to top ]

    Here are the details on what we found when using the Performance Kit to measure Web Service performance on BEA, IBM and SunONE platforms.

    SunONE Application Server
    Minimum transaction time: 140
    Maximum transaction time: 4506
    Average transaction time: 958.59
    Transactions Per Second: 43.62

    IBM WebSphere 4.0.3
    Minimum transaction time: 130
    Maximum transaction time: 11016
    Average transaction time: 2070.61
    Transactions Per Second: 21.18

    BEA WebLogic Server 7.0.0.2
    Minimum transaction time: 150
    Maximum transaction time: 1673
    Average transaction time: 379.73
    Transactions Per Second: 20.52

    The Performance Kit uses TestMaker and Apache SOAP 2.1 to make SOAP RPC encoded requests to a SunONE, WebLogic or WebSphere application server. We deployed a Test Web Service (included in the Kit) that responds to SOAP RPC requests. The average request size was 150 bytes and the response averaged 4000 bytes. We ran the test on a inexpensive commonly available systems: client-side is a 1.7 Ghz AMD XP processor system running Windows 2000 SP3. Server-side system is a dual 2 Ghz AMD MP system running Windows 2000 SP3. We ran the test for 3 minutes at 50 concurrent test agents each making requests. The test measures the Transactions Per Second (TPS) accomplished by each application server.

    We ran the same tests against servers running on a Sun Enterprise 450 with 4 X UltraSPARC-II 480MHz and 4 G RAM. The TPS results were within 10% of the Windows 2000 Intel results.

    -Frank
  3. BEA, IBM, SunONE Measurements[ Go to top ]

    I am a bit confused how WebLogic can have the lowest (by far) maximum transaction time and average transaction time yet produce the least number of transactions per second. Would you care to elaborate on this?

    BTW, it would help if you qualify the units in your results... are we talking milliseconds or hours?
  4. BEA, IBM, SunONE Measurements[ Go to top ]

    The transaction times are measured in milliseconds.

    The BEA results seemed odd... because they were. I typed in the wrong results. Here are the correct numbers:

    Minimum transaction time: 241
    Maximum transaction time: 4026
    Average transaction time: 1703.22
    Number of completed transactions: 4642
    Transactions Per Second: 25.65

    -Frank
  5. BEA, IBM, SunONE Measurements[ Go to top ]

    Sure, one is a bit faster - however the thing that strikes me is that all of them are pretty slow given the speed of the server. However, I havn't looked at the benchmark in detail - what's the nature of the business logic, persistence overheads, client load etc?

    Paul.
  6. BEA, IBM, SunONE Measurements[ Go to top ]

    Another revealing metric is "number of steps to build/deploy" for each AppServer. Based on the deployment instructions in the kit these are:

    SunOne: 9
    WebLogic: 18
    WebSphere: 27

    Very interesting.

    Paul.
  7. BEA, IBM, SunONE Measurements[ Go to top ]

    The updated results for WebLogic are different than the results displayed on your web-site.

    Original results you posted here:

    Minimum transaction time: 150
    Maximum transaction time: 1673
    Average transaction time: 379.73
    Transactions Per Second: 20.52

    The correct results:

    Minimum transaction time: 241
    Maximum transaction time: 4026
    Average transaction time: 1703.22
    Number of completed transactions: 4642
    Transactions Per Second: 25.65

    The web-site still displays WebLogic with 20.52 TPS, as mentioned in the incorrect original data.
  8. BEA, IBM, SunONE Measurements[ Go to top ]

    The updated numbers will be posted on the site later today.

    Also, it looks like we will be revising the Performance Kit to include Glue and Systinet implementations of the Test Web Service.

    -Frank
  9. BEA, IBM, SunONE Measurements[ Go to top ]

    The test is not justice for all parties since each application server will reach it optimizing performance for some specific platforms. However, the test were only carried out in dual 2 Ghz AMD MP system running Windows 2000 SP3 and Sun Enterprise 450 with 4 X UltraSPARC-II 480MHz and 4 G RAM. The other environment parameters are not known, such as, the types of database and so on.

    For sure, it is easy to setup in certain ways that some specific application servers looks better.
  10. BEA, IBM, SunONE Measurements[ Go to top ]

    There are many performance and scalability tests out there. This one is unique in that it is based entirely on open-source technology. My hope is that you will run the benchmark on your own equipment and learn the performance characteristics of your chosen server.

    -Frank
  11. BEA, IBM, SunONE Measurements[ Go to top ]

    Did something change in WebLogic license? I'm pretty sure it was very explicit in prohibiting licensees from publishing benchmark results - mine says "you may not ... or disclose the results of software performance benchmarks to any third party without BEA's prior written consent."

    Or is it part of the license forgotten now after that Pet Store performance comparision debacle?

    --
    Dimitri
  12. Not that there's anything wrong with...[ Go to top ]

    The Performance Kit focuses in on IBM, BEA and SunONE application servers. That's not to say that good performance is limited to these servers.

    I would like to extend the Performance Kit to include Glue, CapeClear, Systinet/WASP, Axis and .NET. Please send me email (fcohennospam at pushtotest dot com) if you are interested in volunteering some time to work with me on these.

    -Frank
  13. Not that there's anything wrong with...[ Go to top ]

    Hello Buddy,
          We will appreciate, if you avoid .NET from the serverside.com discussions.
  14. Not that there's anything wrong with...[ Go to top ]

    It is ridiculous what you say, if you don’t like .Net you don’t have to use this technology, but you cannot exclude this from the discussion. It is a free forum and people can compare and discuss even MS DOS 1.2 with MacOS 10 and I see nothing wrong with this.
  15. Not that there's anything wrong with...[ Go to top ]

    Yeah ,
       You can discuss as long as their approach is open. .Net is not as open technology, only Bill's kids know how it is working, they are not into open frame work as Java or J2EE. They wanna screw up the entire technology field, using their idiotic ideas.
  16. Not that there's anything wrong with...[ Go to top ]

    TQ,

    Gimmie a break will you. Significant portions of .NET are actually an open specification submitted and accepted by ECMA. Java and J2EE are not in any open standards body.

    Seems the only closed system around here is your mind. I for one wanna know how .NET stacks up.
  17. Not that there's anything wrong with...[ Go to top ]

    <Q>
    Gimmie a break will you. Significant portions of .NET are actually an open specification submitted and accepted by ECMA.
    </Q>
    Why do we always have to go over this?
    The core piece has been but not significant portions. Not enough to create Enterprise or Desktop apps (GUI). And nothing prevents MS from doing what they want - ie deviating from the standard. History shows they will.

    <Q>
    Java and J2EE are not in any open standards body.
    </Q>
    True. But it is an open standard. You can download the Java source. You can't download MS.Net source. Framework yes.

    <Q>
    Seems the only closed system around here is your mind.
    </Q>

    I'm with you on this one.

    <Q>
    I for one wanna know how .NET stacks up.
    </Q>
    I would love to too. But it is comparing apples with lettuce.
  18. Not that there's anything wrong with...[ Go to top ]

    I have to say that I agree with T Q. At the same time, calling someone's mind "closed" ... I find it a very rude comment ...

    I'm not gonna speak on behalf of other people, either those who disagree with me (and they are many I'd think) or those who agree with me, but I dont like having .NET here for the following reasons:

    1) I doubt that one can do apples-to-apples comparison between J2EE and .NET (if you want to know how .NET stacks then do .NET benchmarks, same for J2EE with the exception here that you can compare it against other J2EE implementations which you cannot do (yet) with .NET).

    2) this site, (or its parent company), after being secretly sold out to a third company, started quietly introducing .NET here (comparisons) (late summer) and some ppl raised issues about it ... Its last week's (fake) benchmark has cost it its credibility and neutrality in my point of view.

    When I look for .NET resource (and I do so more often than for J2EE resource lately) I go to msdn, when I look for J2EE resource I go to java websites (one of them used to be this site).


    Regards All,

    PS: Maggie (Mark may have answered this already) since .NET is more open that J2EE, you might be able to help me here: one thing that I miss with .NET is that I cannot see the source code of .NET classes I decide to inherit (not decompiled and I need to know more than what is exposed). How do ya get around with this? I hope this is a trivial question. Im not much of Microsoft developer (I know Im breaking my own rules here ... hehe ... but would love to know this). :)
  19. Predag,

    You seem to have forgotten what Encapsulation means. Why do you find it so necessary to know the src of a class you are subclassing? Especially why would you ever need to know more the the IL exposes?
  20. Not that there's anything wrong with...[ Go to top ]

    To fix problems when (not if) they occur or are found. (See Linux)
  21. Maggie,

    I think that Encapulation is a "goal" more than a "way".
    You would do me a favour if you give me only the interfaces of your classes that I want to use in my application but the same may not be (always) enuff if I want to subclass your class. It is simply a matter of knowing what you are building on top of and having control over it. Here is an example: some time ago, I was subclassing a Java class which had a bug in one of its parent classes (4 levels up its hierarchy). Knowing only the interfaces will not assist you much (yeah, you can find the method that breaks, but try asking yourself why and try fixing it).

    Not knowing the source code will not be an issue many times, however there are times when you will hit the wall and lose more time around it that you would expect.


    One approach that I especially dont like with the other side is "you dont need to know that" and "we've assisted you by hiding these things from you". Well, I build software, I dont only use it, and I need to know everything about it (especially the base classes, (BCL?)).

    Regards,
  22. .Not is not viable technology yet. I dont think it will ever be a viable technology in years to come. I would say .Not is a big flop. .Not just sounds good but when put to work it is same as other MS technology like II's, SQL server, commerce server, there is a lot of fluff in it but no real solution. Secondly .Not doesnt even run properly on the .Not server that is real surprise and all .Not runs is in windows. Java with its open source runs on all platform I dont think .Not will ever be to claim this. MS trolls wake up in the real enterprise world windows is not used but UNIX, Linux, OS390 and other flavor os OS's. What made Java such a viable platform is it specs and everyone's contribution to it you cannot say that about .Not(only MS runs it...I smell monopoly). I dont belive windows will be around long since the desktop is getting invade by Linux destop distro's now.
  23. Not that there's anything wrong with...[ Go to top ]

    Stop this .Net/J2EE slugfest already. I tired of listening to this shit. If you want to slug it out, get off this forum and take it somewhere else. I don't want to wade through 20 comments on why .Net/J2EE is better before getting to the comments on this performance kit.
  24. Source of .NET classes[ Go to top ]

    You can try:

    http://msdn.microsoft.com/net/sscli

    Unfortunately, not all .NET BCL classes sources are there but you will get a fair number, on the other hand you will get some interesting stuff for example the source of a C# compiler (written in C# of course). And it even runs on Mac OS. Intriguing stuff.
  25. Source of .NET classes[ Go to top ]

    <Q>
    You can try:

    http://msdn.microsoft.com/net/sscli

    Unfortunately, not all .NET BCL classes sources are there but you will get a fair number, on the other hand you will get some interesting stuff for example the source of a C# compiler (written in C# of course). And it even runs on Mac OS. Intriguing stuff.
    </Q>

    Best not try if you are developing commercial software (free or for money). Read the license (http://msdn.microsoft.com/MSDN-FILES/027/002/097/ShSourceCLILicense.htm). This too has been covered before (Their reasons for doing it).

    Now back to testing the speed of web services. Well, not me. I've got work to do.
  26. Not that there's anything wrong with...[ Go to top ]

    Who exactly are "We" Mister T Q? I think comparing J2EE with any other technology, especially .NET, is good for the community. If you are sure of the strengths of your favourite platform then all the better to have it compared with other (allegedly inferior) products.
  27. I think we are all missing one big piece of information here (re. WebLogic): from what I understand, the benchmark was done using WebLogic Workshop to develop and deploy Web services on WebLogic server.

    I've done some work with Workshop, and here is the thing: while it greatly simplifies and speeds up the development, testing, and deployment process for web services (not matched by any other tool that I've seen), the resulting application performs slower than if it was originally developed directly using WebLogic Server's Web services stack. Apparently, ease of use and powerful framework come at a cost... even though it's arguable what's important at the end of the day:

    1. How fast you can deliver the app to implement the required business functionality, and how expensive that will be to develop and maintain, or

    2. How fast the resulting application will perform run time.

    Obviously, different situations dictate different answers to these questions. My experience has shown that while you can develop a pretty powerful web service app with Workshop with unbeliavebly-low number of steps or lines of code, I have been able to achieve up to 30 times higher end performance when using WebLogic Server Web services tools directly. But it took longer to develop, and required more understanding of the underlying technology...

    In general, as with any benchmarks, I'd take all these very seriously: they only show that in the specific benchmark some particular server behaved faster. In real life, the real issues are so much more important.

    And to my opinion, ease of use, proven reliability and track record are much more important than some new benchmark.

    As usual, my .02

    G.
  28. Your argument makes a lot of sense to me. Incidentally the Performance Kit shows the relative efforts needed to implement the Test Web Service in Workshop versus using WebSphere Application Developer.

    I will implement the Test Web Service using WebLogic Server's build-and-deploy scripts and then test for performance. I'll post an update here when I have the updated Kit and performance numbers. Should take a day or two.

    Thanks.

    -Frank
  29. I added a BEA WebLogic Java-class Web Service implementation of the Test Web Service to the Performance Kit. This implements a Java class and uses the provided Ant build scripts to deploy an EAR file containing the Web Service. BEA's reliance on Ant is very elegant in its simplicity.

    I ran the test and found these results:

    Minimum transaction time: 140
    Maximum transaction time: 5227
    Average transaction time: 990.76
    Transactions Per Second: 42.04

    My next step is to implement the Test Web Service using BEA's EJB container-based Web Service code. I'll post that code to the Performance Kit too.

    -Frank
  30. Frank,

    You might want to be careful. It is illegal to publish the
    numbers for most servers and products without the approval
    of the vendor. The reason for this is that most products
    can be 'tuned', and hardware greatly varies and can impact
    and really screw up performance numbers.

    Without the approval and involvement of the vendor, your
    benchmark may not be valid or even correct. We integrate
    with virtually every app server, and we've found subtle
    differences in how Entity Beans need to be optimized for
    each server. If you implement a bean using one technique,
    it may perform poorly on another platform, but that doesn't
    mean that the platform is slower, it may simply mean that
    your testcase is poorly written for that server.

    You might want to exercise more care in publishing numbers
    as a benchmark. And if you haven't worked with the vendor
    to ensure that your benchmark is optimized for the vendor,
    you should be doubly careful!

    My engineers always work with customers if they have their
    own benchmarks testing products and 9 times out of 10 there
    is a bug in their test, or they haven't built a valid
    benchmark test case when performance is problematic...
    Making a reliable and 'valid' benchmark is an extremely
    difficult and subjective thing. Caution is always wise...
    The last time someone did this with BEA, I remember
    something about a lawsuit, you might want to call your
    attorney to be prepared... And next time, read the shrink
    wrap license before doing this sort of thing - always a
    wise move, even though most engineers rarely do it. It is
    your obligation, not the vendor's...

    Just my $.02

    Ward Mullins
    CTO
    THOUGHT Inc.
    http://www.thoughtinc.com
  31. Ward is dead on here. Check out the EULA.

    Dave Wolf
    Personified Technologies LLC
  32. Interesting, just a pity no one puts JBoss to the test, it could be quite interesting.

    For all those who want to compare .Net to J2EE, just remember that .Net is an 80/20 solution, even if it is ridiculously faster then J2EE its still only half as powerfull as J2EE, besides, as long as .Net remains a Windows only platform there can never be a fair comparison
  33. New performance kits posted[ Go to top ]

    I've expanded the list of Performance Kits available for free from PushToTest. The complete list is found at: http://www.pushtotest.com/ptt/kits/index.html. Enjoy. -Frank