Discussions

News: Microsoft Responds to Sun’s Web Service Benchmarks

  1. In a paper published last month, Sun claimed that Java based web services outperform .NET based web services both in throughput and response times. Microsoft has released a paper on TheServerSide.NET responding to those claims stating that Sun’s representation of the .NET performance was understated by 2 to 3 times and that in many, but not all cases, .NET exceeded the Java benchmarks.

    Microsoft Claims

    Microsoft claims that the tests were oversimplified with the actual SOAP messages not being as large as those they expect business based web services to use. With larger payloads, .NET takes even more of a lead in performance and throughput.

    They also point out that Sun’s tests were performed on a lightweight Apache web server as opposed to one of the commercial J2EE products such as WebSphere or WebLogic and therefore didn’t involve the same overhead that a web service implemented using EJBs would entail.

    Editors Note: Doesn't this just show us that using a lightweight system is the right thing to do? The comparison isn't .NET versus BEA after all.

    Lastly they assert that an adequate test of web services would also involve some amount of database interaction and they claim.NET outperforms in that category.

    Read the Microsoft response on TheServerSide.NET: Web Services Performance: Comparing J2EE and .NET

    Read Sun's original paper: J2EE claimed to have better Web Services performance than .NET

    Yet another benchmark war? Here we go again? Good points? What are your experiences?

    Threaded Messages (49)

  2. nice headline,[ Go to top ]

    "Comparing JavaTM 2 Enterprise Edition (J2EETM platform) and .NET Framework" when they where using J2SE 1.4.2 + Tomcat 5.0!

    They do not only avoid EJB nowadays but drops the whole Big Java Server Elephant. :)

    But anyhow it is good to know that they at last have learned that Tomcat is the fastest Java server. Better late than never!

    Welcome to the club.

    Regards
    Rolf Tollerud
  3. nice headline,[ Go to top ]

    By your estimation wasn't Java supposed to be dead by now? I'm sure we can look forward to your wisdom and insight for years to come.
  4. nice headline[ Go to top ]

    Java isn't dead at all. It is living a healthy life in it's forked state and is not designated for demise in the foreseeable future.
  5. benchmarks[ Go to top ]

    As much as people hate benchmark arguments, I like the fact that ones that are well documented include useful information in terms of recommended settings and setups.

    It also establishes some good baselines as to what to expect in terms of using web services, etc. In the case of the Sun pet store, it caused them to re-examine what they were saying was the "recommended" approach/design.


    Is it really any worse then arguing about EJB vs. hibernate, etc.?
  6. Sun soap opera[ Go to top ]

    Andy: "I like the fact that ones that are well documented include useful information in terms of recommended settings and setups."

    Well it was not Sun that provided the useful information. Microsoft reimplemented the Java-code too! Poor Sun, from now on they can not even get away with the "good-old-no-source-code" trick. :) What will they do next? They must be almost out of options.

    Regards
    Rolf Tollerud
  7. benchmarks[ Go to top ]

    Gr8 job done by Sun.Microsft is only trying to market its products.Doing copy/paste kind of things from java.As all know microsoft solution is always full of bugs .Windows is a type of of OS anti virus companies are living becasue of microsoft.
  8. I *LOVE* it, this is great for developers.
    Who is better C# of Java?
    COOL!!!!

    Do you want a $1. from me Sun or MS?
    Ha ha.
    .V
  9. After working on a project about 6 months ago migrating some .NET web services to J2EE on WebSphere, the services we migrated turned out to be about 10 times faster than the .NET implementation (using Stateless Session Beans and JPOX JDO).

    The main problem with .NET is that the VS tools (by default) generate code that uses .NET DataSets and diffgrams under the covers, which means the XML that gets shipped across the wire includes both before and after images of the data, and even worse the XML schema for the data is included in each and every request/response - so the marshalling and unmarshalling of XML is far less efficient than a WS-I compliant approach.

    So even though MS can hand-craft their code to be as fast as J2EE (i.e. by not using their own default codegen approach), this isn't the style of code that most .NET developers will end of creating
  10. 10-to-1 you didn't build it right[ Go to top ]

    The main problem with .NET is that the VS tools (by default) generate code that uses .NET DataSets and diffgrams under the covers, which means the XML that gets shipped across the wire includes both before and after images of the data, and even worse the XML schema for the data is included in each and every request/response - so the marshalling and unmarshalling of XML is far less efficient than a WS-I compliant approach.
    This is one way of doing it, but sending DataSets is not required if you are doing webservices in .NET. DataSets are not used "by default", either, so I don't know where that came from. And if you do use DataSets, it is possible to turn off the sending of the WXS as well.
    this isn't the style of code that most .NET developers will end of creating
    Sorry, I will not accomany you on the jump to that conclusion. There is nothing in the VS Wizard that defaults to sending or receiving a DataSet (including the diffgram) when you build a webservice. You choose the data type.

    DataSets are generally heavier abstractions, and have their uses, which is why BEA and IBM copied them in SDO (IMO). But DataSets are not requisite when you do webservices in .NET.

    If I have misunderstood, please correct me. But that's silly, I know y'all won't be shy.
  11. (Post Traumatic Stress Disorder is a psychiatric diagnosis for a situation
    common among people that make deliberate efforts to avoid thoughts or feelings about a traumatic event and to avoid activities or situations which may remind them of the event).

    As has been showed many times in this forum, Apache Tomcat (and Resin) is the fastest Java server, followed by "Commercial EJB Server without EJB", followed by the normal EJB Server as Weblogic or Wepshere which is by far the slowest.

    BTW, Tomcat is the most popular Java application server. According to BZ Research’s Study conducted in August 2002 52.2 percent of all respondents said Tomcat was currently in use at their companies, IBM’s WebSphere had only 29.0 percent, BEA’s WebLogic, 24.5 percent; and Oracle’s Oracle9iAS, 20.8 percent.

    And, with the rise of the popularity of lightweight frameworks since then, like Spring, Tapestry and Webwork, there is all reason to believe that Tomcat's share has risen to 60-70% something today.

    And the winner of the appserver marketshare war is...Tomcat!

    As for Websphere, (a collection of 300+ programs that is called a product) the best review IMO is here, Is it ethical to let companies just blindly walk into the gas chambers of websphere?.

    Regards
    Rolf Tollerud
  12. P.S.[ Go to top ]

    And BEA stock is now down to about 1/20 part of its all-time-high value, half the value for 6 months ago.
  13. P.S.[ Go to top ]

    And BEA stock is now down to about 1/20 part of its all-time-high value, half the value for 6 months ago.
    Yes, this kind of benchmarks are usefull for damage only.
    BTW I can prove my PC is better than mainfame, drop mainfames.
  14. I think this is my first post on theserverside ever...

    Just several points:

    1. "Xmx256m" on "2 x 3.06GHz w/ 2GB RAM" is really nice "tuning" :-)

    2. "...nor do they include the use of core features of
    J2EE such as Enterprise Java Beans."

    Can EJB be considerate a "core" feature of j2ee?
    What are we testing? - WEB Services Performance!!!

    I can imagine how can M$ "tune" entity beans...

    3. "Sun did not use a major commercial application server platform such as IBM WebSphere or BEA WebLogic in their testing, choosing instead to use Tomcat..."

    May I choose from some 20 .Net implementatios to get a slowest one? ;-)))

    SIncerely, D.
  15. Sam Jones,
    Why do you claim de-optimized? The response from MS was trying to point out
    that when performing more enterprise level operations like accessing a database, or operations that pass large amounts of data, iis/asp.net performs favorably with java/Apache. I don't think they de-tuned anything. Java/Apache is a very effective combination that, as the tests show excels at lighter weight operations. IIS/asp.net on the other hand is comparable on lighter weight operations, but clearly slower, but also provides more complex support that is more comparable to other application servers like WebSphere, and WebLogic, which in previous benchmarks is shown to be significantly slower then both Apache, and IIS. One thing to keep in mind, most of the types of operations the original Java bench mark performed would be more apropriate to be handle via a static html page, or a custom module in either iis, or apache. The other thing that surprising me is that MS provided everyone the source code and settings they used so everyone can make there own conclusions. Sun on the other hand publishes nothing but the results, and yet people in the Java community still claim MS cooks the books, and Sun is considered to be more trust worthy.
  16. P.S.[ Go to top ]

    And BEA stock is now down to about 1/20 part of its all-time-high value, half the value for 6 months ago.
    Yep. BEA is being sued by shareholders for alleged securities fraud. Among the many allegations (most of them regarding deception) made in the suit, one is quite damning:

    "that the Company's WebLogic 8.1 Platform was far from 'revolutionary'" [as originally suggested by BEA].

    -- http://biz.yahoo.com/prnews/040615/datu021_1.html
  17. The main problem with .NET is that the VS tools (by default) generate code that uses .NET DataSets and diffgrams under the covers, which means the XML that gets shipped across the wire includes both before and after images of the data, and even worse the XML schema for the data is included in each and every request/response - so the marshalling and unmarshalling of XML is far less efficient than a WS-I compliant approach.
    I infer from this that by default .NET is *not* WS-I compliant?!
  18. whose bigger?[ Go to top ]

    this type of vendor benchmarks do not go beyond the point that mine is bigger
    than yours seriousness(!)
  19. Benchmark Wars[ Go to top ]

    The best defense is to attack. So, "Well done!" to Sun.
  20. a) The performance of J2EE and .NET is similar enough not to matter much
    b) Benchmarks can be manipulated to show almost anything you like
  21. Some interesting comments on this forum I would like to respond to. When Sun published their benchmark numbers, we were taken aback by the very slow performance they showed for .NET, so we decided to investigate. They did not post any code (note in .NET benchmarks we do we do we post code so people can review and replicate if they desire), even though we called and asked them for the code so we could investigate. Nevertheless, if you read Sun's document they actually describe in detail the functional specification for their tests, the hardware/software platform, and their tuning applied to .NET and Java. So we re-created their benchmark, and got startling different results for .NET using just simple code and pretty much out-of-the-box tuning on .NET. In fact, our results were 2-3 times faster than Sun got for .NET, and we wanted to correct Sun's report since it is wrong. If you think any tricky wizards or tuning were used to get our results, check out the code, we published all the code so instead of just flaming or accusing, developers can actually see exactly what was tested and replicate/comment on the results.

    We are not sure how Sun got the results they got, probably not intentionally, but somehow they did. Note our Java results closely match Sun's results...its just their .NET results that were off.

    Some responses to comments that have been posted:

    1) Comment from forum "Why was Tomcat tuned as "Xmx256m" on "2 x 3.06GHz w/ 2GB RAM"? This matches how Sun tuned Tomcat, on a similar machine with 2GB RAM. So this was the appropriate tuning setting, since we had to precisely match what Sun did in their Java tests to replicate their tests. By the way, the Web Services tested are so simple, changing this setting to a higher number or running multiple instances of Tomcat would not yield better results---but you can verify this by downloading the code and actually running it.

    2) Comment from Forum: "VS and .NET have wizards that generate heavy-weight DataSet code by default when creating Web services and while we may have hand-coded for this benchmark, in general VS-generated .NET Web Services are 10 times slower than WebSphere". This is simply not true. VS does nothing to generate heavy-weight code for Web Services (of course you can use notepad and straight C# code if you want!), it simply generates stock WebMethods and you fill in the implementation code. Datasets are NOT used by default, you would have to choose to use them, and they are handy from programming standpoint. However, developer is free to not use datasets as well, its totally a choice. To somehow say MS created 'hand-crafted' code for this benchmark that is out of ordinary or would not be what you got if using Visual Studio properly is simply wrong. Download the code, open it in VS, check it out, and you will see for yourself. Creating Web Services in VS is simple, straightforward, and does not 'generate' overhead code for the implementation. As for Websphere being 10 times faster, well, I have not seen a single benchmark from IBM or other that supports this, but guess what? Anyone can take the code we published and (with some changes to run it in Websphere) test it in Websphere to find out!

    At any rate, the purpose of this is not to start a war, but rather to correct results that Sun published becuase they were in error. And it's not all about performance, or .NET vs. J2EE, a single choice. The two technologies are both important, will be used side by side, and need to work together. That's why Web services are so important. But if someone does a benchmark, they should at least publish their code and test harness with their results.

    Greg Leake
    Microsoft Corporation
  22. Tomcat Java appserver is free

    IDE Eclipse/NetBeans is free

    CVS code repository is free

    The whole application runs on Linux which is free.

    Portability to any OS 100%.


    .Net app server cost 30K per cpu. Total 120K for 4 cpu server.

    Visual Studio another 5k

    VSS another 5k for code repistory.

    Portability is a canoe
  23. We were discussing whether web services with Java are notably faster than with .NET. Microsoft, the closed company, is offering a lot of source code to confirm or deny this, while Sun, the open company, has offered none so far. As for you, you decided to discuss pricing... So much for keeping a technical discussions.

    Alas, I'm not a price expert but I think your .NET app server price tag is overboard, your VS 2003 certainly is, and soon you will be able to create this test using Visual Web developer for far, far less, so even though your price comparison is right in principle, the differences are not as high as you try to show. Could we not get back to the technical track?
  24. Edgar is right .. it's worth answering the technical questions. Has anyone reproduced Sun's results for .NET? Microsoft's results? Has anyone tried to build it from scratch using the out-of-the-box methodologies that VS provides to see if they are as inefficient as claimed?

    It is a little bit humorous that we (as a technology industry) produce a stupidly inefficient data standard (XML) then argue about who is "faster" at implementing a standard that (by definition) is slow. However, if we're going to argue about it, let's at least get the facts straight ..

    Maybe it's time for TSS to do another bake-off ;-)

    Peace,

    Cameron Purdy
    Tangosol, Inc.
    Coherence: Clustered JCache for Grid Computing!
  25. Cameron,

    The point is not a competition between Java and .NET, a question whether Java web services is faster or slower than .NET web services. The issue is that Sun had to use Apache Tomcat to approach .NET performance.

    It is an implicit declaration: "A final give up and admission that the big Java EJB Servers is out of the picture".

    How can there be differences between Java Tomcat and .NET when the technologies are so similar? Results from Weblogic and Websphere however, or even from Sun's own EJB Server (whatever its name is this week), will never be published, the results would be too embarrassing. The lightweight server, be Java or .NET outscores Weblogic in all areas, performance, scalability, productivity and maintainability. And reliability.

    So the point is that this benchmark from Sun puts the final nail in the coffin for the elephant servers, they are legacy technologies.

    Regards
    Rolf Tollerud
  26. The issue is that Sun had to use Apache Tomcat to approach .NET performance. It is an implicit declaration: "A final give up and admission that the big Java EJB Servers is out of the picture"
    Rolf, I think the choice to use Tomcat was based more on the fact that it is widely representative of server-side Java - it is widely used itself as well as being a foundation for many of the commercial (and OSS) servers. My *guess* is that many of the commercial servers could've been used to yield better results - I've certainly seen benchmarks showing just that and I'm not aware of any public benchmarks demonstrating Tomcat's performance domination over other commercial web containers - but I'm happy to be enlightened.

    To re-iterate some of the other posts - having a choice in this case is a dilema enjoyed only by the Java world.

    Rich Sharples
    Sun Microsystems
  27. Rich,

    Well now that we know that Java perform about similar to .NET as long as you don't overarchitect the solution I am sure that many would appreciate benchmarks of all the 3 major app servers, Weblogic, Websphere and JBoss, against lightweight solutions from Java and .NET. Both performance and scalability tests.

    That could be accomplished if Sun Open Sourced the SPECj Application Server 2004 benchmark specification, or better still created a whole new test. I am sure that Standard Performance Evaluation Corp (SPEC) would love to get the job.

    On the other hand as long as you keep the tests "Java only" we will never know, do we? You must forgive me if I believe that Sun keep the test proprietary because they are afraid for the results.

    Otherwise why not open?

    Regards
    Rolf Tollerud
  28. Rolf,
    The issue is that Sun had to use Apache Tomcat to approach .NET performance.
    Right. And they used a Yugo to show how fast a car could go ;-)

    Tomcat has definitely improved in its performance, but in load tests it used to be over 30x slower than WebLogic (for example) running the same code, so it had quite a ways to go.

    If Sun wanted performance, they would have used Resin or WebLogic Express (note: both less $$ than a Windows server.)
    It is an implicit declaration: "A final give up and admission that the big Java EJB Servers is out of the picture".
    I am not sure if you noticed, but WebLogic and WebSphere are also web containers.
    How can there be differences between Java Tomcat and .NET when the technologies are so similar?
    Tomcat and .NET? Similar? Honestly, Rolf, you have no idea what you are talking about. Even I don't insult .NET like that ;-).
    Results from Weblogic and Websphere however, or even from Sun's own EJB Server (whatever its name is this week)..
    Finally, something I agree with.
    The lightweight server, be Java or .NET outscores Weblogic in all areas, performance, scalability, productivity and maintainability. And reliability.
    Once again, you came to that conclusion how? All I saw was that Microsoft re-ran the tests and showed that their way of using .NET was much faster than Sun's way of using .NET.

    Can you show me where it said that WebLogic is unreliable? Please, show me that. Just this once, don't change the topic. Just this once, answer the question.

    After all, you are easygoing, open-minded, intellectual, educated, tolerant in the style of Voltaire, Diderot, Jonathan Swift, Benjamin Franklin, Mark Twain and so on, in short a representant (sic) for an enlightened age.

    Peace,

    Cameron Purdy
    Tangosol, Inc.
    Coherence: Clustered JCache for Grid Computing!
  29. Ah, yes, Resin rocks[ Go to top ]

    If Sun wanted performance, they would have used Resin or WebLogic Express
    I don't know about Weblogic Express but Resin is really fast, a breeze to install and a pleasure to use. It's funny how so many people in the Java community don't use it just because it costs a few hundred bucks. The fact that Resin is J2EE standards compliant should be enough, yet many people seems to be so blindly enamoured of the "totally free or nothing" concept that they won't take even the obvious ROI of a little jewel like Resin.
  30. away with old superstition[ Go to top ]

    Cameron:
    "Tomcat has definitely improved in its performance, but in load tests it used to be over 30x slower than WebLogic (for example) running the same code, so it had quite a ways to go.
    1) Apache Tomcat perform about par with .NET. (more or less)
    2) Petstore benchmarks show .NET far ahead Weblogic
    3) But Tomcat is 30x slower than WebLogic?

    But logic has never been your strong point, has it?

    Cameron:
    "If Sun wanted performance, they would have used Resin or WebLogic Express (note: both less $$ than a Windows server.)"
    "If Sun wanted performance", my god Cameron, that is the only thing they was trying to do! The point was "the big Java EJB Servers is out of the picture". Resin or WebLogic Express does not belong in that category.

    Cameron:
    "Can you show me where it said that WebLogic is unreliable? Please, show me that. Just this once, don't change the topic. Just this once, answer the question."
    Very very easy, unless performance and scalability, you see, it is not difficult to measure reliability. Just put a watch on the site (I use Alertsite, you can choose from a dozen). I have done that with TSS for two weeks now. The first week it was up only 93.55%, the second week 98.31 together 96.11 for the 2 weeks, a medium downtime of six and a half hours per week.

    That is still better than the most!
    The survey by Wily Technology Inc showed that half of the organizations in the survey experienced less than 96 percent availability for their J2EE applications, which is nearly seven hours of down time per week. And the worst availability for applications indicated was 81 percent downtime, or 4.5 hours a day.
    http://www.eweek.com/article2/0,1759,1388603,00.asp

    All EJB servers are unreliable. It is a EJB thing! If you argue that TSS don't use Weblogic anymore, (Bea withdrew their participation), and that Weblogic is different from other Elephant servers, then show me the site and I will report!

    Voltaire, Diderot, Jonathan Swift, etc

    I am represent for an enlightened age yes, chasing away superstitious belief in "Java EJB Semi-scientific Theoreticians". I you read more Cameron, you would not have fallen for that crap.

    Regards
    Rolf Tollerud
  31. P.S.[ Go to top ]

    I would advice you to stop being a champion of lost causes and rally behind the Spring, Tapestry, Webwork camp. Otherwise the bad reputation from EJB will spill over and destroy Java last stand.

    Every week, almost every day there are now new reports about .NET traction and Netcraft has reported that the number of ASP.NET sites now surpasses the number of Java sites.

    Jul 14, 2004
    Is .Net Stealing Java's Thunder?
    .Net is fast becoming the developer's platform of choice
    http://www.webservicespipeline.com/23900832

    Jun 08, 2004
    Microsoft Dominates In Web Services Space
    Microsoft Windows operating systems rule both as Web services development workstation platforms, and as target application platforms
    http://www.crn.com/sections/breakingnews/breakingnews.jhtml?articleId=21402208

    May 6, 2002
    Microsoft's C# gaining ground
    Microsoft's new C# programming language is gaining in popularity, with usage nearly doubling in the last six months, a new study shows.
    http://zdnet.com.com/2100-1104-899493.html

    And stop making personal attacks. I know that you are trying to lure me into saying something like "Cameron is a dump and uncultivated American that doesn't even speak a second language", to get me banned.

    But I will not say it!

    Regards
    Rolf Tollerud
  32. away with old superstition[ Go to top ]

    1) Apache Tomcat perform about par with .NET. (more or less)
    2) Petstore benchmarks show .NET far ahead Weblogic
    3) But Tomcat is 30x slower than WebLogic?

    But logic has never been your strong point, has it?
    Suffixed by your feeble attempt at yet another ad hominem attack, there are several significant issues with your line of thinking:

    1) Apache Tomcat and .NET are apples and oranges, kind of like comparing iPlanet to Visual Basic.

    2) The PetStore benchmarks showed WebLogic (which was not named) ahead of .NET using EJBs. (Again, I'm not sure if this is a fair comparison for a number of minor reasons, but it is what it is.)

    3) Tomcat _was_ 30x slower than WebLogic in tests that I personally performed to determine the overhead per request for various web/app servers and the ability to handle extreme load. How many tests have you performed comparing the load / performance attributes of WebLogic against Tomcat? (As an aside, in English, the word "was" is used to indicate past tense, and "is" is used to indicate present tense.)
    "If Sun wanted performance", my god Cameron, that is the only thing they was trying to do! The point was "the big Java EJB Servers is out of the picture". Resin or WebLogic Express does not belong in that category.
    Download Resin and Tomcat. Neither has EJB support. Tell me which is bigger. (They are both available for download. You can actually do what I have just described. Please don't make a fool out of yourself again by downloading the "enterprise" version of Resin, which does add EJB support.)
    "Can you show me where it said that WebLogic is unreliable? Please, show me that. Just this once, don't change the topic. Just this once, answer the question."
    Very very easy, unless performance and scalability, you see, it is not difficult to measure reliability. Just put a watch on the site (I use Alertsite, you can choose from a dozen). I have done that with TSS for two weeks now. The first week it was up only 93.55%, the second week 98.31 together 96.11 for the 2 weeks, a medium downtime of six and a half hours per week.
    And you are telling me that you traced the outages to WebLogic? Even though there are three or four app servers being used, an Apache sprayer in the front, and a database in the back? Not to mention various network devices and connections?

    I'm just curious, since some of our customers have been up for over a year on WebLogic (also some on WebSphere, Resin, SunOne) without a single second of downtime.

    I'm starting to think that you are blindly attacking things that you have absolutely no comprehension of.
    If you argue that TSS don't use Weblogic anymore, (Bea withdrew their participation) ..
    Oh, that's very rich. So you are saying that TSS doesn't even use WebLogic, but that you just trashed that product in public using TSS uptime stats as your proof? Diderot would be proud of this son of his, eh?

    Rolf, you are beneath contempt. I pity you.

    Peace,

    Cameron Purdy
    Tangosol, Inc.
    Coherence: Clustered JCache for Grid Computing!
  33. away with old superstition[ Go to top ]

    My stance is that all EJB Servers are unreliable, caused by underlying flaws in EJB basic design combined with over-architecture.

    You seem to have a different opinion,
    but it is a little unclear what do you mean,

    1) Are all EJB Servers reliable?
    2) Only commercial EJB servers are reliable?
    3) Only the 3 top EJB Servers are reliable?
    4) Only Weblogic reliable?

    Please specify.

    The fact remains:
    When Sun is extremely anxious to show that J2EE can be fast they use the Apache Tomcat server which by now is as popular for Java as it’s cousin the Apache server is for the Web.


    That is an éloge to the professionals at Jakarta foundation. I have heard that
    Jakarta is heavenly influenced and financed by IMB? Is that true?

    I wouldn't be surprised.

    Regards from
    Rolf Tollerud
    (the Dr. Semmelweis of the IT industry)

    to
    Cameron Purdy
    (High-Priest of the quasi-scientific EJB hierarchy)
  34. P.S.[ Go to top ]

    Cameron,
    Apache Tomcat and .NET are apples and oranges, kind of like comparing iPlanet to Visual Basic.
    Is that so?

    ASP.NET has its own build-in web server called HTTP
    runtime, which processes HTTP requests, controls caching, provides state management and support for Web Forms and Controls, essentially all the things that Tomcat does for Java.

    So when you use JSF (a webforms copy) with Tomcat the frameworks are almost comically alike, down to the underlying languages, Java and C# (a Java copy).

    Such is the results of good old capitalistic competition. You are not allowed to keep anything for yourself and "you have not entered this world for your own high pleasure."

    There is just not room for any fluff as the stock price of Weblogic clearly is showing, diving for junk-bond status.

    Regards
    Rolf Tollerud
  35. It wouldn't really have mattered whether we used Tomcat or any other appserver for
    these tests. Tomcat was used as it is packaged with JWSDP. Note that Tomcat
    is used only as the webserver to handle the http request and since the number
    of clients is small, performance isn't an issue. The actual JAX-RPC
    request is handled by JWSDP.

    Shanti
  36. It is a little bit humorous that we (as a technology industry) produce a stupidly inefficient data standard (XML) then argue about who is "faster" at implementing a standard that (by definition) is slow.
    Why do you presume efficiency was the driving force behind the creation of XML?

    P.S: BTW, why do you waste your time & energy responding to Rolf? You know very well he has no idea what he is talking about (about 100% of the time that is).
  37. BTW, why do you waste your time & energy responding to Rolf? You know very well he has no idea what he is talking about (about 100% of the time that is).
    Some of us roll boulders up hills in vain hope.

    Others reply to Rolf.

    ;-)

    (I'll take a look at the article that you linked to. Thanks.)

    Peace,

    Cameron Purdy
    Tangosol, Inc.
    Coherence: Clustered JCache for Grid Computing!
  38. the end of the EJB nonsense[ Go to top ]

    This finish this year’s performance cheat from Sun and the thread go to the archives together with the Petstore benchmark. Once again Sun has made a fool of themselves caught "in flagranti". And as usual Java zealots try to pretend it is raining.

    It shall be interesting to see what they will try next year, assuming they still exist then. :)

    EJB will not, except as legacy technology, "rolling boulders up hills in vain hope" is a perfect metaphor.

    Regards
    Rolf Tollerud
  39. the end of the EJB nonsense[ Go to top ]

    This finish this year’s performance cheat from Sun and the thread go to the archives together with the Petstore benchmark. Once again Sun has made a fool of themselves caught "in flagranti". And as usual Java zealots try to pretend it is raining. It shall be interesting to see what they will try next year, assuming they still exist then. :) EJB will not, except as legacy technology, "rolling boulders up hills in vain hope" is a perfect metaphor.RegardsRolf Tollerud
    Also,

    7) quote people out of context.
  40. it is entirely your decision[ Go to top ]

    My person is irrelevant. Mendacious accounts of the Elephant EJB servers performance and scalability have flourished for years now, hiding the biggest fiasco in IT industry history ever. That is thousands times more important.

    An inferior technology, High-Priest with a disgusting language and followers that shall we say, you wouldn't want as neighbors. A nice collection.

    On contrast, the new people behind Spring and similar frameworks is an entirely different lot. They are even making a port to C#! That alone shows the giant gulf between the old and the new in Java land.

    The choice is up to you.

    Regards
    Rolf Tollerud
  41. My experience[ Go to top ]

    I run a linux/websphere internet datacenter and another .net extranet.
    Both platforms have their own pluses and minuses.
    My lowly experience with both platforms tells me if i can get the job done on our .net extranet, it is always a win for our company. Java is just not that easy to develop. All our webservices are run on .net and deliver better than expected.

    By the way our extranet is not just pure .net, but is consumed by partners running Java and .net, and integrates into an a backend sysplex IBM/zOS / CICS/MQSeries machine. BizTalk with .net web services has always delivered for us.

    A major bank that i know of has investigated both platforms for a similar environement that our company have , test ran both platforms for the bank's IVR based transaction processing system [.net/BizTalk vs. WebSphere] and opted for .net solution which is now online for more than a year. The reason given by the banks CIO: " We requested the bidding companies to deliver less than 30 millisecond response time per transaction and only .net achieved better ! 23 milliseconds. IBM WebSphere was unacceptable to us for this project'"
    BTW the bank's backends are all IBM incarnations....
  42. Wrong again, or what fiasco?[ Go to top ]

    From what I have read, I'm not sure it is really worth replying, but I simply cannot let your posting stand like this.

    I have been involved in various successful J2EE projects in the past years. These projects included the usage of the EJB container, most used entity EJBs for various purposes.

    The applications developed in these projects perform well, because the features provided by EJBs were not an overhead but really required. If not provided by J2EE/EJBs, distributed transaction support, method level security etc. would have been programmed "manually" which maybe would have made the code a bit faster but certainly more expensive and less maintainable.

    The main reason for the success of these projects is, however, good engineering. All projects have been designed by people with expert knowledge of middleware in general and the components provided by J2EE in particular. The system designers were experienced and not some newbies who had read one of those J2EE books (of which 99% are pure rubbish).

    A particular technology is never sufficient for a project's success. It is just one of several preconditions. Much more important are the people envolved. Which, by the way, easily explains why some people more frequently than others experience projects ending in a fiasco.

     - Michael
  43. It is a little bit humorous that we (as a technology industry) produce a stupidly inefficient data standard (XML) then argue about who is "faster" at implementing a standard that (by definition) is slow.
    Since I respect your technical prowess a lot I request you have a close look at this article:
    http://msdn.microsoft.com/library/default.asp?URL=/library/en-us/dnxml/html/xmlmanifesto.asp
    You may have already seen it for all I know. _After_ reading it you can come back and argue about efficiency.
  44. Jamie,

    You commented in your post that ".NET App Server costs 30K per CPU and total of 120K for 4 CPU server; also that Visual Studio costs 5K, and VSS 5K.

    Actually, you may (or may not) be surprised that in fact these prices are not correct. First of all, anyone getting a copy of Windows Server 2003 has the full .NET application server without the need to buy anything extra, since .NET and the .NET framework is fully integrated into Windows Server 2003. So in reality, for deploying a Web application that uses its own authentication mechanism (anonymous access or anonymous access with forms-based authentication with a registered user database), the retail (not volume licensing) cost for Windows Server 2003 Std Edition, which supports up to 4 CPUs, is only $1,199.00, not the $120,000.00 figure you quote. Running on an 8CPU machine would require Windows Server 2003 Enterprise Edition with retail price of $3,999.00. Contrast this to what it would cost to license a fully supported version of Redhat Linux Enterprise Edition. Also consider that the IBM WebSphere and BEA WebLogic enterprise versions are licensed on a per-CPU basis at around $10,000 to $12,000 per CPU, which is charged on top of whatever the user must pay for the OS (perhaps free if unsupported Linux but certainly not free if supported version). Also, Visual Studio Professional is $1,079 US or an upgrade from previous version for $549 US, not the $5K you quote. There are different versions of Visual Studio, some of which include Visual SourceSafe, and the descriptions and pricing can be found at http://msdn.microsoft.com/vstudio/howtobuy/choosing.aspx.

    Also, Webmatrix is a free tool for building ASP.NET Web apps, and can downloaded for free (1.3 MB download) at http://www.asp.net/webmatrix/default.aspx.

    Bottom line is once you have the Windows XP or Windows Server 2003 OS, hosting .NET apps (web apps or windows apps) is free; the .NET 1.1 framework can also be downloaded for free for use on XP or Windows 2000 systems that do not have it pre-installed (although most new XP machines now have it pre-installed).

    Greg Leake
    Microsoft Corporation
  45. Scale on graphs in document[ Go to top ]

    Interesting, look at the scales on the graphs in the document. For echoVoid, the original graph shows a y-axis scale from 0-3000. The revised graph has a y-axis scale from 0-300. Which one "appears" like it has a bigger difference?

    I think someone should re-do this document and compare the values on the SAME graph.
  46. Ignore parent....

    Should have looked closer at the graphs.
  47. yawn, boring... Another attempt to compare apples to oranges. Don't we have better things to do than bash each others products?

    Greg.
  48. I was looking through client code and found a significant flaw in the calculation of throughput. In the code the "callCount" and "stdyState" are declared as long. The "throughput" is declared as double but the code which calculates "throughput" does not do explicit type casting to double before doing long/long so ends up with a long converted to double loosing decimal places. The document states that "stdyState" is set at 300 and in the case of "echoList listsize 200" the "throughput" is about 130. Number of Agents for this case is 16 (8 Agents x 2 client machines). So calculating backwards

    Agent throughput = 130/16 = 8/sec

    "callCount" for this agent during "stdyState" is 8 x 300 = 2400

    You will get the result 8 for any value of "callCount" from 2400 to 2699 because of the long/long division. This means the variation is more than 12% for just one Agent and when considering all 16 agents, the variation can become so high that result may become invalid.
  49. Ramesh,

    Thanks for actually reviewing the code and pointing out the error! You are correct the driver should convert callCount to a double before doing the division. We will be posting updated code with the fix today.

    The correct driver calculation should be:

    throughput = ((double)callCount)/stdyState;

    However, the impact of the error is actually not as significant as your post states, even in the worst case. The possible error percentage is a function of throughput, with the lower throughput cases impacted more than higher throughput cases. Also, the impact applies equally to .NET and J2EE since the same driver was used for each. See the end of this post for the actual differences we found when re-running the tests with the fix to the calculation.

    The original tests and our re-tests were both run with 8 agents on two different machines (16 total agents). At 8 requests/sec per agent the error will impact results between 0% and 12.5%. At 50 requests/s per agent the error impacts results only between 0% and 2%. (50 requests/s per agent equates to a total throughput of 800 request/s, some tests generated a throughput of over 2000 requests/s and for these the impact is even lower). Specifically:

     1.) In the worst-case example that you give if the callCount is between 2400 and 2699 then the error will be between 0% and 12.5%. However you can't sum the error across all agents, you need to take the average, so the error will still be between 0% and 12.5% regardless of the number of agents.
     
    2.) This is the worst case scenario, since in the other tests the callcount is much higher. If you take the echoList size 20 test, for example, the error is between 0 and 2%. The size of the possible impact is inversely proportional to total throughput for the specific test.

     3.) Finally and most importantly, the same client was used for both the Java and .NET test; hence the same error was applied to both .NET and J2EE.

    We ran the old and modified driver and the results matched the conclusions above for both .NET and J2EE. Both .NET and J2EE produced slightly higher throughput with the corrected calculation:

    -For EchoList using a list size of 200, the increase in throughput was ~6% for both .NET and J2EE.

    -For EchoList using a list size of 20, the increase in throughput was < 2% for both .NET and J2EE.

    Anyone can apply the fix to the code and rerun the tests to verify our conclusions. Even though the impact in our tests was not substantial, however, the bug needs to be fixed becuase the impact could become more substantial if customers run the code with larger list sizes than we did (200 was largest list size/lowest throughput case for our tests). So we will post an update today.

    This is a great catch, and precisely why its so important to post code and go with full disclosure when a benchmark is performed and results published.

    Thanks again for reviewing the code!

    Greg Leake
    Microsoft Corporation
  50. This is a great catch, and precisely why its so important to post code and go with full disclosure when a benchmark is performed and results published.
    I smell a Microsoft conspiracy ..

    p.s. ;-)

    Peace,

    Cameron Purdy
    Tangosol, Inc.
    Coherence: Clustered JCache for Grid Computing!