Discussions

News: Tomcat Performance Higher on Linux

  1. Tomcat Performance Higher on Linux (24 messages)

    WebPerformance.com has published an in-depth comparison of Tomcat running on Windows and Linux, with a significant difference in performance in favor of Linux. Published as a two-part article, it shows differences in how the VMs handle various conditions. As a result, it's useful to examine even if you're not using Tomcat.

    The comparison is split into two parts:Another report is available as well, entitled "Comparing the Performance of J2EE Servers".

    The Tomcat report describes the very different behavior of the two platforms under load, and shows there is a significant different in performance. Under the restricted conditions of the test Linux was able to handle 32% more load than Windows with identical versions of Tomcat on identical hardware.

    From the first article:
    Here, we've noted the behavioral and performance differences for each server up to, during, and after a bottleneck. Different bottlenecks will of course have key indicators as to their cause. However, from the end-user's point of view, many bottlenecks will end up being dealt with in the same way. Some preliminary testing was performed afterwards by tweaking the VM memory settings in order to remedy our performance slump, and verify that it was in fact a memory limit. However, we do see indicators that the VM must have been forced to block for otherwise background optimizations, as the performance for both servers did eventually recover. In the next part of this report, we will find that tuning the JVM memory parameters resolve the performance slump that we saw in this round of testing, confirming that the problem hit by this test was a memory limitation. Despite the specific cause of our slump, it is not inconceivable that other performance pitfalls will end up being dealt with in the same fashion.

    Prior to reaching capacity, our Linux server showed it's ability to scale subtly better, notably with a higher completed response rate. This trend arises again after the servers have been able to recover from their performance slump.

    When our servlet found itself hitting memory limits of the app server, the platforms had an opportunity to reveal different error handling techniques. Linux maintained it's lead over it's Windows counterpart, except when it was forced to deal with the memory shortage. Users were potentially forced to wait minutes or more for their page to complete loading. Potential waits turned into repeated waits for users navigating through a long sequence of pages. Windows users saw a different story. Under the same memory shortage, the OS was forced to turn away traffic, but delivered roughly the same number of successful hits as our Linux server.
    The analysis of the second report:
    Here, we have given Tomcat the opportunity to compete when installed on two different platforms. We see that the average duration of each URL, even under extensive load, remains relatively instantaneous for both servers. However, our Linux server slowly has an increasing number of URLs that do not get serviced quickly, and delays service for longer periods of time than our Windows server, when forced under load. By contrast, our Windows server shows somewhat shorter durations per URL by instead turning away traffic that it is unable to handle. Despite this quick but incomplete response on some resources, many users will still find themselves waiting on another resource that was not serviced as promptly.

    From the administrator's stand point, our Linux server did not tie up it's CPU resources as early, and was able to service many more connections overall. However, some of those connections were consistently delayed, leaving users waiting on the other end. However, even with some normal delays in consideration, our Linux server was still able to serve more users than it's Windows counterpart.

    Threaded Messages (24)

  2. This is interesting. About 3 years ago I was part of a large scale health care enterprise app. We found through our benchmarking that we achieved the best performance by running JBoss with Tomcat on Windows and Oracle on Linux.

    We were using JDK 1.3 if IIRC.

    We got lots of connections refused from Tomcat on Linux. It's good to see that the performance gap has been closed.
  3. Tomcat Performance Higher on Linux[ Go to top ]

    Java on Linux improved a lot since JDK 1.3 / Linux 2.4, that's for sure.
  4. Java on Linux improved a lot since JDK 1.3 / Linux 2.4, that's for sure.

    Linux has evolved a lot too. I would imagine that the new threading model has something to do with it. Besides the Java threading on Linux was always icky before.
  5. It would be interesting to see results when # of users goes to 10,000 or 50,000 or even higher...

    Thanks for the tests!

    MaxJava
  6. odd-looking throughput graph[ Go to top ]

    Hey, Mike, long time. Anyhow I was showing that throughput graph (first graph in the first paper) to some of the guys here trying to figure out what was going on. Someone suggested it might be re-allocation of Tomcat-internal containers. I'm unconvinced because I'd have expected a stronger CPU spike in that case. But it struck me that the Y axis on the graph although labelled "users" is probably also effectively measuring elapsed time, as the test script probably added one user every N seconds. If the re-allocation scenario were correct then I'd expect the throughput to recover EVEN IF no further users were added after the one that knocked the server off its tightrope. Did they run any such tests?
  7. What these charts tell me is that the functional capacity of each system is about the same, but Windows will fail faster than Linux.

    What's interesting about that is that you can perhaps configure a load balancer to leverage that fast fail ability of Windows to redirect the request to another machine, and in the end perhaps serve the client faster than if the server hangs on the connection struggling to serve it.

    If you look at the first set of charts in the first article, around 800 users, they both start going in to a slump and then things get messy.

    If you then check the second article and it's bar charts, Windows has more capacity below the 4 second range than Linux but Linux will move on a wee bit more. Even tho Linux can handle, conceptually, 30% more users (looking at that bar chart), it's a question of whether those extra 30% percent are well served at all.

    It was interesting that the Windows JVM was running out of memory whereas the Linux one was not, but perhaps the socket implementation for Windows is simply more memory expensive.

    Either way, just like many of the servlet container benchmarks we've seen, while one platform may well beat the other at the finish line, either one is capable and suitable for the vast majority of applications.

    If I saw these benchmarks for my application, I'd be using the 800 user mark, where they both seem to be equal, and scale from that number, rather than from any "maximum" number the charts may show. No system is interesting to use when they're that hammered.

    Finally, it would be interesting to see this same benchmark using the new Jetty NIO or the Grizzly connecter for Tomcat from Glassfish, and see how the numbers change.
  8. How Many Users Calculations[ Go to top ]

    The comments about the performance of Windows/Linux are true with regards to the first test in which both virtual machines were memory limited. Once the memory limitation is removed, however, I do think that Linux shows a clear advantage. First, you can look at the I/O, in which the Linux side is handing quite a bit more throughput with the exact same hardware.

    The point about performance is a good one. The "how many users" calculations takes into account performance when estimating capacity, with default settings of that 95% of the users have a total page duration of 6 seconds or less, and with no more than 1% of the responses indicating errors. We could tighten up that performance criteria and see what happens, but our software does of course have to take this into account when estimating capacity.

    The point about either system being capable is of course true. In publishing this people using either system could use this information as a starting point for tuning or doing their own performance testing. The "headlines" approach to the result is only "marketing".
  9. From the 'response time' graphs it looks like latency is a lot better behaved on Windows. The fact that at only 800 users the latency on Linux is so erratic probably can be fixed by tuning some parameters.

    These are some (much more telling) benchmark that demonstrate just how well linux tcp scales these days with 5000 open connections (see the Conclusion):

    http://bulk.fefe.de/scalability/

    Windows in not there, but my guess is that some version of windows out there can do as well. If not, I really wonder what MS is waiting for.

    Guglielmo

    Consider using an Apache-Licensed Java(tm) implementation of the totem single-ring protocol to implementation your replication requirements.
  10. Other JVMs?[ Go to top ]

    While I understand the choice of one single JVM to keep the test matrix down, it would be interesting to see the same results using BEA JRockit and the IBM JVM. In our experience, the JVM often has more impact on performance than the operating system.

    Peter Lin ran some Tomcat benchmarks a while back which showed better performance and latency with JRockit than the Sun JVM. That was of course on that particular benchmark, and may have changed since since all products have evolved.

    Henrik, BEA
  11. Other JVMs?[ Go to top ]

    IBM too, plz
  12. Just to pay attention...[ Go to top ]

    When talking about performence of Tomcat and where it is better to run the server... just take a look at the benchmarks between .net and j2ee. Although I develop on J2ee, I realize all the good stuff is, for my opinion, in the .Net side. performence, time of development, support...
  13. Just to pay attention...[ Go to top ]

    When talking about performence of Tomcat and where it is better to run the server... just take a look at the benchmarks between .net and j2ee. Although I develop on J2ee, I realize all the good stuff is, for my opinion, in the .Net side. performence, time of development, support...

    I'm curious which benchmarks you are referring to. Other than the benchmarks paid for by Microsoft, they all seem to show .NET trailing Java.

    Peace,

    Cameron Purdy
    Tangosol Coherence: The Java Data Grid
  14. Benchmarks at TheServerSide.net[ Go to top ]

    You can find benchmarks on the TheServerSide.net website

    Netanel
  15. You can find benchmarks on the TheServerSide.net websiteNetanel
    Do you trust lame TSS benchmarks ?
  16. Benchmarks at TheServerSide.net[ Go to top ]

    You can find benchmarks on the TheServerSide.net website

    Yes, but like I said:
    I'm curious which benchmarks you are referring to. Other than the benchmarks paid for by Microsoft, they all seem to show .NET trailing Java.

    Peace,

    Cameron Purdy
    Tangosol Coherence: The Java Data Grid
  17. Some naive people.[ Go to top ]

    Yes, but like I said:
    I'm curious which benchmarks you are referring to. Other than the benchmarks paid for by Microsoft, they all seem to show .NET trailing Java.
    Peace,Cameron PurdyTangosol Coherence: The Java Data Grid

    They are naive. They do not understand English. :-)
  18. We recently launched a new website for a major Swedish government branch using our portal software and they did extensive performance testing as part of the acceptance tests. We reached about ~350.000 hits/hour/machine with response times < 1s. This was on a dual CPU 3.0Ghz Linux machine. When the site launched they did performance tests by using external test points around Europe that measured the response time, and they got a new record: 20ms/page (old one was 24ms/page). Which is nice.

    But the funny thing is that the old site used only Apache with static DreamWeaver pages. Which kinda shows that it is possible to make a dynamic content system that not only delivers better functionality than static pages, but also delivers better performance. Java aint that slow, I guess.

    The portal software is entirely based on Tomcat 5.0.
  19. Tomcat performance experience[ Go to top ]

    mhh... we had similar tests with a large univeristy portal and Resin + Windows. However our tests showed that these results can only be reached for in-memory results (more precisely, with an high cached/non-cached page rate).
    One interesting point are TCP / queue / threading settings. These settings are far more important then the OS and then the JDK: to further complicate the scenario we also found that full-load tests require settings that are non-optimal (at least...) for saturation-load/recovery testing.
    All in all I'd say that the published tests show that Linux starts in a better position then Windows, however I wouldn't extrapolate the possible results for a full-load test on the real environement.
    p.s.: we also had to change Windows TCP resigstry settings to correctly run the tests
  20. Tomcat performance experience[ Go to top ]

    mhh... we had similar tests with a large univeristy portal and Resin + Windows. However our tests showed that these results can only be reached for in-memory results (more precisely, with an high cached/non-cached page rate).
    This is my experience as well, which I think explains how a dynamic system could provide better performance than a static one: it is easier to break pages up into pieces, large parts of which can be heavily cached (navigation, header, footer), so that a page load of a non-cached page causes minimal disk loading.
  21. Tomcat performance experience[ Go to top ]

    This is my experience as well, which I think explains how a dynamic system could provide better performance than a static one: it is easier to break pages up into pieces, large parts of which can be heavily cached (navigation, header, footer), so that a page load of a non-cached page causes minimal disk loading.
     We went even further: we designed the CMS and the Portal to cohoperate, so that almost 100% can be cached (even page content) and thus during peak load no disk access at all happens. When something is published from the CMS we do actively invalidate pages and so changes are immediately visible.
     And here I surely agree with you that a structured (framework etc...) approach is surely needed, otherwise it would be almost impossible to grand the needed degree of coherence.
  22. Tomcat performance experience[ Go to top ]

    We went even further: we designed the CMS and the Portal to cohoperate, so that almost 100% can be cached (even page content) and thus during peak load no disk access at all happens. When something is published from the CMS we do actively invalidate pages and so changes are immediately visible.&nbsp;And here I surely agree with you that a structured (framework etc...) approach is surely needed, otherwise it would be almost impossible to grand the needed degree of coherence.

    We did that too, so we're on the same page (pun intended). We use SoftReferences as much as possible for content, in order to maximize usage of JVM memory. That helps a lot, and people with lots of memory can use it to the max. It is rather annoying, however, that a JVM cannot use all the memory that is available on a server. We once had a customer misunderstand the hardware specs, so they put 20Gb of RAM into it (when we really meant that 20Gb of disk would be good). If we could have used all that memory all content would have been completely cached at all times.
  23. Tomcat performance experience[ Go to top ]

    We once had a customer misunderstand the hardware specs, so they put 20Gb of RAM into it (when we really meant that 20Gb of disk would be good). If we could have used all that memory all content would have been completely cached at all times.

    Just us an NIO-buffer-based cache to chew up large amounts of memory without actually keeping the data on the heap; a 64-bit JVM is required for 20GB, though.

    Or run 10 JVMs on it, and cluster them together .. Coherence would love the 20GB ;-)

    Peace,

    Cameron Purdy
    Tangosol Coherence: The Java Data Grid
  24. Was X loaded on Linux or not?[ Go to top ]

    I am just wondering if the Linux PC had X windows running (System Level 5 startup?) or was it turned off (System level 3 startup?). I imagine if you used system level 3 on Linux you would save a whole lot of resources that the X windows system would otherwise be using which I imagine would further increase the performance on Linux. Turning off the windowing system is something you just couldn't do with a Window's server.
  25. Was X loaded on Linux or not?[ Go to top ]

    Whether it was loaded on Linux or not, huh? I think the resources wouldn't be free, but I'm not sure. I found this on Google hoping to find an answer to exactly that question.