Servlet Report: what would you like to see?

Home

News: Servlet Report: what would you like to see?

  1. In July 2004, Web Performance published "Servlet Performance Report: Comparing The Performance of J2EE Servers." This report covered page load time, throughput, error counts, and performance. We're now getting ready to do a new report, and are asking TheServerSide.com readers what you'd like to see.

    Our current ideas include:
    • Update our existing test to include the latest versions of the servers.
    • Do some version-to-version comparison of the most popular servers (e.g. Tomcat 3.x, 4.x, 5.x) to help those evaluating the cost/benefit of upgrades.
    • Compare Tomcat vs Apache vs IIS for serving static content
    • Compare Tomcat+Apache vs Tomcat+IIS on Windows
    • Compare Tomcat+
    Those are just a few ideas. Obviously, what we can do is limited by the time and resources we've been given - but we'd like to know what performance comparisons you would like to see?

    Threaded Messages (53)

  2. It would be interesting to see data comparing hardware platforms (Intel, AMD, Sparc, PPC), operating systems (Windows, Linux - different distributions, kernel versions, Solaris 9 & 10 etc) and JVM's (Sun Hotspot, IBM VM, BEA JRockit).

    Ambitious perhaps, but very interesting. Perhaps you can pick a few common combinations.
  3. Hardware, OS and JVM variants[ Go to top ]

    +1
  4. Tomcat vs Resin[ Go to top ]

    Tomcat 5.x vs Resin 3.x

    "Handwritten" Servlets vs container managed.
  5. It would be interesting to compare the performance when JSP taglibraries are involved, since it is well known that different engines handle pooling and re-use of tags differently. Resin 3.x vs Tomcat 5.x would be an interesting combination for this test.

    Cheers,

    Dick
  6. Resin uses FastJSTL[ Go to top ]

    It would be interesting to compare the performance when JSP taglibraries are involved, since it is well known that different engines handle pooling and re-use of tags differently. Resin 3.x vs Tomcat 5.x would be an interesting combination for this test.Cheers, Dick

    when JSTL first came out, Resin had their own fastJSTL. it was faster than normal RI JSTL back then. Don't know if it is still true. I helped do some performance tuning when JSTL1.0 was being released. to get an accurate measurement, you'd have to test plain JSTL on both containers.

    peter
  7. I'd like to see this too.

    I'd like to see a nice little Tomcat app running on the same machine hosted on Linux, Windows, and Solaris 10. But I'm more interested just for curiousity than anything else.

    The problem I see, for me, is that from the last benchmark that I saw, most of the containers seemed to work pretty well. And, specifically, Tomcat seems to work pretty well.

    A lot of issues go into container selection, and a 10% difference in performance is probably the lowest on my list of deciding factors.
  8. Tomcat with APR support...[ Go to top ]

    ... should be in the next 5.5 release. It's past internal testing and now on to internal and external benchmarking and tuning. It'll be optional and not the default choice, but will surely be an exciting option for some environments and users. This is still early: you can subscribe to tomcat-dev and follow our discussions, or wait for the release notes if you'd like to hear more details.
  9. Tomcat standalone on Production[ Go to top ]

    Talk to the Tomcat team before doing any new benchmarks. Specifically Remy Maucherat and Mladen Turk. They have written some JNI pluggins to Tomcat that allow it to scale 100% better than standalone Apache. I don't think they have announced it yet, but it should be out soon.

    I have been hearing about this for around 3-4 months that Tomcat alone is as good as Tomcat+Apache or maybe even better! Are there any concrete implementations of a tomcat only site on production with both static and dynamic content?? And what about security limitations of not having a webserver front facing tomcat??

    I have been trying to convince my team and my client to have a tomcat only environment instead of the tomcat+apache but without much proof that this actually works, none (me included) want to take the risk to be pioneers!!

    -Anoop
  10. It always depends...[ Go to top ]

    ... on your requirements, for performance, security, and other metrics.

    A couple of times, we've started wiki pages and/or mailing list surveys asking who uses tomcat-standalone in production, and while we have a limited set of public answers (http://wiki.apache.org/jakarta-tomcat/PoweredBy), I get 5-10 responses privately saying "we do, but can't publicly disclose it" for every 1 response willing to say it publicly.
  11. One more JVM[ Go to top ]

    We would appreciate eventual inclusion of our Excelsior JET JVM in the tests, or granting us access to tests source so that we could run them internally.

    Excelsior JET's distinctive feature is a native code precompiler, so setting it up to compile dynamic apps can be tricky. Here is how to do that for Tomcat and Resin:

    http://www.excelsior-usa.com/kb/000018.html
    http://www.excelsior-usa.com/kb/000020.html
  12. I personaly do not find use of performance of IIS or Apache.
    Maybe think Tomcat vs Resin vs Orion.

    I think Hessian vs XML-RPC vs Ajax vs new binary JaxB v2.
    Anotherwords performance of service. (no need to test Soap, we know how fast that is).

    There are legacy servelts we maintain that we know, but... a lot of new developemnt is services, and we don't know.

    .V
  13. ...I think Hessian vs XML-RPC vs Ajax vs new binary JaxB v2.Anotherwords performance of service. (no need to test Soap, we know how fast that is)....

    I think you have a good point here - the topic of application server performance has been beaten to death - this is all good press for the app server vendors, but generally architects and developers are usually stuck with an existing application server.
    Most performance-related issues occur at the boundaris of the application server - database access, cross-jvm calls, mainframe access. It would be interesting to see what effect various protocols have when crossing system boundaries with the effect on network, CPU, and memory included. For instance web services are all the rage these days, but I've seen many applicaitons which overuse them and end up suffering because of the need to parse bulky XML, when it might be easier and more efficient to stay a bit more native (i.e. RMI).

    Clay

    j2ee911.com
  14. Servlet Container: Tomcat 5.x, Jetty 5.x, Resin 3.x
    I don't care much about Orion since it is unfortunaly pretty stale, but would not mind seeing full blown app server like Trifork ( http://www.trifork.com/ ) or Pramati ( http://www.pramati.com/ ) assuming the license permit to do so.

    What I would be interested in seeing:
     - raw performances (up to 100KB data)
     - performance using their provided compression filter
     - https (ssl) performance
     - performance with a apache httpd 2.x front end
         - mod_proxy
         - mod_jk
     - with and w/o mod_deflate
     - load balanced using
         - mod_proxy
         - mod_jk

    99% of production use are done using sticky sessions but it would be also interesting to figure out performance for a small cluster w/ http session replication using tcp

    This may sound like going beyond the scope of the servlet but this is often the configuration in production.

    Now I realize that you have at least a 3-month work full time :)
  15. ++ :
    ====

    servletcontainers
    -----------------
    tomcat 4.x 5.x
    jetty 4.x 5.x

    non-servlet webservers
    ----------------------
    Simple! 2.4.x 2.5.x http://simpleweb.sourceforge.net/
    apache 1.x 2.x

    responses
    ---------
    content-type text/xml
    size 5 MB
  16. WebLogic????
  17. Weblogic[ Go to top ]

    Unfortunately, the license agreement for BEA Weblogic does not allow publication of benchmarks. Repeated requests for permission have been denied :( Maybe if enough of their customers complain...
  18. Weblogic[ Go to top ]

    Unfortunately, the license agreement ...

    You're welcome to benchmark with Coherence*Web for session clustering. It's decent with only two servers (about 2x the session throughput compared to the next fastest implementation that we've tested), but that difference grows to be quite substantial by the time you're up to only 4 or 8 servers, and we keep scaling linearly up to the width of the switch. (Chris - I'm sending you an email with my contact info as well.)

    Peace,

    Cameron Purdy
    Tangosol, Inc.
    Coherence: Cluster your POJOs!
  19. Weblogic[ Go to top ]

    <complaint>What a shame.</complaint>

    How about JBoss? We are considering moving to JBoss in the long run anyway.
  20. Weblogic[ Go to top ]

    How about JBoss? We are considering moving to JBoss in the long run anyway.

    JBoss uses Tomcat.

    Peace,

    Cameron Purdy
    Tangosol, Inc.
    Coherence: Cluster your POJOs!
  21. deployment and audit[ Go to top ]

    -- ease of deplyment, for example
    ---- drop jar and appl is up in 2 sec
    ---- drop jar and wait 3.5 minutes
    ---- load jar via admin interface and wait 20 min

    -- server startup time
    -- application startup time

    -- JMX and admin features, monitor, audit, hot
    modification of server parameters (for
    example - log level or connection pool wait time)

    -- tune time: used as is, changed couple of parameters,
    spent 2 days to adjust parameters just to start server...?

    -- completeness of server packaging: used as is, couple
    of libraries were downloaded, 55 libraries had to be added...?

    -- memory and CPU and network monitoring, idle
    server memory, + idle application memory, application
    under 10 TX/sec load, 100 TX/sec load...?

    -- failure mode: memory limit, CPU limit, network saturation...?

    -- failure mode: outofmemoryexception, timeout (in what
    module ?)

    -- success mode: beer, cognac, cofe...?

    Alex V.
  22. deployment and audit[ Go to top ]

    +1
    Performance comparison is somewhat useless since it depends too much on the application code.
    but dever productivity is the exiting part...
    Java is slow to code with, containers are slow to start, hot deploy never works.....

    bottom line : in big companies I never have the choice regarding production servers that are always a political choice; but i can choose my devpt server/ide, so there is a real case to have some kind of producticity benchmark.

    Laurent.
  23. Clustering[ Go to top ]

    Clustering.
  24. Please evaluate ATG Dynamo Application Server, DAS 6.3.0. I am very interested in seeing how this server performes.
  25. Evaluate ATG Dynamo DAS[ Go to top ]

    I've used this application server in development and production before and hated every minute of it. Why would you want to compare it against, surely there are very few places left running it. In fact, I heard that ATG themselves were going to drop it and pick a list of preferrec partners instead with IBM's Websphere as their preferred choice?
  26. Tomcat vs Websphere[ Go to top ]

    My company is currently replacing Tomcat by Websphere.
    We only produce servlet based applications (jsp/struts/hibernate), but the powers that be do not trust open source software for our servers.

    So I'd be quite interested to see an update on the perf. tests Tomcat 5.x vs Wesphere 5.1/6.0.

    Introducing Load balancing and clustering would bring the tests closer to our production envrionment.
  27. Evaluate ATG Dynamo DAS[ Go to top ]

    I've used this application server in development and production before and hated every minute of it. Why would you want to compare it against, surely there are very few places left running it. In fact, I heard that ATG themselves were going to drop it and pick a list of preferrec partners instead with IBM's Websphere as their preferred choice?

    I have used ATG on atleast 4 of my previous projects - a small number to go by I agree. Regardless, I have found ATG to be a robust application server - never experienced a problem / issue that was not of my own undoing. Performance wise I do not have much to comment upon but it will be a sad thought someday to know that ATG DAS is completely off the shelf.

    -Anoop
  28. Do you need help?[ Go to top ]

    There are lots of requirements, and lots of them are good.

    Can you do it all by yourself? Do you need help?

    Thank you,
    Edmon
  29. I'd like to see what the impact is of enabling useful tracing on each server when it is under some strain.

    We often see such bad performances that nobody ever dares enabling tracing again, which in theory is a very useful feature.
  30. Even in theory...[ Go to top ]

    ... tracing is only useful in test/debug environment, and in production only if there's such a severe error condition that you're willing to trade off significnat production performance to debug it. It's not supposed to be a performance-neutral tradeoff. Testing for it is mostly a waste of the Servlet Report team's time, and I say that without even being a member of that team ;)
  31. a comparision on the relative performance between different presentation frameworks including:
    - raw servlet (println)
    - JSP
    - Struts
    - JSF
    - Tapestry
    maybe others as well
  32. a comparision on the relative performance between different presentation frameworks including: -raw servlet (println) -JSP -Struts -JSF -Tapestry maybe others as well

    We've kicked around that idea in the past - but struggled with how to make the comparison fair. If you have something in mind, we'd love to hear it!

    It seems like we would need to define a reference application (or maybe use Java Pet Store?) and compare an implementation under each framework. Given that we have little expertise with any of those frameworks, it would require finding an "official expert" to represent each framework and implement the reference application. Are there some applications in existence that support several of these frameworks?
  33. We've kicked around that idea in the past - but struggled with how to make the comparison fair.
    I know that it ain't an easy job... we tried something also last year. But I thought the request for ideas did not specify "easy stuff only" ;-)
    If you have something in mind, we'd love to hear it!It seems like we would need to define a reference application (or maybe use Java Pet Store?) and compare an implementation under each framework. Given that we have little expertise with any of those frameworks, it would require finding an "official expert" to represent each framework and implement the reference application. Are there some applications in existence that support several of these frameworks?
    The wafer-project aims to compare the presentation frameworks and on its wafer2-page
    http://www.waferproject.org/wiki/Wiki.jsp?page=Wafer2 they list some frameworks on which they have implemented the "wafer weblog". Maybe that could be a starting point.

    Maybe each framework-team (the original framework-creators) should be asked to verify the validity of this implementation. And those frameworks where the sample from wafer is missing, might supply that to be included into the comparision...
  34. peformance series[ Go to top ]

    It would be interesting to see the same set of tests I ran on tomcat against Jetty, Resin and Weblogic. the jmeter test plans I used for the static file tests are located here http://people.apache.org/~woolfel/testplans.zip.

    peter lin
  35. I wan to see how different servers performs when Logging is switched On and Off.Performance comparision between simple Servlet/JSP and Servlet/JSP Strut Tiles implementation.
  36. WebLogic[ Go to top ]

    WebLogic is conspicuous by its absence. It should be included in the next version of the report.
  37. Include Oracle OC4J[ Go to top ]

    It would be nice if the test could included Oracle OC4J.
  38. At the risk of expanding the scope, please see if you can include Borland Enterprise Server.

    Also, (to second the thoughts expressed in an earlier response), let me know if you need help in this project.
  39. Unfortunately, the Borland license agreement also contains a no-benchmarking clause. If we perform another test of multiple servers, I will ask them for permission...but don't hold your breath.
  40. Chris:

    I might be able to help with you getting a benchmark-conducive license from Borland.

    Let me know if you're interested.
  41. Talk to the Tomcat Team[ Go to top ]

    Talk to the Tomcat team before doing any new benchmarks. Specifically Remy Maucherat and Mladen Turk. They have written some JNI pluggins to Tomcat that allow it to scale 100% better than standalone Apache. I don't think they have announced it yet, but it should be out soon.

    Bill
  42. Talk to the Tomcat Team[ Go to top ]

    I don't think they have announced it yet, but it should be out soon.Bill

    It is not announced because it is not tested yet. It seems to work best on Linux, which is good as it will provide much improved one-Linux-box web server efficiency (we'll see actual efficiency after testing; there is little benchmarking data) which is not a market addressed that well compared with native solutions.

    SSL is not implemented yet (will use OpenSSL in a similar way as mod_ssl), and neither is AJP (APRizing this connector will provide better support for certain specific use cases).

    (oh, no, I posted on TSS - Bill asked for it, and it's usually not a good idea to deny the Boss ;) -, apocalypse will probably soon follow; I might as well start blogging)
  43. Resistance is futile[ Go to top ]

    I had a good laugh :) I still haven't had a chance to run the APRized tomcat on my linux box. Been getting over a flu. As as I run those tests, I send them to you.

    peter
  44. Resistance is futile[ Go to top ]

    Actually, I noticed after posting I has posted a couple messages in the past, but I had forgotten.

    Thanks for the testing.
  45. I didn't see JMS container metrics (and rarely do), but often they can be the success or failure of a project. Consider things like:

    1. Throughput of queue and topic processing
    2. JMS selector performance (some do table scans - ick)
    3. Scalability of queue and topic processing of clustered JMS containers (scalability is a loaded term, but you get the gist)

    James
  46. Oh, and add Active MQ to the JMS containers for your test, as well as JBoss and Weblogic.
  47. Er, uh.. yeah.. guess I should read the full subject.. I saw J2EE and jumped to the whole stack.. But, should you expand to the full J2EE stack, feel free to consider JMS

    /me signing up for reading lessons
  48. Sun's Web and Application Server should be included - please let me know if you need help with licences (email address is obvious).

    Rich Sharples
    Sun Microsystems

    http://blogs.sun.com/roller/page/sharps
  49. Suns app server[ Go to top ]

    I was hoping that the Sun server would chime in, and its nice to see that it has.

    Rich's blog pointed to another new benchmark, and not surprisingly, it's not far off the mark of the other one (the numbers are dramatically different, but when you look at the configuration, they're very similar -- ~90+ JOPS/server). Say, Rich, was the DB server on the last benchmark over specified? You had 13 servers driving a 24 CPU box for that one, and here you have 3 servers driving a 2 CPU box, yet you get similar numbers.

    Ya know, their server is pretty good, and with the integration to NetBeans, it's even better. The documentation is pretty good, and the forum support seems decent as well. But it has no mindshare.

    Rich, I'd love to see you guys do that same benchmark against Postgres. Then you'd have a truly free solution to promote. While 90 JOPS is pretty good, I bet a lot of smaller companies don't even need that much throughput (assuming that the limiter here would be Postgres vs Oracle), and it would be interesting to see what it's like on a completely free platform.
  50. Suns app server[ Go to top ]

    I was hoping that the Sun server would chime in, and its nice to see that it has.Rich's blog pointed to another new benchmark, and not surprisingly, it's not far off the mark of the other one (the numbers are dramatically different, but when you look at the configuration, they're very similar -- ~90+ JOPS/server). Say, Rich, was the DB server on the last benchmark over specified? You had 13 servers driving a 24 CPU box for that one, and here you have 3 servers driving a 2 CPU box, yet you get similar numbers.

    For the PE benchmark the DB Server was a low-end dual Opteron machine, for the previous SE benchmark the DB server was a largish (12 CPU) Ultra SPARC IV machine - very different workload characteristics. These are fairly typical machines for the two different workloads.
    Ya know, their server is pretty good, and with the integration to NetBeans, it's even better. The documentation is pretty good, and the forum support seems decent as well. But it has no mindshare.

    Thanks.

    Well, we did a survey on the TSS a while back that showed our mindshare is good but our brand bifurcation has damaged our brand recognition.

    We get approximately 150k downloads a month of the Platform Edition (about 2.6 million since it was released) - many, many people are learning, developing, prototyping and deploying with Platform Edition.
    Rich, I'd love to see you guys do that same benchmark against Postgres. Then you'd have a truly free solution to promote. While 90 JOPS is pretty good, I bet a lot of smaller companies don't even need that much throughput (assuming that the limiter here would be Postgres vs Oracle), and it would be interesting to see what it's like on a completely free platform.

    Yes that WOULD be interesting wouldn't it.

    Rich Sharples
    Sun Microsystems
    http://blogs.sun.com/roller/page/sharps
  51. I would love to see a comparison between 32-bit and 64-bit servers, (OS, JDK and Tomcat versions).

    Also the effect of adding CPUs (multi-core?) and RAM to see where they top out in terms of performance. Maybe even RAID levels 0,1 vs non-RAID?
  52. 32bit vs 64bit, Hardware max[ Go to top ]

    I would love to see a comparison between 32-bit and 64-bit servers, (OS, JDK and Tomcat versions).Also the effect of adding CPUs (multi-core?) and RAM to see where they top out in terms of performance. Maybe even RAID levels 0,1 vs non-RAID?

    I really doubt RAID will boost performance. more than anything else, RAM and network IO seems to be the bottleneck today with most mature servlet containers. In my own tests with my AMD 2ghz linux box, 100mbit ethernet was the bottleneck. What would be interesting to me would be comparing Sun with 4 100mb ethernet trunked vs 1gb/2gb ethernet.

    peter
  53. The only benefit from 64bit hardware for most app is if the ability to cache that much more data from the DB requiring the need of a 64Bit heap over comes the 20-30% cache performance hit you take in basic execution.

    As soon as you hit 64Bit, half of your CPU cache is gone, since now you're working with 64Bit pointers. It matters.

    You may gain more peformance by clustering across two (or more) smaller heaps.
  54. Websphere 6.x[ Go to top ]

    I would be nice to see the Websphere 6.x reports in the study.
    Thanks