Web Performance Inc. has released a benchmark report that profiles the performance of popular low-cost servlet containers when operating under heavy load. The report is focused on projects utilizing a single, standalone application server. Tomcat, Orion, Jetty, Resin, Sun ONE 7.0, and IBM WebSphere 4.0 were tested.

See the report at:
http://webperformanceinc.com/library/ServletReport.

Feedback on the reports implications and future improvements are appreciated.

They did a test to determine when the server starts refusing connections - isn't this a simple initilization for each server? I believe it is in Tomcat. Silly.

I agree this is strange.

There is also a setting in resin that controls the max number of threads in thread pool for serving the requests:

<thread-max>150</thread-max>

Can the authors of the report confirm they did not just use the default values for the various servlet engines, and set each one to an arbitrary equal very high value before doing the test?

I imagine this to be true, since it would be such a basic mistake to make, but a confirmation would be nice.

Apologies to the authors for being too quick on the draw.

From the report:
<quote>
Install, Configure & Run the Server
Servers are all tested in their default configuration with these exceptions:

Set the JVM heap memory limits to 350M starting and maximum.
If there is a setting for maximum simultaneous connections, set this value to infinite if possible, or to the largest number it will accept.
</quote>

Unfortunately, BEA's product was not tested alongside the others.

< Unfortunately, BEA's product was not tested alongside the others.>
A NOTE FROM THE REPORT
The lowest-cost (e.g. single-server) versions of BEA Weblogic and Pramati Server were excluded because their license agreements prohibit the publication of performance benchmarks without permission

Go Orion!

So essentially the message I got was that Resin and Orion are overall the best servers while SunONE, although not as fast, is more reliable. Although I know the article was trying to avoid the who wins and loses question, it's pretty clear the rest don't compare performance wise as a whole according to those charts. Interesting.

I think I would have said Orion and Resin to come out on top too just from practical experience.

Resin fell from the top of the pack to the middle once the testers began requesting static resources in addition to servlets. One method of avoiding this performance hit is use Apache or IIS to server static resources and Resin to serve dynamic ones. My company has used the Apache-Resin combo for years for several high-volume sites and we have found it to be both performant and stable. This configuration is a little out of scope for the purposes of comparison, but it more accurately reflects how Resin would be deployed in the real world.

One thing that I didn't see mentioned - was Jetty run with or without the JDK1.4 specific (NIO) optimisations?

As stated in the report, each server was tested in it's default configuration, except for the noted changes. You'll have to ask the Jetty guys whether that would be using NIO APIs or not.

> default configuration, except for the noted changes.
> You'll have to ask the Jetty guys whether that would be
> using NIO APIs or not.

Yeah, I was a little quick off the mark and didn't quite register that part of the report before posting :)

As I understand it, NIO is not enabled by default. I would be very interested to see an standard/new IO API comparison, but maybe that is a subject for a different report.

I would also add that my experience of Orion mirrors the results of the report. It is extremely fast and lightweight, and for a long time has been my first choice for an appserver. However, it does refuse connections gracelessly when the load gets high. And its so far behind the spec these days its just not funny.

It seems to me the performance lines laying on top of eachother, until the highest loads, indicates that the performance was JVM and hardware bound. I would like you to please send this benchmark to Tom's Hardware (www.tomshardware.com), and ask them to please add the Tomcat run as a motherboad/processor benchmark. Tom's tends to focus on client-side benchmarks. I'd like to see how far modern hardware improves these metrics.

I would like to know if and how much the hard disk was being accessed during these runs. Is there any way to tell if the pages and resources were being held in RAM or the L2 cache (they certainly could have fit)? If so, would these benchmarks be indicative of the performance of dynamically generated pages from a large pool of pages and resources?

Did you Ctl-Alt-Del -> Task Manager -> Processes, during these benchmarks to see the CPU utilization and memory footprint of the server and the JVM java.exe? Was it ever more than the physical RAM and did you see if there was a significant rate of page faults (Options -> Add Column to see)? If so, I'd like to know what they were when the server started to faulter under load.

Did I miss something or was there a way to tell what the actual number of pages served and connections. I only saw time on the x-axis.

Very interesting. Thanks!

-Tom Schaefer

Feel free to point Tom's Hardware to our report...he (they?) can download any of the required files.

There did not appear to be any disk access by any of the servers at any point during the test.

Most of the servers were pegged at 100% CPU utilization at their peak, but not all. As stated in the report, each server (where possible) was allocated 350M of heap space to run in and the others were monitored for CPU usage during the test. The memory usage on the server computer never exceeded the physical memory available (e.g. no swapping).

No, the statistics for "number of pages served" was not in the report. However, the data is available in the datafiles produced by the report. Install our software and open the file...there is a lot more data there than was included in the report.

Nice to have, but why not JBoss instead?

[Nice to have, but why not JBoss instead?]

Perhaps because JBoss doesn't have its own servlet engine, but instead uses either Tomcat or Jetty, both of which were already represented? Websphere, on the other hand, has its servlet engine integrated with the rest of the container (like WebLogic).

Oh yeah, any company that won't allow 3rd-party benchmarks is lame. (Yes, this means BEA.)

[Nice to have, but why not JBoss instead?]

Perhaps because JBoss doesn't have its own servlet engine, but instead uses either Tomcat or Jetty, both of which were already represented? Websphere, on the other hand, has its servlet engine integrated with the rest of the container (like WebLogic).

Oh yeah, any company that won't allow 3rd-party benchmarks is lame. (Yes, this means BEA.)

[Nice to have, but why not JBoss instead?]

Perhaps because JBoss doesn't have its own servlet engine, but instead uses either Tomcat or Jetty, both of which were already represented? Websphere, on the other hand, has its servlet engine integrated with the rest of the container (like WebLogic).

Oh yeah, any company that won't allow 3rd-party benchmarks is lame. (Yes, this means BEA.)

[Nice to have, but why not JBoss instead?]

Perhaps because JBoss doesn't have its own servlet engine, but instead uses either Tomcat or Jetty, both of which were already represented? Websphere, on the other hand, has its servlet engine integrated with the rest of the container (like WebLogic).

Oh yeah, any company that won't allow 3rd-party benchmarks is lame. (Yes, this means BEA.)

[Nice to have, but why not JBoss instead?]

Perhaps because JBoss doesn't have its own servlet engine, but instead uses either Tomcat or Jetty, both of which were already represented? Websphere, on the other hand, has its servlet engine integrated with the rest of the container (like WebLogic).

Oh yeah, any company that won't allow 3rd-party benchmarks is lame. (Yes, this means BEA.)