While IBM has been the only vendor submitting SPECjAppServer2004 benchmarks, Oracle+Dell and BEA+HP/BEA+Sun have been one upping each other on the older SPECjAppServer2002 benchmark. The benchmark is specifically designed to test J2EE (as of EJB 2.0) in a standard platform for development of portable, scalable, multi-tier enterprise applications.

Most recently submissions from HP and Sun running Weblogic 8.1 both matched Oracle's scores and in once case doubled Oracle's performance but at 7 times the cost.

The new BEA SPECjAppServer2002 records include:

• Fastest performance of any 4-CPU system in the dual-node category. Running BEA WebLogic Server 8.1 and BEA WebLogic JRockit 1.4.2 on an Intel Itanium 2 processor-based HP Integrity rx4640 server running Red Hat Enterprise Linux delivers 1710.23 TOPS@DualNode with a price-to-performance
  ratio of only $115.72 USD/TOPS@DualNode - these results essentially match Oracle's own best performance in this area.
• Fastest performance in the dual-node category. BEA WebLogic Server 8.1 on Sun Fire E6900 servers with UltraSPARC IV 1.2GHz processors running Solaris, delivers 4220.39 Total Operations Per Second (TOPS@DualNode) and
  834.98 $/TOPS@DualNode. BEA WebLogic running Solaris holds the top result; delivering more than twice the performance of the nearest closest competitor in this category, but at 7 times the cost of Oracle's top performing and best price/performance of 114.88.

Check out the latest results:
http://www.spec.org/jAppServer2002/results/res2005q1/

All results:
http://www.spec.org/jAppServer2002/results/jAppServer2002.html

And IBM's lone jAppServer2004 results:
http://www.spec.org/jAppServer2004/results/res2005q1/

These benchmarks are fairly useless in terms of making real buying decisions, they are useful from the perspective that it keeps the vendors continually optimizing and improving the performance of their application servers. What would be really interesting would be to see JBoss and JoNas competing here with the big boys.

Interesting. And really are there any figures about JBoss? Yes, I know about tests limitations etc. but it is a direct request from our client who is aware about this report -:)

Dmitry
http://www.servletsuite.com

P.S. actually we have the similar request about Orion too.

Dmitry, the most probable reason for the fact that there are no SPEC-tests for JBoss is that it isn't very suited for EJB's. The CMP engine in JBoss is not very scalable. Had they've had a reasonable implementation, I would think that they've released a benchmark long ago.

I've seen results from a bake-off with JBoss against another vendor on CMP2 (explicitly), and JBoss got beaten in that particular benchmark by a wide margin. That was not equivalent to SPECJAppServer200x, but rather CMP only. It could be different had the tests included Web, JMS etc.

My gut feeling is that JBoss works better with BMP, Hibernate, JDO and alternative persistance frameworks more than EJB's and CMP's specifically (don't know about 3.0 though).

Interesting. And really are there any figures about JBoss? [..] actually we have the similar request about Orion too.

I think anyone can run the tests and submit test results for publishing. The problem is that (a) you need the hardware (b) you need the people who can run them (c) you need the time and (d) you need to buy the test (US$2000 for SPECjAppServer, or be a SPEC member for US$6000 per year).

Peace,

Cameron Purdy
Tangosol, Inc.
Coherence: Shared Memories for J2EE Clusters

We're willing to provide the test hardware and SPEC tests if we can get a volunteer team to conduct the tests. We're willing to coordinate tests on a variety of app servers and publish the results if the license agreements will allow us to do so.

Contact me if you are interested in volunteering to conduct the tests.

Kurt Seidensticker
echoMountain, LLC
www.echomountain.com
kurt@echomountain.com

...yawn (-O