BEA today announced performance benchmark results demonstrating that BEA WebLogic Server 6.1 is up to four times as fast as IBM WebSphere Application Server 4.0 and 54 percent faster than Oracle 9iAS. The IBM results were achieved using IBM's own WebSphere Performance Benchmark Sample application, which models an online brokerage firm providing Web-based financial services.
Read the full press release:
BEA WebLogic Server 6.1 Is Four Times As Fast as IBM WebSphere 4
Detailed information on the BEA vs. IBM test
Detailed info on the BEA vs. Oracle tests
Wow, I was expecting to see something like this coming out of BEA. With all the anticipation of Websphere 4 and all the talk that Orion as Oracle 9iAS might shatter BEA's advantage, BEA needed to provide some kind of validation.
And what controversial validation! I love this paragraph from the Oracle test page:
"Oracle claims it takes only 30 minutes to demonstrate that Oracle9i Application Server runs 2-4 times faster than BEA WebLogic Server. BEA accepted the challenge and found that, contrary to Oracle’s claims, BEA WebLogic Server is at least 54% faster than Oracle 9i AS. Here are the facts."
Ouch! Talk about rubbing Ellison's Java One performance results back in his face. :)
Although I always get a kick out of seeing vendors run their own competitive tests and blast it over the press, in future the ECperf benchmarking initiative should change all of that. Once all the vendors have submitted benchmark results to ECperf, ECperf will provide performance validation, and I don't think vendors will be issueing their own unchecked benchmark claims anymore.
Congragulations to BEA in anycase. :)
Scary! These guys are bragging about doing 9-13 pet store transactions per second on a 4-CPU P-III xeon with a gig of RAM!?? Doesn't anybody else think that's ridiculously low for this kind of hardware?
I do...for what I need I need performance of at least 50 or 60 TPS...scarrry!
There is no mention of what jvm they used (I'm assuming Sun's). I wonder if Websphere's performance would improve if they were using IBM's jvm?
Also, does anyone know what happened to the TPC-W benchmark? The other tpc benchmarks have been the standard for years, why aren't companies using this one for web benchmarks?
Finally, it would be interesting to hear from one of the JBoss guys, as to how theirs would compare in an indentical test.
IBM's JVM was used in those tests. Good question, though.
Are there any comparisons of Weblogic/Oracle/Websphere with Gemstone? Have not worked with Gemstone but of all the app. servers on the market they seem to have the best architecture and idea of how things should be done. Please correct me if I'm way off but it just seems like Weblogic et al are just generic app. servers and there's not much differentiating factors (technological) among them. I've never heard of anybody going with Weblogic or Websphere because of performance.
To me, this isn't a huge thing.
The benchmark says that WebLogic is only 13.14 TPS and Oracle 9iAS is 8.54. Big deal. Do we really think that there is a huge difference between 13.14 TPS and 8.54 TPS? Especially in the 'pet store' application? I'm no Oracle fan, but I don't see how WebLogic can brag about 13 TPS.
I agree with the guy on this thread who said that he'd like to see it go up against Jboss. I'd like to see that, too.
Was this version of Oracle 9iAS running Orion? Just curious. I though that Orion was faster than WLS.
What I would love to see one day is an INDEPENDENT group run some tests. I doubt that if BEA were slower than WebSphere and/or 9iAS they would actually publish the information. Same with Oracle, same with IBM.
By the way, as an Orion developer I am aware of some misinformation regarding the particulars mentioned in the BEA-Oracle test page.
Oracle 9i AS does not support EJB clustering or the latest versions of key J2EE specifications, such as JSP 1.2, and JMS 1.1.
JMS 1.1??? The latest that I can find reference to is 1.0.2 - and was unable to locate it searching through java.sun.com....is it a new spec in review?
(Not to say, however, that I might have missed it entirely)
Oracle 9i AS does not support Web Services (SOAP, WSDL, UDDI), J2EE Connector Architecture, RMI/IIOP, JMX, JNDI, JavaMail, JAAS, and JavaRMI.
It (Orion anyway) does work with SOAP and it DOES support JNDI (duh), JavaMail and JAAS.
How benefits does Orion have over BEA?
YES, this is Oracle/Orion server.
And NO, it's not faster than WebLogic as these tests show.
These sorts of things are amusing, but it brings up a question that I have always had about such vendor-sponsored 'performance tests' and that is this (For fill-in-the-blank vendor):
When these tests are performed are they
a) Out of the box for each app server tested with no tuning?
b) Out of the box competition, tuned vendor?
c) Some tuning on all platforms (highly unlikely, just thought I would throw this in here)
I would guess b) but I don't know, of course. Perhaps I am being far too cynical....
It's a shame that independant tests are not possible due to license restrictions.
Well, the results from BEA indicate that they report results from (a) and (c).
All the test shows is that, out-of-the-box, BEA ran better. Supposedly the systems were "minimally tuned". One such "minimal" tuning was setting up the datasource in WebLogic to handle 50 concurrent requests; there's no mention about setting the WebSphere or Oracle datasources to handle that many requests. That would be a simple explanation for the differences.
No doubt we will see reports from IBM and Oracle shooting down the BEA report. Maybe by reading between the lines in all the mess we'll get a clear picture.
In any case, why rely on a generic performance test? Build your application to a portable level, construct a load-test environment for it, and actually run it in each environment. Nearly all EJB vendors now provide evaluation versions, so you can see for yourself which server is best for your applications.
I'm curious, what PetStore does so performance is 8.54-13.14 TPS on a 4-way NT server ???
Maybe it is because I am realtively new to J2EE/EJB (coming from a COM background) but the Petstore uses some really bizarre strategies that seem to my casual observation to be incredible resource intensive...such as a myriad of levelks of abstraction between the web and data tiers, reams of serialized objects.
I know some of this is a necessary part of working in the EJB environment, but it still seems like overkill.
While Pet Store isn't the greatest application, as it makes some questionable design choices, the heavy abstraction that Toby mentions above isn't unique to EJB/J2EE, it's just part of OO design using a dynamic language. A lot of people from the procedural community aren't used to OO design in this sense, since the "lots of little objects and little methods" didn't seem to sink in (mainly because it's the opposite of what C, Fortran, or COBOL encourages).
The EJB's should expose value objects which should be potentially mapped to distinct view objects. This enables the business tier to evolve and be maintained separately from the presentation, with interface changes only impacting the code that does the mapping between the two tiers (which should be isolated to one place).
These value objects are Serializable in the pet store mainly because EJB 1.1 does not support local objects -- even though it's a server-specific optimization which is used 99% of the time in the real world and probably would have sped up the transaction well beyond 13 TPS.
The real problem with layers of abstraction is the code-bloat associated with using a strongly-typed approach of get/set methods and "merge" methods that pour data between different object representations. There are several ways around this (some of which are discussed in Floyd's upcoming book on patterns), but generally they involve putting properties in a container, like a Hashtable or Map.
I've developed a highly abstracted system with GemStone that easily could handle over 100 TPS on a single-processor Sun Ultra Enterprise 250 (much smaller machine than the one in BEA's benchmark.) While this used GemStone's cache, I'm quite confident it could be replicated or even made faster with a tuned configuration of EJB + Oracle. It's just a matter of having the expertise to tune the design and configuration.
Thanks for the clarfication.
I guess part of the issue is the continued tention between fine-grained objects and coarse-grained ejb...which is part of the reason we end up with value objects, i guess.
I remember a few years ago Microsoft had a reference application built using ASP and COM/MTS that was roughly equivalent to the petstore. Problem was it didn't reflect most of the 'real-world' strategies that people in the trenches used on projects everyday.
I tend to be rather wary of such things in informing my learning...
The IBM WAS available for download is single server only. Most folks don't run on this in production, I suggest BEA use the WAS Enterprise Edition for benchmarks. Also note that the Single Server WAS default installation puts the DataBase on the same machine as the Application Server. BEA put their app server on one machine and the DB on another, what's up here? Lets compare Apples with Apples. These kinds of tests mean nothing. In the real world all systems are tuned and not run out of the box, this is due to the fact that all applications and integrations are diffrent. Once again I stress running the IBM Enterprise Edition of WAS with the DB2 database on a seprate box.
This is “apples to apples”:
BEA WebLogic Server tested was also a "single" edition, well comparable to WebSphere single edition. Both ran as single instances.
BEA ran WebSphere and the database on separate machines, as well as the clients. The database, however, was not DB2 - it was Oracle. Is WebSphere limited to, or behaves well with DB2 only?
I'm not surprised ... if you look at the numbers that *Oracle* published, they had a TPS of 5 for 100 users on the following configuration:
app server: 2x 450 MHz Sun Ultra U60, 1GB RAM, Solaris 2.7
client: 1x 450 MHz Sun Ultra U60, 512MB RAM, Solaris 2.6
db: 4x 400 MHz Sun Ultra E3500, 1GB RAM, Solaris 2.7, Oracle 8.1.7
Face it, 13 TPS on that hardware just sucks. Is this because EJB is too slow for real world use? Or is Pet Store just poorly coded?
I agree with the jboss guy.
I know for a fact that jboss on CP NT system kills IBM on a way more expensive Solaris box. A couple of buddies of mine ported an app over and tested it out. I don't remember the details but the results weren't even close.
As for the Pet Store, there certainly seems to be alot of nested if structures for an application created with design patterns in mind, the ideas are great but the implementation leaves a little to be desired. It is my humble opinion that the Pet store was not that badly coded, most EJB implementations just don't offer great performance. I wouldn't bother putting a production app on NT with Weblogic or Websphere. Unless you like 8 -13 tps.
Yes, Pet Store demo is just a piss-poor benchmark... didn't this come out with the original J2EE Reference Server? What was that, 2 years ago? Furthermore, just the thought of using a B2C-mockup application is antiquated! First of all, there aren't many real world b2c sites left, and second of all, J2EE isn't the best thing to use for developing shopping carts, category lists and other b2c idiosyncracies!
ECPerf is a good benchmark, but too low-level. Perhaps its time Sun Blueprint whip out an updated J2EE demo!
On the allegedly low number of transactions shown by BEA vs. Oracle servers running the Pet Store demo:
A single transaction in this context is the whole sequence of atomic transactions executed by a single client that on the average results in the workload tested. Please see Oracle's site for the workload pattern description (white paper).
Only when the whole mix of operations for a specific client is completed the transaction is considered finished, and is counted at this point.
From this angle, the demonstrated number of transactions is not so low as it may seam. It may contain thousands of actual physical transactions executed by the application server!
The Pet Store is definitely not the most efficient application, but it does not matter as long as we compare apples to apples.
IMHO: ECPerf should be the best way to mesure
performance of J2EE appserver. ie. Out of the box
(with out any tunning) Borland AppServer 4.5 is
5 times faster than BEA WebLogic; does that
mean BAS 4.5 outperform IBM or Oracle????
I'm just amused that BEA says "We continue to encourage customers to benchmark application servers using applications and workloads that model their own business requirements, and to our knowledge, BEA always comes out as the performance leader in these customer benchmarks."
Their license states something completely different (unless they have changed it)... well, you may compare, but don't publish.
I wonder, are they afraid? We would really like to publish our results BAS 4.5 vs. BEA 6.0 (or 6.1 now). Well, ECPerf should be a better indicator.
No surprise that IBM WAS is slow though.
Bernhard Messerer said:
Their license states something completely different (unless they have changed it)... well, you may compare, but don't publish. I wonder, are they afraid? We would really like to publish our results BAS 4.5 vs. BEA 6.0 (or 6.1 now). Well, ECPerf should be a better indicator.
No surprise that IBM WAS is slow though.
It has always concerned me that many Vendors (Oracle, BEA, etc) don't allow users to publish benchmark results while they release such useless comparisons. Given the limits, it would be helpful if a third party conducted, or at least certified the results. Besides, we've all seen the many claims of all DB vendors that they are the fastest. Sometimes two vendors have claimed leadership on the same benchmark (TPC) on what looks like nearly identical hardware.
Best solution - If you are in a major IT shop, get the vendors to perform an ECPERF or other benchmark that you define and deliver to you the DETAILS. Let them use their own hardware, but make sure you set the ground rules appropriately. If they don't cooperate, drop them.
Folks, did you know that BEA and Oracle product's license agreements still have restrictions preventing the reporting of performance results of applications using WebLogic and Oracle application servers' software (and Oracle database too)? (BTW: IBM has no such restrictions for DB2 or WebSphere Application Server 3.5.x) Therefore Oracle and BEA both violate license agreements of each other and potentially can sue each other. That will be amusing! ;-)
As far as benchmarking, I suggest to look at the recent Giga 's "Ignore Application Server Benchmarketing" IdeaByte by Randy Heffner, July 13, 2001.
IMHO: Only independent performance benchmarks can be considered, not vendor's. The only such benchmark that is freely available to the public that I know of is recent (May'01) PCMagazine's perf benchmark: "IBM's WebSphere Application Server, Advanced Edition 3.5, was by far the fastest and most scalable server on the scenario test. At its peak, it maintained 4,000 virtual users at 177 pages per second. Because of its sophisticated dynamic page-caching algorithms, WebSphere was also the clear winner on the title search test. It also had the best A.R.T. (Average Response Time) at peak performance on nearly all of the tests… IBM's WebSphere Application Server, Advanced Edition 3.5 has an array of speed-enhancing caching techniques, rock-solid load balancing and failover handling, and a complete complement of developer tools for creating Web applications. Its management console offers minute control over settings; support for a multitude of operating systems and Web servers sets WebSphere apart. Excellent performance in our testing is the icing on the cake… If your business is looking for a complete scalable and high performance environment for Web applications, WebSphere Application Server is ready." WebSphere beat Sybase, Borland, Microsoft, and iPlanet in the High-End Application Servers category.
// ********** HERE IT GOES !!!!!!!! ************
BEA and Oracle "declined, citing lack of resources to support our review process or an unwillingness to undergo performance testing
// ********** HERE IT GOES !!!!!!!! ************
" according to the review.
Full report see at:
The reason BEA did not participate in the PC Magazine tests is that WebLogic 6.0 was just about to ship and our engineers were fully employed in getting it out on time.
The acid test (IMHO) is the number of users deploying an appserver for high volume transactional applications.
Ask your vendor of choice for reference implementations.
Borland has announced their ECperf numbers:
Third party analyst research group's site released their competitive J2EE AppServer: analysis.http://www.cmis.csiro.au/adsat/j2eeexec.htm
Also worth pointing out:
Sun announced it has demonstrated the world's largest system ever to run ECperf Benchmarks using an Eight-way Sun Fire[tm] V880 server with 16 GB of memory, running the Solaris[tm] 8 Operating Environment (OE). This test resulted in a world-record performance of 5961.77 BBops/min@Std and $51/BBops running Borland AppServer 4.5 with Oracle 8i and Java 1.3.1.
But don't take our word for it... Prove it to yourself!
You know just when you thought you heard enough here comes some more .... for years all we heard was Oracle is better than Sybase and SQL Server and you had company after company publishing comparison after comparison .... now we are in the new millenium and what do we hear Oracle is better than Weblogic is better than IBM. Isn't enough enough? I will tell you this I have worked on all 3 application servers listed above for and they all have their quirks and problems and performance issues. I will say this if we have come down to using the PetStore example from Sun as a bench mark standard God help us, that thing is written with the java purest in mind and in no way should reflect any kind of performance standard it is in my opinion quite the opposite, if you want to know how to slow your applications down to a crawl implement using the java petstore example. Don't get me wrong there are SOME clever tricks that you can learn, but it is a performance nightmare.
Just my two cents,
I think the only reason BEA used petstore was to counter Oracle's claim that they (Oracle) were the fastest server on the planet -- Oracle used petstore to "illustrate" this fact, so it was natural for BEA to use the same benchmark to prove otherwise.
Long term, I agree that ecperf is the way to go.
If used the generally available version of WebSphere, which is the Single Server Edition(Development Edition). WebSphere Advanced Edition full version is not available for download. It contains the full whistles. The single server version doesn't use a database repository. They should take a look at the full version first.
Oracle has posted its response at
I think in terms of advanced features Oracle9i Application Server supports it would be helpful to look at the following:
(Checkout the discussions on idevelop chats)
In addition Oracle9iAS has support for
1) Wireless integration
2) Portal support
3) Web Services
4) First application server to support Edge Side Includes (ESI) - (full and partial page Caching)
5) Oracle Workflow and support JMS and Advanced Queues
6) support for single sign on server, LDAP server from Oracle Internet Directory
7) Support for building XML applications using the Oracle XDK and JDeveloper toolset
8) Intergration services part of Oracle9i Application Server
9) Decision support services and Reporting Services
Much of this info is available at
Is it just me or is anybody else getting a wee bit tired of these "benchmarketing" wars?
What really matters is the number of referenceable customers /production sites that an app server vendor can provide. As Oracle has demonstrated in the past, talk is cheap (remember, they were going to put Microsoft and Siebel out of business a few years ago?). Last I checked MSFT and SEBL are still around and doing very well, thank you!
Have just read the response from Oracle.
So, which 9iAS are they talking about now? Is it the one they got from Orion, OC4J (and the one that BEA tested), or the one they announced just few months before and that never really took off so Oracle dropped it? Or, is it the one before that? Oracle seams to be having this thing about dumping their J2EE containers (at least 3 so far that I know of), not being able to settle with any so far.
Yes, WLS 6.1 is in beta, that’s true. So, which 9iAS is available for sale?
Web Services... at the latest JavaOne, Oracle’s people did not even know what Web Services were...
The 9iAS available for sale, as I understand it, is the one with Orion...
Where can I get a white paper on some of the primary differences between WLS 6.1 and WAS 4.0, with respect to developing components as well as with respect to feature like HTTP Session Replication, CLustering, Component replication, deployment etc?