News: Web Services Benchmark Under Fire: Price / Perf Metric Ban?
SPEC is in the process of creating a new Web Services benchmark, AppPlatform. The benchmark will consider issues typical in customers' applications, such as transactions, persistence services, Web services and messaging.
- Posted by: Dion Almaer
- Posted on: February 09 2004 22:23 EST
There has been a bit of a ruckus over the issue of including a price/performance metric. Some vendors want it banned as it can be easily skewed.
Some are arguing: "This would enable some vendors to run the tests on the biggest most expensive boxes without having to mention it. That way they would get great performance and wouldn't scare people with a large Price/TOPS result."
Others say: "I think its actually the right thing to do. It was very hard to do an 'apples to apples' comparison price-wise. Some vendors give the software away cheaply, and make you sign an expensive service agreement. They all find loopholes to hide the numbers anyhow, so it becomes meaningless."
What are your thoughts? Should price/performance be banned or included?
Read New Web Services Benchmark Under Fire
- This will make the SPEC little more than academic curiosity... by Edgar Sanchez on February 09 2004 23:20 EST
- Web Services Benchmark - Oxymoron by John Davies on February 10 2004 05:04 EST
Without any idea of prices, the performance numbers are just an academic curiosity. What do I win from knowing that I can get 3,000 operations per second if it happens to be well above my budget? I think solutions like Resin or Glue which has worked just right for me (or JBoss, Tomcat, Apache, and of course .NET) are the most to suffer for this ban supported by (among others) IBM and Sun.
I also noticed that AMD was against price/performance publication. Since their chips are currently the fastest XML processing server chips out there (at least in Java benchmarks,) and basically the cheapest to boot, I wonder why they opposed it? In other words, maybe there is more to it than was reported in the article.
Coherence: Clustered JCache for Grid Computing!
re: John Davies' comment, "why bother?"
The AppPlatform benchmark is not just for web services. The article says the benchmark "will consider issues typical in customers' applications, such as transactions, persistence services, Web services and messaging, said SPEC officials." Web services got the headline, but it is bigger than that.
I won't get into the "Java world is way ahead" with binary protocols. That particular dead horse is starting to stink.
But, SPEC is the owner of jAppServer, right? Isn't price/perf included there?
Why would this be different?
Is this "ban" of price info only for the new AppPlatform benchmark, or for all SPEC benchmarks?
re: John Davies' comment, "why bother?"
> I won't get into the "Java world is way ahead" with binary protocols. That particular dead horse is starting to stink.
*Protocol that is used by the computers to talk to each other*
*Protocol that is overly verbose and human readable*
It's stinky to combine the above two.
Why bother? It's like testing FAX machines, it's going to be SLOOOOW! Unless of course you buy more and more hardware.
If Sun and IBM dropped WebServices and just supported RMI or CORBA with JNDI where would WebServices be? Buggered, it would have no use what so ever. It's like the cold war arms race, WebServices is Microsoft's starwars initiative, useless if the opposition don't have ICBMs.
Don't get me wrong, I see some good uses for WebServices, it's here to stay but let's not get too excited about it. The Java world is way ahead of this ancient form of communication and we should get on with the future instead of being dragged down by this legacy framework implementation.
Enjoy the benchmarks, next up "Scalable SOA with the Seattle Carrier Pigeon".
I can not agree more with you. Web services performance sucks like big huricane.
It is interesting to see that everybody is bashing entity EJBs because they are slow, but there are not many people saying the same about Web Services.
Not to mention that performance degradation from Stateless session EJBs + JDO or alike to Entity EJBs (about couple of times slower) is less then performance degradation from CORBA/RMI/.NET remoting to Web Services (about order of magnitude slower).
But I suppose that as people start using WebServices in mission critical applications the performance lessons will be learned the hard way and Web Services will be where entity EJBs are now.