Enterprise integration vendor Tibco is suing Sonic Software over its comparison of a Tibco product to a Sonic product in a white paper on its Web site. The white paper compared results on Java Message Service capabilities of Sonic's SonicMQ and Tibco's Enterprise for JMS. Tibco claims that the results of the testing were unfair and the white paper wasn't from an independent third party.
The whitepaper can be found here:
Tibco Sues Sonic Over Product Comparison
Sonic Software's official statement
In the InformationWeek article:
"Sonic said on Sept. 2 that too many software companies, including itself, have tried to exclude direct comparison of products. Tibco's terms for downloading and trying Tibco Enterprise for JMS exclude testing the product for comparison purposes and making public the results."
Are there any precedents in case law for this sort of thing?
What a good way to attract attention to something which will be bad for your business and product !
In future let TMC do your performance comparison tests. That way theres not even the remotest hint of bias :-)
In this case, Tibco is doing the right thing. Sonic screwed up.
> In this case, Tibco is doing the right thing. Sonic screwed up.
Don't agree. Tibco just needed to produce their own white paper with their own results and publish it on their own website. Why get the lawyers involved ? Who does it serve ? Not the customers or the businesses.
In this case, Tibco is doing the right thing. Sonic screwed up.
>Don't agree. Tibco just needed to produce their own white paper with their >own results and publish it on their own website. Why get the lawyers >involved ? Who does it serve ? Not the customers or the businesses.
Daniel, I don't think you understand the problem with benchmarks. If Tibco produced a corresponding report, well, then we'd have yet another benchmark that doesn't make sense. The real issue here is that companies must stop publishing internally produced performance benchmark. They are ALWAYS misleading in some way or another.
In this case, Tibco is doing the right thing. Sonic screwed up.
> >Don't agree. Tibco just needed to produce their own white paper with their >own results and publish it on their own website. Why get the lawyers >involved ? Who does it serve ? Not the customers or the businesses.
> Daniel, I don't think you understand the problem with benchmarks. If Tibco produced a corresponding report, well, then we'd have yet another benchmark that doesn't make sense. The real issue here is that companies must stop publishing internally produced performance benchmark. They are ALWAYS misleading in some way or another.
Han, I do understand the problem with benchmarks, and I agree they are always misleading which is why the most important aspects of any benchmark/test/review is firstly who conducted it and secondly the results. I just don't see how this action is going to benefit Tibco or its customers. Can you explain ?
I think it would serve it's customers in a 'feel good' type manner. Tibco are telling it's customers:
1) Don't worry, what you've bought (and paid lots for) is good. Not like that other guy says.
2) If others are going to tarnish Tibco's reputation, then they will be dealt with.
I don't agree with getting lawyers involved, but you can see why they might.
Don't worry, what you've bought (and paid lots for) is good. Not like that other guy says.
Surely if someone's already bought it and paid lots for it, they already know whether it's any good or not.
It seems to me that the only reason for Tibco to pursue the lawsuit is because they think that Sonic's benchmarks are likely to result in lost sales for Tibco. It's potential customers they're worried about, not existing ones.
"Han, I do understand the problem with benchmarks, and I agree they are always misleading which is why the most important aspects of any benchmark/test/review is firstly who conducted it and secondly the results. I just don't see how this action is going to benefit Tibco or its customers. Can you explain "
Daniel, I serisouly doubt this will affect Tibco's customers directly. What's important with this lawsuit is that it can set precedence. That would be beneficial to everyone in the industry.
Tibco just needed to produce their own white paper with their
> own results and publish it on their own website.
Ah, but Sonic's results are a nice red colour, while the
best that Tibco can manage is a washed out blue against a
grey background ;-)
There was something recently like this that made it into case law where a district court ruled that limitations in licenses which try to prevent benchmarking were illegal. I wish I could remember who this was in relation to. Likely someone else does.
The decision you are talking about may be a New York Supreme Court decision (People vs. Network Associates) that found such a clause as overreaching and preventing public discourse. The best article I can find on the decision is from the New York Law Journal: http://www.brownraysman.com/pubs/articles/pdf/NYLJ-2-11-03.pdf
Unfortunately, this clause is included into the TIBCO evaluation license:
if you do make a benchmark, keep it for yourself.
I'm sorry to hear that bit of news. I did this type of benchmark myself. For my kind of business (financial back office) and for my QoS reqs, TIBCO E4JMS 3.1 was so superior... in many respects, by a big margin. Please don't get me wrong: I'm tracking the advances that Sonic MQ has made, and they are incredible.
It is just plain stupid, or whatever you want to call it. Much like hiding a gem deep in your closet. Sigh... :)
I wish TIBCO fired their lawyers.
Always remember numbers are only numbers, you can prove anything and everything with it.
One of the problems in this particular comparison was that the test was done by a "independant" company, however the test wasn't executed in their laboratory. The test was done on Sonic premises on Sonic hardware.
Other points to remember
1. the so-called independant company is paid by the requesting company, is the company going to tell the requesting company that they are slower?
2. In head to head comparisons you can tune your own system and use a vanilla installation of the other system. (where is autonomic computing when you need it :-) )
3. When tuning a system you can turn of some features and maken it a non-real life environment but still be compliant with the test
BTW, the gag attempt by tibco was bounced.
Their product is somewhat average but their marketing dept is top notch.
And note that vendor sponsored benchmarks always result that the sponsor is the best. Why else should he sponsor it?
As much as I like Sonic, I hope Tibco wins this case. The reasons is that these kinds of comparisons are always unfair - at least they should have asked Tibco to tune an installation of their product. I totally understand companies that prevents benchmarking through the licence agreement!
I don't even trust third-party evaluations... only my own. How third-party is a third-party anyway? We can never know - the alliances map in the software industry is mysteriously secret and very complex.
By the way, our shop use Tibco, Sonic, MQ and a host of other messaging products and I can tell you that none of them are good at everything. Sonic is not always the most scalable, nor the fastest, but we like the simplicity and overall performance is good enough for many applications.
I'm an old Tibco employee, I remember the stress bench I've done for a customer between Tibco Jms and Sonic Mq. Tibco Jms in it 1.0 version had an excelent performance in pub.sub, queuing on diffrent mode comparing to sonic. The main advantage of Tibco is the strong management of multiple concurrent client, so concurrent session. Sonic Mq can managent a few concurent client.
No Offense yazid, but I don't think you ran the tests correctly. We just recently ran an ad-hoc stress test on connecting to the older sonicmq v.4 broker simulating 500 physical tcp/ip connections to a single broker with an average connect time of about 200ms per connection and flat memory usage profile on the broker after a few seconds. I would agree with other posters here that it would be interesting to see a true third party comparison. I'm not familiar with tibco jms, so I can't comment. I have used and tested other MOMs and from just my experience, I can say that MQ is great if you are on a mainframe connecting to other platforms, but the jms support that I worked with (pre-griffin, greyhound, whatever) was quite poor. Tib RV was ok, but a dog performance wise. Tallarian smartsockets which I think tibco acquired was very fast, but like dumping a truck full of rocks into a bucket, some fell in, some didn't. JMHO.
oops, my math was wrong, 500 connections, ~20 seconds, so ~ 40ms per connection on a dinky sun box ;-)
mainframe connecting to other platforms, but the jms support that I worked with (pre-griffin, greyhound, whatever) was quite poor. Tib RV was ok, but a dog performance wise. Tallarian smartsockets which I think tibco acquired was very fast, but like dumping a truck full of rocks into a bucket, some fell in, some didn't. JMHO.
heh, this runs counter to a benchmark that I was involved in between Tibco and Sonic. We gave each vendor a week to tune the benchmark. We didn't want a level playing field... we wanted each vendor to produce the best numbers that they could using what ever they needed. The constraints that we placed on them were ones that came from the problem domain. Everyone knew everyone elses results so, there were no secrets.
IMHO, the only benchmarking results that you can use are the ones that relate to your problem being solved in your environment. Thus, I'm not surpised that Sonic has been able to produce a test where they can out perform Tibco. In our case, the problem was scatter/gather of bulky messages with one producer and several thousand consumers. Under these conditions, Tibco won that contest with ease. We ended up giving Sonic an extra week and offered them more time but they decided that the gap was so large that there was no point in continuing.
I guess it depends on what you were benchmarking :-). At the time, we were testing guaranteed delivery of various message sizes. Using tib rv (not the ejms which I am not familiar with) and "certified" messaging, it was indistinguishable from mq series(both out of process and fast path binding modes) in throughput (basically, they were both pretty slow :-). Smartsockets seemed incredibly fast, until we were able to reproduce scenarios that caused buffer overflows and simulated disk failures by corrupting the control files, that is why I referred to it as a dump truck.
What Sonic did might be legally wrong but they have got the right idea. The fact that a company like Tibco doesn't let you publish benchmarks must tell you something about their product. If you knew you had a superior product you're hardly going to shy away from public benchmarking.
Although Tibco took part in the original JMS spec they seem to have dragged their feet for years. It was only two years ago, when we already had nearly 40 JMS implementations, that Tibco's JMS was so crap that they had to buy one of the competition (SmartSockets). You wouldn't believe how many patches, versions and updates we got through at a major bank before sadly dropping JMS and resorting back to RV.
More recently I've been working again with Tibco's new JMS and I must say it's come on leaps and bounds since the earlier versions, it now even supports JMS 1.1.
I must admit though that in the case between Sonic and Tibco I would side with Sonic, their product is more polished, cheaper and in terms of "pure" JMS more mature. I know JMS is just an interface and RV is an implementation and therefore far more mature as a product but Tibco had to make a lot of hacks to get the RV architecture to conform to the JMS spec and that makes it less solid in my book.
I wouldn't like comment on our performance figures (as a comparison) but we (C24
) would recommend either product for high performance integration using IO
, XML, FpML, SWIFT or whatever.
If anyone cares to try them out, give each of them an hour to play with, you'll get about 55 minutes to play with Sonic and still be installing Tibco's JMS when your hour's up. Once you've got it up though Tibco's can definitely hold its own at last.
Play devil's advocate for a moment, this licensing limitation is a protection against FUD. If anyone can benchmark your product and publish results, they may purposely published a skewed test. And even if there's public outcry, and even if it's obviously skewed, alot of people _will_ believe the original report. Just like, in the opposite direction, many people believed another JMS product's numbers for persistent messaging for their own implementation, even though it wasn't disk forcing.
The point is - the original report often lingers on, and people tend to forget protests, and a company's reputation can take an undeserved hit.
Of course, it's all nonsense anyway. Anyone who's chosing a JMS implementation for performance had better load test products in their own environment. Never, ever rely on other people's performance tests.
If anyone can benchmark your product and publish results, they may purposely published a skewed test.
You're right, they probably will but what's the difference, everyone's going to skew towards their own preference. I suppose where it becomes unfair is when a powerful company with a good marketing engine gets involved. For example a large cooperation with a flawed product that spend $1billion trying to market it against an obviously superior platform.
It's all just market force, we frequently see benchmarks between hardware vendors, hard-disks, printers, monitors, scanners etc. etc., it never seems to do them any harm. Reliable delivery of messages at high speed isn't everything, true it's almost everything but Tibco has other assets, they are much better placed in the financial market place for example, they have other products in the line like BusinessWorks and InConcert although calling those "assets" is rather stretching the meaning of the word somewhat.
Sonic need more marketing in the financial services market place, what they've done with Tib. is just marketing and as you (Mike) have said, "it's all nonsense anyway", as is most marketing!!! Did I mention my company
> Never, ever rely on other people's performance tests.
Couldn't agree more.
"What Sonic did might be legally wrong but they have got the right idea. The fact that a company like Tibco doesn't let you publish benchmarks must tell you something about their product. If you knew you had a superior product you're hardly going to shy away from public benchmarking. "
John, you are missing the point. Benchmarks are unfair and no product is superior in every respect. That's why Sonic and other manufacturers can easily skew test results in any direction they want.
I hope Tibco wins this one, not because they have a great product (they do, so do Sonic), but because benchmarks are evil.
John, you are missing the point.
I don't think so, I think I just wrote it down wrongly, I agree with you, benchmarks are generally just marketing and nearly always unfair.
My point though was that they shouldn't be outlawed, let people make up their own minds. If they have exactly the same set-up as the benchmark and want pure speed then it's a good guideline but I suspect 99.9% of the readers have different requirements, the benchmark is therefore total "bollocks" for most of them.
What the benchmark did do is to increase the awareness of Sonic JMS as a product and to place it as a direct competitor to Tibcos'. I have 6 Reuters/Tibco people sitting within 5m (15') of me and all of a sudden they know who Sonic is and have asked me for a demo (since I seem to be the only one here who knows anything about it in this office). It's amazing how few people know about the other 30 plus JMSs on the market, most of the the Reuters/Tib. people here (a bank in the City of London) seem to believe they're the only Pub/Sub vendor and that IBM is the only other company that has a messaging offering. In the space of 2 working days they're now asking questions about the longevity of their own product and why this bank actually chose RV (they're sadly not using JMS).
> I hope Tibco wins this one
OK, I'll buy you a beer if Sonic win.
> because they have a great product
> so do Sonic
> but because benchmarks are evil
ditto (I agree).
Benchmarks can definitely be skewed in almost any direction you want.
Tibco should be sued for fraud for selling an entirely worthless product.
The Rendezvous messaging backbone is a fine product, but Tibco has built a set of tools on top of RV that are complete drivel. The development tools work well for simple demos, but on real development projects they cost more time then they save. Version Control can be a nightmare with the tools, so multi-developer projects lead to unresolvable conflicts.
Tibco originated in the financial sector for delivering market updates. In this environment, lost messages are common and expected. Unfortunately, Tibco has attempted to adapt this product to the enterprise integration market. The result is one mammoth kludge. Enterprise integration typically requires once and only once delivery of messages. Lost messages are unacceptable. Tibco can be configured to operate in this manner, but it is a square peg in a round hole. As another poster noted, the JMS implementation is a similar kludge.
Buyer Beware. Tibco products force you into very proprietary solutions. Standards are supported, but as an afterthought.
I would never pay a dime for Tibco products. Actually, I would pay not to use them. Typically, the developers that speak highly of Tibco possess little experience with anything else. I've worked with Vitria, various other JMS vendors, MSMQ, and IBM MQ. I actually have the most experience with Tibco, and it is by far my least favorite.
I will again reiterate that my criticisms pertain to the development tools that sit ontop of RV. RV, as a standalone messaging backbone, works perfectly well. I also must commend the marketing department for successfully selling a worthless product.
Tibco is RV and the fact that RV is a great stand alone product must not confuse the users to see it and other "tools" based on RV as an integration platform. I wonder why do some fall for Tibco's marketing gag though?
As for the Tibco suing some one doing a bench mark, I think it juat demonstrate how much experience they have in their product.
Meant to say confidence instead of experience, mut I might be on to something here!
IMO this is badly wrong. First you sue a compeptitor. Than you sue paid independant 3rd party evaluators. Than you sue everyone how does benchmarking with results that don't look your way. Sad...
The sad part is that the scum of the earth (the lawyers) will be the only real winners!
Ahhh...now I know the reason why Sonic is pestering Tibco.
It' that SIMPLE!
Don't believe marketing blabla. Borland will replace SonicMQ with Tib/JMS and that's the reason why Sonic fights back.