JBoss is pleased to announce the publication of a performance comparison of enterprise messaging systems using the JMS API.

We chose a simple set of performance benchmarks to measure throughput with the most common messaging use cases, from lightweight publish/subscribe messaging to persistent publish/subscribe messaging.

The benchmarks were chosen to give a representative and useful overview of relative performance in the most common uses cases, not to give an exhaustive view of the performance characteristics of all systems in all situations.

Due to licensing restrictions, for several proprietary messaging systems, we are unable to attribute results to them, therefore their results are anonymised in the report.

The results clearly position HornetQ as the performance leader across almost all use cases, and demonstrate it is no longer necessary to purchase expensive proprietary messaging systems to get the very best performance.

Read the full report.

HornetQ folks should focus more on stability than performance. After all, that's what REALLY matters when people choose message oriented middleware.

I have extensive experience with HornetQ, ActiveMQ and other new and old JMS implementations.

Bottom line? Yes, HornetQ is fast, but in practice the difference isn't all that great. In fact, for most real-life scenarios I have come across, the slight gain is lost as noise in network overhead.

The problem with HornetQ is that it has lots of annoyances that pops up under real-world projects, and in some cases it's just ouright unstable. Also, the documentation is shit. For the core api, it's even wrong in more than one case.

An example of annoyances is the huge message implementation. It fails to implement a safe way to delete the blob after successful delivery, for instance, and getting the latest release to actually work isn't easy.

I have been using HornetQ to transfer large amount of small messages (occasionally with AIO) and didn't have any kind of stability issues, and I could actually integrate it quite nicely during testing, i.e. I coule start embedded HornetQ withing Eclipse while testing without problems etc. I used version 2+.

Don't know what issues you encountered in your spefic case.

We are really efficient in fixing eventual issues. 

@Chief Thral. If you eventually ever find an issue, just tell us and we will fix it. We have done it quite well.

We put a lot of effort on testing it being proactive in maintaining it and fixing any eventual issues.

HornetQ's code is really clean, what makes it easier to maintain it, as well as giving us space for innovation at future releases.

We are really crazy about stability. More than we are on performance, and to keep up with a a good documentation. Very few Message Systems have a documentation as complete as hornetQ's what didn't make my job any easier.

I could say HornetQ's doc is the best on the messaging market, but to avoid any miss interpretation of my words.. I will just say it's one of the best on the market for both open source and commercial messaging systems.

I could say HornetQ's doc is the best on the messaging market, but to avoid any miss interpretation of my words.. I will just say it's one of the best on the market for both open source and commercial messaging systems.

That's just not true. A huge amount of methods lack javadoc api docs, for instance, and some of the examples shown no longer works.

I could say HornetQ's doc is the best on the messaging market, but to avoid any miss interpretation of my words.. I will just say it's one of the best on the market for both open source and commercial messaging systems.

That's just not true. A huge amount of methods lack javadoc api docs, for instance, and some of the examples shown no longer works.

That's odd, all examples are tested before release as part of the release process so we know they work. If you can be more specific about which ones don't work for you, someone can investigate.

Also all public API javadoc is documentated. (These are the classes with "api" in the package names). If you can point us to one that does not have javadoc, then again someone can investigate, although I am noty aware of any public API that is not documentated.

Public api javadoc is here: 

 

I agree with the post above. Performance is not everything.

As the test show any decent jms implementation will max out a Gb interface in non-persistent messaging with realistic payloads and realistic number of consumers.

Sure hornet much faster that the rest with 12byte payloads and 1 consumer but who uses jms for this. There are much better high performance pub/sub products on the market for this.

For the case of persitent messaging hornet appears to be faster than the rest but my own experience with guaranteed messaging is that if you are working in a transactional information flow then raw speed is rarely of paramount importance and even when it might be the JMS server has never been the problem.

I think reliability, ease of use, ease of management (especially for persitent messaging), alerting, support/community would be my prime concerns.

well done for the high speed implementation though!

Let's not generalise about this. Sure, performance is not critical for some users,  but for others users it definitely is. It just depends on the user.

Quite apart from non persistent perfornance, persistent performance can often be the bottleneck, e.g. with large order processing systems.

Now if HornetQ can do 9000 transactions per sec, when A.N. other system can only do 500 on the exact same hardware, this can be a *big* win for the user since it means they can do the same one one server that takes 20 servers to do with the other system.

Has anyone compared the performance of ZeroMQ (http://www.zeromq.org/) against any of these? I understand that it is a non-persistent messaging system, so the comparison would be in that mode too. Their numbers good very good, and I could use its inproc messaging to implement lockless concurrency, which is an added advantage. Any experience from anyone here would greatly be appreciated.

Really nice messaging system :)

I do have to say that I hate it that JBoss has the tendency of changing names of their projects. This used to be called JBoss Messaging, but now it's called HornetQ. JBoss Cache did the same thing.

The forum, download pages, etc of JBoss Messaging are still there, making the whole thing just confusing. JBoss has three messaging systems? Which one should I choose? Is JBoss Messaging still being developed? I like JBoss' software, but renaming of projects like this is just confusing.

JBoss Messaging still being developed in bug fix mode. All new enhancements are going towards hornetQ.

We just renamed JBoss Messaging 2 as HornetQ because after refactoring and refactoring, it became a different and much better system. Simple as that.

People didn't have any problems on reaching us :-)

We just renamed JBoss Messaging 2 as HornetQ because after refactoring and refactoring, it became a different and much better system. Simple as that.

I understand that. It seems to be a trend in the industry, but one I don't particular like. What if say Oracle renamed Java 7 to Bogor, with the motivation that Java 7 is a different and much better version than Java 6 ever was, and that it's as simple as that.

And then, if people had a question about say a for loop, that Oracle forum members would shoe those people away saying: That's a JAVA question, please ask on a Java forum pretending Bogor is a completely new language.

Rebranding is one thing, but why the associated attempt to make it look like it are two different products? In the case of Java Messaging, why not just rename the existing forum?

The analogy doesn't apply to HornetQ here. You could make the analogy to JBoss.

JBoss AS still alive.

JBoss Messaging isn't a very creative name... it denotes more a sub-project of JBoss (What JBoss Messaging seemed more like).

HornetQ can run embedded, standalone... and inside the application server as well.

We also changed licensing. as you can look at www.hornetq.org.

Anyway.. in the end everyone can find us and nobody got really confused about it.

I respect your opinion though. However the majority seems to have understood the change.

The analogy doesn't apply to HornetQ here. You could make the analogy to JBoss.

JBoss AS still alive.

JBoss Messaging isn't a very creative name... it denotes more a sub-project of JBoss (What JBoss Messaging seemed more like).

From the point of view of "JBoss", this is maybe not a very creative name, but it's a clear name nevertheless ;)

HornetQ can run embedded, standalone... and inside the application server as well.

I understand this. For the end user, this is a blessing and a curse at the same time. It's a blessing since this assures that you apply ace engineering principles by having the "messaging module" of JBoss AS among others completely honor separation of concerns. It means JBoss AS is a modern, modular thing, and not a monolithic beast that the old J2EE servers were. This is obvious a Good Thing.

It's a little bit a curse, since I'm using JBoss AS. Our clients are using JBoss AS. We have a problem with JMS in JBoss AS. Where do we go to? The general JBoss AS forum directs us to "some JMS forum". But what is "some JMS forum"??? Is it ActiveMQ? Is it JBoss Messaging? Is it HornetQ? Why does JBoss AS have 3 messaging forums? Which one is really is concerned with just JMS anyway? Let's try JBoss Messaging... hmmm no response at all. Let's try HornetQ? Hmmpph... we're turned away since it's a JMS question, and JMS is JBoss AS and HornetQ is about general messaging. But what is the difference between "general messaging" and JMS? Let's try ActiveMQ then... still no dice... now I'm directed to a forum about "remoting"??? What is the "remoting" forum???

This can go on and on.

It would be helpful if there was at least a view of the support forums and project pages with JBoss AS as the root and then all projects it consisted of as children. This would undoubtedly upset you guys, since I know you are very proud on your "core API", which is a standalone thing and not related to JBoss AS, but remember this would just be -a- view. You could provide a second view which organized the same projects differently, i.e. such that HornetQ was as far as possible away from JBoss Messaging and JBoss AS :P

You could have this JBoss AS centric view like this:

JBoss AS

    JMS

       ActiveMQ (JBoss AS 4.x)

       JBoss Messaging (JBoss AS 5.x)

       HornetQ (Jboss AS 6)

    JPA

        Hibernate

    ...

Etc.

I respect your opinion though. However the majority seems to have understood the change.

 

Of course I respect your opinions a lot too, and it may make sense to periodically rename the project when some major milestones are reached (maybe Sun should have did this with EJB3 too), but I just wanted to let you guys know it can be confusing too.

Augustienje,

FWIW the HornetQ FAQ has some explanation of the relationship between HornetQ and JBoss Messaging. http://community.jboss.org/wiki/HornetQGeneralFAQs

Also the JBoss Messaging project page has some info on this: http://jboss.org/jbossmessaging

It's also covered in wikipedia:

http://en.wikipedia.org/wiki/JBoss_Messaging

http://en.wikipedia.org/wiki/HornetQ

By "ActiveMQ" I guess you mean JBoss MQ? ActiveMQ is not a JBoss project.

Why does JBoss have 3 messaging forums? Well... because JBoss has 3 different messaging systems, two legacy ones: JBossMQ and JBoss Messaging and one current one HornetQ. The old ones are still there since people still use them.

It's not uncommon, especially with market consolidation for companies to have more than one messaging system. Oracle is a good example, it now has: OracleAQ, WeblogicJMS, OpenMQ (SunMQ).

Progress has SonicMQ and ActiveMQ. Red Hat actually has four messaging systems if you count MRG Messaging.

If you're using JBoss AS, it isn't too hard to find out which messaging system you are using. Once you know that it should be a simple matter to locate the right forum for that system.

It's well documented (we hope) that imq.persist.file.sync.enabled=true is the slowest possible way to configure Open MQ's persistent message storage. Open MQ might compare more favorably, at least with the two persistent message use cases, if the persistent storage options were on a more even footing to the Journal file-system capability of RedHat Linux.

The most recent enhancement along these lines (we called it a "transaction log") was released in December of 2009 (v 4.4 update 1). This feature is controlled via the property: imq.persist.file.newTxnLog.enabled set true.

This new option does have the same resilience characteristics as "sync" does, but it performs substantially better and can be used in any of available deployment modes (cluster or non-cluster), and has no operating environment dependencies.

Furtjher details about this option are available in the product documentation, here.

If you'd like, I'd be happy to work more directly about this, if you'd like.

Of course. It is slower because you need to sync.

On the tests we performed we applied the same rules to every system. (i.e. sync=true).

We were interested in numbers that would be reflected in production systems (not just for benchmark showing), and most enterprise systems will require sync=true.

If the definition of "sync" is: after a restart, the data that was comitted prior to going down, will be recovered after restart ... with no ambiguity -- then you achieve that with the imq.persist.file.newTxnLog.enabled property I mentioned above.

In our tests, we've seen quite a bit of variability between how messaging products behave with various "sync" settings. Since it's not standardized, this isn't surprising.

Sync=true would mean, the data should survive at any sort of crash or restart.

That means.. committed = true, means synced on the disk.

You could have your tx log being synced during shutdown what isn't good enough for the purpose of the benchmark. 

If you guarantee your syncs only at restart, that means you would lose data in case of a crash

For Open MQ, the txn. log is synchronized with each operation. When the message is acknowledged, the data-sync. to disk has been completed. If the message was acknowledged, it will be recovered. Check-pointing the log, is done periodically to keep it from growing without bound (we do not simply checkpoint at start-up or shutdown). Therefore, no loss after restart.

Ok, as you say this is a new option on 4.4. Update1.. and we used 4.4. as the manual states that option wasn't available at the version we used. (4.4 without the Update1)

We were interested in numbers that would be reflected in production systems (not just for benchmark showing), and most enterprise systems will require sync=true.

Unfortunately it is my experience that most customers will not modify the out-of-the-box configuration of their JMS infrastructure.

JBoss Messaging 1.4 has some nice features, but was often let down when operating in environments with unstable connections to the back end database. I hope the journal in HornetQ will improve this significantly. Still, the biggest pain point I found with JBM was its handling of 'full' queues, I am interested to know how this works in HornetQ.

If a JMS queue has reached its max messages limit, and a consumer attempts to enqueue a message. How is this handled in the case of a) a persistent message, and b) a non-persistent message. Does the message get enqueued, and does the client get any notification of what has occured?

I am also interested in how ActiveMQ handles the same scenarios.

This is a topic of great interest to many of our customers who are heavy users of these features in their existing JMS infrastructure, many are in the process of migrating to ActiveMQ and JBoss Messaging.

Cheers,

Matt

http://www.c2b2.co.uk

We always keep the most performant options as default on HornetQ. (without compromising the data, so the out of box is not for benchmark showing only, as most systems unfortunetely will do).

Also: you should look at paging at the documentation. You could maybe then ask specific questions on both hornetQ's user forum.

That will be true probably for Apache's ActiveMQ also. Ask them specific questions on their user's forum. as TSS is not a good place for technical questions.

Should read "from lightweight publish/subscribe messaging to persistent point to point messaging". :)

I think it is the frequency that JBoss change names of project that makes the problem even worse.

Changing project names every so often is a symptom of bad management.

I may be miss it in your benchmark but did you activate this option for the persistence ? (http://hornetq.sourceforge.net/docs/hornetq-2.1.2.Final/user-manual/en/html/persistence.html#installing-aio
)

If no, do you have an idea of what would be the improvement ?

And if no, the inverse question ?

AsyncIO is enabled by default on Linux. And as you can see on the report we used LInux.