Performance tests of open source JMS

Discussions

News: Performance tests of open source JMS

  1. Performance tests of open source JMS (28 messages)

    Matt Brasier, a consultant with niche Java EE consultancy C2B2 Consulting, compares the performance of JBoss Messaging, JBoss MQ and Sun Java System Message Queue, in their out of the box configurations. This first article looks at the performance of these three implementations when sending messages to a JMS queue. The results indicate that the overhead of establishing a connection to a server, and looking up a queue is lowest in JBoss MQ, but that JBoss Messaging is the fastest at actually sending a message. However the time taken to send messages tends towards 1-2ms in all implementations. The testing also highlights the importance of configuring your JMS server to protect your application server from running out of memory. The full article can be read here

    Threaded Messages (28)

  2. It is bad that it does not include ActiveMQ. I've done ActiveMQ vs OpenMQ a year ago, it would be interesting what happened since...
  3. It is bad that it does not include ActiveMQ. I've done ActiveMQ vs OpenMQ a year ago, it would be interesting what happened since...
    What was your conclusion back then? -talip
  4. ActiveMQ[ Go to top ]

    Hi, I plan to look at ActiveMQ deployed in Geronimo when I get some more time. Matt http://www.c2b2.co.uk
  5. Hello All- We published some performance results when we recently released JBoss Messaging 2.0. They're available here http://www.jboss.org/file-access/default/members/jbossmessaging/freezone/docs/userguide-2.0.0.alpha1/html/performance.html#performance.results Just to make clear, I work for JBoss (I'm the JBM project lead).
  6. Benchmarking performance in terms of time/message is not as useful or interesting as measuring the total message throughput per second, as in the JBM tests. I also thought the different configurations shown in the JBM tests are informative (I'm guessing you may be covering different configurations in your 3rd part about tuning?). Last criticism, I really think ActiveMQ belongs in this comparison. Having said all that, I really appreciate you taking the time to do this comparison and writeup. It definitely seems there's a lack of this sort of information out there (although I feel like many of us have written similar benchmarks internally for evaluation purposes). On a side-note to Tim Fox, I can't wait for JBM 2.0 :) Keep up the good work!
  7. wrong title[ Go to top ]

    If you put title as "open source JMS" but did not include "ActiveMQ", it is hard to convince people. :-)
  8. Other factors[ Go to top ]

    I understand the reason for using the vendor's default configuration, but have you considered comparing performance for persistent queues & in memory queues? Also (at a glance) I couldn't find where you published the test message size. Before evaluating other JMS implementations it might be worth expanding your test suite to run against messages of set sizes, and even a "mixed" size test.
  9. Re: Other factors[ Go to top ]

    Hi, I have considered comparing performance of a number of configurations, including; different persistence types, different acknowledge modes, different transaction types, etc. I have not done any of this yet, as this work is something I have been doing between 'paying' customers, and there are a lot of combinations and configurations to consider (such as what DB for DB persistence, should it be local or remote, does this make any difference....?). The message sent was "this is a short message"+n Where n was an incrementing integer. It is configurable as a string in the test harness configuration file, and I initially planned on using some much larger (10k, 100k 1000k) strings too, but ran out of time/got side tracked. I will try and get round to doing this soon. My next aim is to get ActiveMQ tested with the same use cases as the others, after that I will look at speed pulling messages off queues. Matt
  10. SonicMQ[ Go to top ]

    has been very good for our projects both using P2P and PubSub and in both persistent and non-persistent modes. In combination with their CA configuration, I haven't seen anything better.
  11. O.K., you have to start somewhere, but these tests don't tell you much. I’d like to see some of the following questions explored further:  How do these products compare to their commercial rivals like SonicMQ or WebSphereMQ (formerly MQSeries)?  How well do these products balance the workload and provide reliability (e.g. use of Queue Managers to service a named queue, failover, etc.)?  Persistent queue versus in-memory queue performance.  Persistent queue management.  Other management and reliability features (e.g. SonicMQ’s Continuous Availability Architecture)?  Publish/Subscribe performance and features?  Integration (with other messaging middleware) features? I’d also like to see how these products perform when tuned as per vendor recommendations rather than out-of-the-box defaults. Default settings don’t prove much of anything. They can be, for instance, set to use minimal resources or set to support what the vendor thinks is the most common use. Default settings do not tell you what the product is capable of.
  12. William- Unfortunately you can't often publish performance results against proprietary messaging products, since it's often prohibited in their licence agreements.
  13. William-

    Unfortunately you can't often publish performance results against proprietary messaging products, since it's often prohibited in their licence agreements.
    True and dbms and other products as well. This should have a discussion of its own. How is this not a violation of free speech? I wonder if this has ever been challenged in court. I don't expect a small consulting company to take this on. This is in a similar category as patent trolls in that it abuses the court system in order to violate your rights. Any lawyers around here who care to render an opinion on the validity and legality of such a restriction?
  14. William, You are right, it would be interesting to know how these products compare to commercial rivals. However the license agreements of commercial products such as WebSphere and WebLogic explicitly prevent you from publishing performance test or benchmark results. I am more interested in performance, although you are right that it would be useful to have a comparison matrix for features too, maybe if I get enough time I will look into that, however I could spend years just running comparisons in this small area of the Enterprise Java environment. With regards to tuning the configurations, this is one of the main objectives of this work, doing this kind of tuning is bread and butter business for us (C2B2). One of the reasons I tested out of the box configurations is that almost every time I attend a customer site, they have not even considered applying performance tuning to the JMS subsystem. Part of what I hope to eventually show is that the importance of tuning. Matt
  15. Add to that list: Measure performance with - Different ack modes, i.e., AUTO vs DUPS_OK - Transacted message delivery and consumption (JMS transactions) - Distributed transactions involving JMS and non JMS (e.g., DB) resource managers - this will, of course, depend on the performance characteristics of the JTA implementation as well but this is a fairly common scenario in lot of domains.
  16. As far as I know, these all JMS implementations use RDB internally as the messasge store. So, it's very slow, as a message queue service. But "some" proprietary MQ has more performance, and poor recovery capabilities compared with RDB. They don't have a reliable "transaction-log". If you anyway using RDB, why don't you simply use jdbc From the beginning? The programmer who is familiar with JMS is rare. The DB-programmer is existing all around the world, isn't it?
  17. As far as I know, these all JMS implementations use RDB internally as the messasge store.
    So, it's very slow, as a message queue service.
    But "some" proprietary MQ has more performance, and poor recovery capabilities compared with RDB.
    They don't have a reliable "transaction-log".
    FWIW this isn't the case with Apache ActiveMQ which offers a variety of persistence options such as a high performance database and journal without using JDBC. James Iona Open Source Integration
  18. You are right. I wish it should be. But wrong. Let me get this explained. ActiveMQ uses HOWL ( the open source transaction log ). ActiveMQ write messages to a file-base-message-journal, and HOWL writes logical-log, but both don't "careful-write". It gains less number of flush ( i.e. physical disk write or mechanical action of hard disk ), so it gains higher performance, but it has not "ACID" property. It has Only "ACI", the Light Weight Transaction, so called. It is an example of the poor recovery capabilities that I have already mentioned. I will, with all due respect, strongly recommend you to use JDBC-message-store, if you want to use ActiveMQ with reliability.
  19. You are right. I wish it should be. But wrong.
    Let me get this explained.
    ActiveMQ uses HOWL ( the open source transaction log ).
    It never did actually. HOWL was tested out but found to be unsuitable since it's tuned for the small fixed size records that Transaction Managers log. ActiveMQ uses a custom transaction journal implementation.

    ActiveMQ write messages to a file-base-message-journal, and HOWL writes logical-log, but both don't "careful-write".
    It gains less number of flush ( i.e. physical disk write or mechanical action of hard disk ), so it gains higher performance, but it has not "ACID" property.
    It has Only "ACI", the Light Weight Transaction, so called.
    What are you talking about? Garanteed QoS operations are logged in the journal synchronously waiting for the journal to get synced to disk. The journal achieves it's speed ups because it batches together multiple concurrent write operations in a single sync to disk. Get your facts right.
  20. I'm afraid so. I correct one. -> AMQStore write messages to a file-base-message-journal, and AMQTransactionStore writes logical-transaction-log, but both don't "careful-write". But I think my point is still stand. And JBoss's in the same boat, right? And about "careful-write"? OK, If you want to know about this, you may read "Transaction Processing :Concepts and Techniques" (Gray & Reuter '93). As common sense, I'm expecting that all of the programmer who implements transaction system have read this book. But "group-commit" that you mentioned is good idea. I agree with that. Say no more. To be honest, I'm really really waiting for a open-source-JMS that we can use. Deal with it!!
  21. WTF?[ Go to top ]

    Even if all JMS implementions did use RDB internally there's no reason to go an re-implement what JMS gives you for free. Stuff like pub/sub, message listeners and guaranteed once only messaging. Not to mention compatability with other java apps (or even non java apps depending on provider). On top of that the JMS spec is one of Sun's better efforts. Any developer worth their salt should have no trouble picking it up.
  22. Hi Takashi- JBM 1.x uses a relational database for persistence, but JBM 2.0 (still alpha) does not. I don't think it's always true to say there's a trade-off between performance and reliability. The JBM 2.0 transactional log is fully transactional, highly reliable and XA aware, it's is also extremely high performance. We are able to write messages reliably at close to the maximum hardware write throughput of the disk. I'd recommend having a look at the persistent message performance results from the link I posted earlier. (Actually they've got even faster since we released those :) ) We are currently putting it through some rigorous crash and recovery tests to make sure it's also really rugged, so we have the best of both worlds: top class performance and reliability. More info on the log is available on mine and Clebert Suconic's blogs: http://jbossfox.blogspot.com/ http://clebertsuconic.blogspot.com/ Hope that helps :)
  23. About the only test performed with some weight to it was the timing of creating the initial connections. It would appear by the results indicating that the queues filled up with a glut of messages that there was no consumer on the other end. Because JMS messaging is a highly concurrent environment on the JMS provider side (messages in, messages out), I'm not so sure that merely counting how many and how fast you can put messages on to the queue is all that valid if there's not a process competing concurrently to also pull them out. I'm not sure about anyone else, but I've never worked in an environment where messages were put on a queue only to sit there and rot, or eventually time out. I feel that a simple MDB to pull the message off the queue and either throw it away or log its message ID or something would drastically alter the results of your tests. I'd be leery of counting on the results after adding this element though because a whole slew of variables are now introduced, such as the application server used to execute the MDBs, or the number of beans in the pool for example. This now becomes a container test and you can pretty much count on the debate wars over which JEE application server should be used for performance consideration X or feature Y. I agree with the recommendation that messages of varying sizes will also be a worthwhile test, but it might be hard to gauge what it is you're testing. Is it the IO at the JMS provider level or the IO of getting to the provider that is inflating the results? It might take some cautious filtering and a whole lot of math to make that statistic count. As much as I appreciate you taking the time to configure the providers, set up the tests, log and report the results, I'm afraid that this is something similar to the "Mythbusters Syndrome" where tests are conducted and then the experts at-large pipe up with why they are invalid or how the tests should have been conducted to provide more accurate and scientific results. Don't let this dissuade you from continuing on, I look forward to seeing more results.
  24. The main result for people that care about their messages is: When overloaded with messages, JBoss MQ and JBoss messaging suffered failures that brought the whole application server down. OpenMQ, in contrast, prevented the client from sending any more messages once the message queue reached a size limit. This behaviour protected the application server from failure.
  25. The main result for people that care about their messages is:

    When overloaded with messages, JBoss MQ and JBoss messaging suffered failures that brought the whole application server down.
    No the main result is:
    ... but developers usually give little thought to their choice of implementation and accept the default.
    Which on JBossMQ is completely broken in terms of performance and default persistence storage. For those finding this thread: Brief notes on JBossMQ tuning with links to the actual configuration issues. The max messages behaviour is a configuration on the Queue in JBossMQ, e.g. 100000 On a side note, the link posted by Tim also didn't set the JBossMQ MemoryCache sizes. So even non-persistence messages will be written to disk because it thinks heap is getting full once 64M is used. As the saying goes, "benchmarks and posteriors, everybody's got one" :-)
  26. This particular performance test isn't at all a realistic indicator of anything in my opinion. In fact, having worked for a JMS vendor in the past, out of the box configurations tend to favour raw tests such as these and are of little or no value to real world applications. Sending messages to a destination as fast as possible for an out of the box configuration tells me nothing about how the JMS server acknowledges and persists those messages into its store (e.g. if it is synchronous to disk that will have a huge impact on performance). In the past, I've seen vendors go as far as changing the default behaviour of JMS to not persist by default in order to appear favourable out of the box and of course others follow suit to remain competitive. The timings would have been more useful if the test was repeated to not include the time it takes to lookup the factory and create the connection to validate some of the conclusions that have been made. It is likely that JMS products will be used alongside a framework which would manage connection pooling, again rendering these results fairly meaningless since the overhead of retrieving the connection would be less significant. I could go on and on about the problems in the test itself but would like to point out specifically the point Matt raises about users accepting the default JMS bundled in their application server. Using a resource adapter makes it easier to use a different JMS provider than your application server provides. Users of GlassFish & Open MQ have a number of frameworks (i.e. resource adapters) available to them for using Open MQ or another JMS provider. I would personally lean towards the sun-jms-adapter (aka JMSJCA) which is a feature-rich, standard adapter used for integrating JMS into Sun's open source ESB, Open ESB. If you download Open ESB you will also get a GlassFish / Open MQ bundle which has JMSJCA pre-installed: https://open-esb.dev.java.net/ Read more about the sun-jms-adapter / JMSJCA and how to use it with other JMS providers here: https://jmsjca.dev.java.net/
  27. Most application servers provide a way to plug in a different JMS implementation, what this series of tests is supposed to be doing is demonstrating that developers may want to think about the JMS implementation that they use in the same way that they think about the database that they use. These tests tell you how fast I could load messages onto a queue, using the out of the box configuration, in my environment. Future tests will look at different configurations, and how performance and reliability are traded off against each other, which is the ultimate objective of the JMS testing exercise I am doing. I chose these tests to start with as I wanted to test JMS in the way that most developers seem to use it. Most Java EE developers, while they are familiar with JMS APIs, have no knowledge or desire to change how the back end works (this is the same with most Java EE technologies, developers concern themselves with solving business problems). While JMS aught to be used in a way that keeps a connection/session open and sends multiple messages through it, I have often found that it isn't (for example, in Springs JMS template). That is why I have tried to highlight the relatively expensive overhead involved in opening and closing the connections. As more messages are sent down each connection, the overhead in establishing the connection tends to 0. I can't test every combination, and can't tell people which JMS implementation to use. What I hope to tell people is that they need to think about which implementation they use, and they need to tune that implementation to achieve the desired behaviour. Matt
  28. One thing I think is missing is the reliability factor. We’ve used JBossMQ for a number of years to handle emails in a reliable way, but periodically JBossMQ will choke and stop handling messages. I’m currently using ActiveMQ which has proven to be much more reliable. We’ve tried switching from default hypersonic store to Oracle. The bottom line is ActiveMQ doesn’t run into these types of issues… org.jboss.mq.SpyJMSException: Could not store message: 575 msg=574 hard NOT_STORED PERSISTENT queue=QUEUE.email priority=4 lateClone=false hashCode=22796972; - nested throwable: (java.sql.SQLException: File input/output errorerror java.io.IOException closing file - file C:\jboss-4.0.5.GA\server\default\data\hypersonic\localDB.data in statement [INSERT INTO JMS_MESSAGES (MESSAGEID, DESTINATION, MESSAGEBLOB, TXID, TXOP) VALUES(?,?,?,?,?)]) org.jboss.mq.SpyJMSException: Could not store message: 483 msg=260 hard NOT_STORED PERSISTENT queue=QUEUE.email priority=4 lateClone=false hashCode=1886808; - nested throwable: (java.sql.SQLException: ORA-01483: invalid length for DATE or NUMBER bind variable )
  29. Hi There, Check your error message. There is some problem with the way your parm that is holding DATE or NUMBER. snippet is here... org.jboss.mq.SpyJMSException: Could not store message: 483 msg=260 hard NOT_STORED PERSISTENT queue=QUEUE.email priority=4 lateClone=false hashCode=1886808; - nested throwable: (java.sql.SQLException: ORA-01483: invalid length for DATE or NUMBER bind variable check your attributes in your JMS message, their types and length. and make changes and it should fix your problem. I had similar issue, where I used, NUMBER(3,0) and the actual NUMBER value I was sending is larger than(3,0) I chnaged it to NUMBER(5,0). similarly if it is your DATE field check what you are actually sending.. I hope it may help. obviosly there is some problems with JBOssMQueues I myself looking for solution. my error begins like yours but it is different, Caused by: java.sql.SQLException: ORA-01460: unimplemented or unreasonable conversion requested 00:16:35,014 ERROR [STDERR] at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:134) 00:16:35,014 ERROR [STDERR] at oracle.jdbc.ttc7.TTIoer.processError(TTIoer.java:289) 00:16:35,014 ERROR [STDERR] at oracle.jdbc.ttc7.Oall7.receive(Oall7.java:573) 00:16:35,014 ERROR [STDERR] at oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.java:1891) 00:16:35,014 ERROR [STDERR] at oracle.jdbc.ttc7.TTC7Protocol.parseExecuteFetch(TTC7Protocol.java:1093 I am still looking for how to fix ?