|
Sponsored Links
Resources
Enterprise Java
Research Library
Get Java white papers, product information, case studies and webcasts
|
News
News
News
|
Messages: 28
Messages: 28
Messages: 28
Printer friendly
Printer friendly
Printer friendly
Post reply
Post reply
Post reply
XML
XML
XML
|
 |
Performance tests of open source JMS
Matt Brasier, a consultant with niche Java EE consultancy C2B2 Consulting, compares the performance of JBoss Messaging, JBoss MQ and Sun Java System Message Queue, in their out of the box configurations. This first article looks at the performance of these three implementations when sending messages to a JMS queue. The results indicate that the overhead of establishing a connection to a server, and looking up a queue is lowest in JBoss MQ, but that JBoss Messaging is the fastest at actually sending a message. However the time taken to send messages tends towards 1-2ms in all implementations. The testing also highlights the importance of configuring your JMS server to protect your application server from running out of memory.
The full article can be read here
|
|
|
Message #263987
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: Performance tests of open source JMS
It is bad that it does not include ActiveMQ. I've done ActiveMQ vs OpenMQ a year ago, it would be interesting what happened since...
|
|
|
Message #264018
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: Performance tests of open source JMS
It is bad that it does not include ActiveMQ. I've done ActiveMQ vs OpenMQ a year ago, it would be interesting what happened since...
What was your conclusion back then?
-talip
|
|
|
Message #264089
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Other factors
I understand the reason for using the vendor's default configuration, but have you considered comparing performance for persistent queues & in memory queues?
Also (at a glance) I couldn't find where you published the test message size. Before evaluating other JMS implementations it might be worth expanding your test suite to run against messages of set sizes, and even a "mixed" size test.
|
|
|
Message #264102
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: Performance tests of open source JMS
O.K., you have to start somewhere, but these tests don't tell you much. I’d like to see some of the following questions explored further:
How do these products compare to their commercial rivals like SonicMQ or WebSphereMQ (formerly MQSeries)?
How well do these products balance the workload and provide reliability (e.g. use of Queue Managers to service a named queue, failover, etc.)?
Persistent queue versus in-memory queue performance.
Persistent queue management.
Other management and reliability features (e.g. SonicMQ’s Continuous Availability Architecture)?
Publish/Subscribe performance and features?
Integration (with other messaging middleware) features?
I’d also like to see how these products perform when tuned as per vendor recommendations rather than out-of-the-box defaults. Default settings don’t prove much of anything. They can be, for instance, set to use minimal resources or set to support what the vendor thinks is the most common use. Default settings do not tell you what the product is capable of.
|
|
|
Message #264103
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: Performance tests of open source JMS
William-
Unfortunately you can't often publish performance results against proprietary messaging products, since it's often prohibited in their licence agreements.
|
|
|
Message #264105
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: Performance tests of open source JMS
William,
You are right, it would be interesting to know how these products compare to commercial rivals. However the license agreements of commercial products such as WebSphere and WebLogic explicitly prevent you from publishing performance test or benchmark results.
I am more interested in performance, although you are right that it would be useful to have a comparison matrix for features too, maybe if I get enough time I will look into that, however I could spend years just running comparisons in this small area of the Enterprise Java environment.
With regards to tuning the configurations, this is one of the main objectives of this work, doing this kind of tuning is bread and butter business for us (C2B2). One of the reasons I tested out of the box configurations is that almost every time I attend a customer site, they have not even considered applying performance tuning to the JMS subsystem. Part of what I hope to eventually show is that the importance of tuning.
Matt
|
|
|
Message #264106
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: Other factors
Hi,
I have considered comparing performance of a number of configurations, including; different persistence types, different acknowledge modes, different transaction types, etc. I have not done any of this yet, as this work is something I have been doing between 'paying' customers, and there are a lot of combinations and configurations to consider (such as what DB for DB persistence, should it be local or remote, does this make any difference....?).
The message sent was
"this is a short message"+n
Where n was an incrementing integer. It is configurable as a string in the test harness configuration file, and I initially planned on using some much larger (10k, 100k 1000k) strings too, but ran out of time/got side tracked. I will try and get round to doing this soon.
My next aim is to get ActiveMQ tested with the same use cases as the others, after that I will look at speed pulling messages off queues.
Matt
|
|
|
Message #264110
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
SonicMQ
has been very good for our projects both using P2P and PubSub and in both persistent and non-persistent modes. In combination with their CA configuration, I haven't seen anything better.
|
|
|
Message #264115
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: Performance tests of open source JMS
Add to that list:
Measure performance with
- Different ack modes, i.e., AUTO vs DUPS_OK
- Transacted message delivery and consumption (JMS transactions)
- Distributed transactions involving JMS and non JMS (e.g., DB) resource managers - this will, of course, depend on the performance characteristics of the JTA implementation as well but this is a fairly common scenario in lot of domains.
|
|
|
Message #264116
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
RDB is popular and reliable, JMS not.
As far as I know, these all JMS implementations use RDB internally as the messasge store.
So, it's very slow, as a message queue service.
But "some" proprietary MQ has more performance, and poor recovery capabilities compared with RDB.
They don't have a reliable "transaction-log".
If you anyway using RDB, why don't you simply use jdbc From the beginning?
The programmer who is familiar with JMS is rare.
The DB-programmer is existing all around the world, isn't it?
|
|
|
Message #264120
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: RDB is popular and reliable, JMS not.
As far as I know, these all JMS implementations use RDB internally as the messasge store.
So, it's very slow, as a message queue service.
But "some" proprietary MQ has more performance, and poor recovery capabilities compared with RDB.
They don't have a reliable "transaction-log".
FWIW this isn't the case with Apache ActiveMQ which offers a variety of persistence options such as a high performance database and journal without using JDBC.
James
Iona
Open Source Integration
|
|
|
Message #264123
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: Performance tests of open source JMS
William-
Unfortunately you can't often publish performance results against proprietary messaging products, since it's often prohibited in their licence agreements.
True and dbms and other products as well. This should have a discussion of its own. How is this not a violation of free speech? I wonder if this has ever been challenged in court. I don't expect a small consulting company to take this on. This is in a similar category as patent trolls in that it abuses the court system in order to violate your rights.
Any lawyers around here who care to render an opinion on the validity and legality of such a restriction?
|
|
|
Message #264125
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
WTF?
Even if all JMS implementions did use RDB internally there's no reason to go an re-implement what JMS gives you for free. Stuff like pub/sub, message listeners and guaranteed once only messaging. Not to mention compatability with other java apps (or even non java apps depending on provider).
On top of that the JMS spec is one of Sun's better efforts. Any developer worth their salt should have no trouble picking it up.
|
|
|
Message #264127
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: Performance tests of open source JMS
Benchmarking performance in terms of time/message is not as useful or interesting as measuring the total message throughput per second, as in the JBM tests. I also thought the different configurations shown in the JBM tests are informative (I'm guessing you may be covering different configurations in your 3rd part about tuning?). Last criticism, I really think ActiveMQ belongs in this comparison.
Having said all that, I really appreciate you taking the time to do this comparison and writeup. It definitely seems there's a lack of this sort of information out there (although I feel like many of us have written similar benchmarks internally for evaluation purposes).
On a side-note to Tim Fox, I can't wait for JBM 2.0 :) Keep up the good work!
|
|
|
Message #264136
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: RDB is popular and reliable, JMS not.
You are right. I wish it should be. But wrong.
Let me get this explained.
ActiveMQ uses HOWL ( the open source transaction log ).
ActiveMQ write messages to a file-base-message-journal, and HOWL writes logical-log, but both don't "careful-write".
It gains less number of flush ( i.e. physical disk write or mechanical action of hard disk ), so it gains higher performance, but it has not "ACID" property.
It has Only "ACI", the Light Weight Transaction, so called.
It is an example of the poor recovery capabilities that I have already mentioned.
I will, with all due respect, strongly recommend you to use JDBC-message-store, if you want to use ActiveMQ with reliability.
|
|
|
Message #264154
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: RDB is popular and reliable, JMS not.
Hi Takashi-
JBM 1.x uses a relational database for persistence, but JBM 2.0 (still alpha) does not.
I don't think it's always true to say there's a trade-off between performance and reliability.
The JBM 2.0 transactional log is fully transactional, highly reliable and XA aware, it's is also extremely high performance. We are able to write messages reliably at close to the maximum hardware write throughput of the disk. I'd recommend having a look at the persistent message performance results from the link I posted earlier. (Actually they've got even faster since we released those :) )
We are currently putting it through some rigorous crash and recovery tests to make sure it's also really rugged, so we have the best of both worlds: top class performance and reliability.
More info on the log is available on mine and Clebert Suconic's blogs:
http://jbossfox.blogspot.com/
http://clebertsuconic.blogspot.com/
Hope that helps :)
|
|
|
Message #264156
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Close, but what are you trying to show?
About the only test performed with some weight to it was the timing of creating the initial connections.
It would appear by the results indicating that the queues filled up with a glut of messages that there was no consumer on the other end. Because JMS messaging is a highly concurrent environment on the JMS provider side (messages in, messages out), I'm not so sure that merely counting how many and how fast you can put messages on to the queue is all that valid if there's not a process competing concurrently to also pull them out. I'm not sure about anyone else, but I've never worked in an environment where messages were put on a queue only to sit there and rot, or eventually time out.
I feel that a simple MDB to pull the message off the queue and either throw it away or log its message ID or something would drastically alter the results of your tests. I'd be leery of counting on the results after adding this element though because a whole slew of variables are now introduced, such as the application server used to execute the MDBs, or the number of beans in the pool for example. This now becomes a container test and you can pretty much count on the debate wars over which JEE application server should be used for performance consideration X or feature Y.
I agree with the recommendation that messages of varying sizes will also be a worthwhile test, but it might be hard to gauge what it is you're testing. Is it the IO at the JMS provider level or the IO of getting to the provider that is inflating the results? It might take some cautious filtering and a whole lot of math to make that statistic count.
As much as I appreciate you taking the time to configure the providers, set up the tests, log and report the results, I'm afraid that this is something similar to the "Mythbusters Syndrome" where tests are conducted and then the experts at-large pipe up with why they are invalid or how the tests should have been conducted to provide more accurate and scientific results. Don't let this dissuade you from continuing on, I look forward to seeing more results.
|
|
|
Message #264157
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: Performance tests of open source JMS
The main result for people that care about their messages is:
When overloaded with messages, JBoss MQ and JBoss messaging suffered failures that brought the whole application server down. OpenMQ, in contrast, prevented the client from sending any more messages once the message queue reached a size limit. This behaviour protected the application server from failure.
|
|
|
Message #264159
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: RDB is popular and reliable, JMS not.
You are right. I wish it should be. But wrong.
Let me get this explained.
ActiveMQ uses HOWL ( the open source transaction log ).
It never did actually. HOWL was tested out but found to be unsuitable since it's tuned for the small fixed size records that Transaction Managers log. ActiveMQ uses a custom transaction journal implementation.
ActiveMQ write messages to a file-base-message-journal, and HOWL writes logical-log, but both don't "careful-write".
It gains less number of flush ( i.e. physical disk write or mechanical action of hard disk ), so it gains higher performance, but it has not "ACID" property.
It has Only "ACI", the Light Weight Transaction, so called.
What are you talking about? Garanteed QoS operations are logged in the journal synchronously waiting for the journal to get synced to disk. The journal achieves it's speed ups because it batches together multiple concurrent write operations in a single sync to disk. Get your facts right.
|
|
|
Message #264161
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: Performance tests of open source JMS
The main result for people that care about their messages is:
When overloaded with messages, JBoss MQ and JBoss messaging suffered failures that brought the whole application server down.
No the main result is:
... but developers usually give little thought to their choice of implementation and accept the default.
Which on JBossMQ is completely broken in terms of performance and default persistence storage.
For those finding this thread:
Brief notes on JBossMQ tuning with links to the
actual configuration issues.
The max messages behaviour is a configuration on
the Queue in JBossMQ, e.g.
<attribute name="MaxDepth">100000</attribute>
On a side note, the link posted by Tim
also didn't set the JBossMQ MemoryCache sizes.
So even non-persistence messages will be written to disk
because it thinks heap is getting full once 64M is used.
As the saying goes,
"benchmarks and posteriors, everybody's got one" :-)
|
|
|
Message #264583
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: RDB is popular and reliable, JMS not.
I'm afraid so.
I correct one.
-> AMQStore write messages to a file-base-message-journal, and AMQTransactionStore writes logical-transaction-log, but both don't "careful-write".
But I think my point is still stand.
And JBoss's in the same boat, right?
And about "careful-write"?
OK, If you want to know about this, you may read "Transaction Processing :Concepts and Techniques" (Gray & Reuter '93).
As common sense, I'm expecting that all of the programmer who implements transaction system have read this book.
But "group-commit" that you mentioned is good idea.
I agree with that.
Say no more.
To be honest, I'm really really waiting for a open-source-JMS that we can use.
Deal with it!!
|
|
|
Message #264717
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: Performance tests of open source JMS
This particular performance test isn't at all a realistic indicator of anything in my opinion. In fact, having worked for a JMS vendor in the past, out of the box configurations tend to favour raw tests such as these and are of little or no value to real world applications.
Sending messages to a destination as fast as possible for an out of the box configuration tells me nothing about how the JMS server acknowledges and persists those messages into its store (e.g. if it is synchronous to disk that will have a huge impact on performance). In the past, I've seen vendors go as far as changing the default behaviour of JMS to not persist by default in order to appear favourable out of the box and of course others follow suit to remain competitive.
The timings would have been more useful if the test was repeated to not include the time it takes to lookup the factory and create the connection to validate some of the conclusions that have been made. It is likely that JMS products will be used alongside a framework which would manage connection pooling, again rendering these results fairly meaningless since the overhead of retrieving the connection would be less significant.
I could go on and on about the problems in the test itself but would like to point out specifically the point Matt raises about users accepting the default JMS bundled in their application server. Using a resource adapter makes it easier to use a different JMS provider than your application server provides.
Users of GlassFish & Open MQ have a number of frameworks (i.e. resource adapters) available to them for using Open MQ or another JMS provider. I would personally lean towards the sun-jms-adapter (aka JMSJCA) which is a feature-rich, standard adapter used for integrating JMS into Sun's open source ESB, Open ESB.
If you download Open ESB you will also get a GlassFish / Open MQ bundle which has JMSJCA pre-installed:
https://open-esb.dev.java.net/
Read more about the sun-jms-adapter / JMSJCA and how to use it with other JMS providers here:
https://jmsjca.dev.java.net/
|
|
|
Message #264727
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: Performance tests of open source JMS
Most application servers provide a way to plug in a different JMS implementation, what this series of tests is supposed to be doing is demonstrating that developers may want to think about the JMS implementation that they use in the same way that they think about the database that they use.
These tests tell you how fast I could load messages onto a queue, using the out of the box configuration, in my environment. Future tests will look at different configurations, and how performance and reliability are traded off against each other, which is the ultimate objective of the JMS testing exercise I am doing. I chose these tests to start with as I wanted to test JMS in the way that most developers seem to use it. Most Java EE developers, while they are familiar with JMS APIs, have no knowledge or desire to change how the back end works (this is the same with most Java EE technologies, developers concern themselves with solving business problems).
While JMS aught to be used in a way that keeps a connection/session open and sends multiple messages through it, I have often found that it isn't (for example, in Springs JMS template). That is why I have tried to highlight the relatively expensive overhead involved in opening and closing the connections. As more messages are sent down each connection, the overhead in establishing the connection tends to 0.
I can't test every combination, and can't tell people which JMS implementation to use. What I hope to tell people is that they need to think about which implementation they use, and they need to tune that implementation to achieve the desired behaviour.
Matt
|
|
|
Message #264728
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: Performance tests of open source JMS
One thing I think is missing is the reliability factor. We’ve used JBossMQ for a number of years to handle emails in a reliable way, but periodically JBossMQ will choke and stop handling messages. I’m currently using ActiveMQ which has proven to be much more reliable. We’ve tried switching from default hypersonic store to Oracle. The bottom line is ActiveMQ doesn’t run into these types of issues…
org.jboss.mq.SpyJMSException: Could not store message: 575 msg=574 hard NOT_STORED PERSISTENT queue=QUEUE.email priority=4 lateClone=false hashCode=22796972; - nested throwable: (java.sql.SQLException: File input/output errorerror java.io.IOException closing file - file C:\jboss-4.0.5.GA\server\default\data\hypersonic\localDB.data in statement [INSERT INTO JMS_MESSAGES (MESSAGEID, DESTINATION, MESSAGEBLOB, TXID, TXOP) VALUES(?,?,?,?,?)])
org.jboss.mq.SpyJMSException: Could not store message: 483 msg=260 hard NOT_STORED PERSISTENT queue=QUEUE.email priority=4 lateClone=false hashCode=1886808; - nested throwable: (java.sql.SQLException: ORA-01483: invalid length for DATE or NUMBER bind variable
)
|
|
|
Message #265029
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
wrong title
If you put title as "open source JMS" but did not include "ActiveMQ", it is hard to convince people. :-)
|
|
|
Message #307680
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: Performance tests of open source JMS
Hi There,
Check your error message. There is some problem with the way your parm that is holding DATE or NUMBER.
snippet is here...
org.jboss.mq.SpyJMSException: Could not store message: 483 msg=260 hard NOT_STORED PERSISTENT queue=QUEUE.email priority=4 lateClone=false hashCode=1886808; - nested throwable: (java.sql.SQLException: ORA-01483: invalid length for DATE or NUMBER bind variable
check your attributes in your JMS message, their types and length. and make changes and it should fix your problem.
I had similar issue, where I used, NUMBER(3,0) and the actual NUMBER value I was sending is larger than(3,0)
I chnaged it to NUMBER(5,0). similarly if it is your DATE field check what you are actually sending.. I hope it may help. obviosly there is some problems with JBOssMQueues I myself looking for solution. my error begins like yours but it is different,
Caused by: java.sql.SQLException: ORA-01460: unimplemented or unreasonable conversion requested
00:16:35,014 ERROR [STDERR] at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:134)
00:16:35,014 ERROR [STDERR] at oracle.jdbc.ttc7.TTIoer.processError(TTIoer.java:289)
00:16:35,014 ERROR [STDERR] at oracle.jdbc.ttc7.Oall7.receive(Oall7.java:573)
00:16:35,014 ERROR [STDERR] at oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.java:1891)
00:16:35,014 ERROR [STDERR] at oracle.jdbc.ttc7.TTC7Protocol.parseExecuteFetch(TTC7Protocol.java:1093 I am still looking for how to fix ?
|
|
|
| New content on TheServerSide.comNew content on TheServerSide.comNew content on TheServerSide.com |
|
|
Reza Rahman explores the features of the proposed JSR 299, Contexts and Dependency Injection for Java EE (CDI). When approved, it promises to be a key feature of Java EE 6.
(November 2, Article)
SAML is an XML-based standard for exchanging authentication and authorization data between security domains. The single most important problem that SAML was created to solve is the Web browser Single Sign-On problem. Many organizations are debating whether to stay with version 1.1 or move to 2.0. This article makes observations about both options.
(September 28, Article)
Joe Ottinger takes a look at how people learn, and applies it to the practice of programming. He notes that understanding how people learn is an essential part of working in a programming team.
(September 22, Article)
Stephen Maryka gave us an article about the Asynchronous Web and posed a number of questions that get examined like an approach to delivering Asynchronous Web capabilities through extensions to existing Java EE technologies.
(July 14, Article)
JavaServer Faces Flex goal is to provide users capability in creating standard Flex components, part of flexSDK which is open sourced through MPL license, as normal JSF components. This article by Ji Hoon Kim will provide an overview of creating a simple multilingual JSF page consisting of JSF Flex tags.
(June 29, Article)
In this session Jeff explores the key characteristics of successful SOA projects. He covers some of the patterns, and anti-patterns, tool sets, and strategies that he himself learned the hard way. Last, he provides a strategy and blueprint for achieving a high likelihood of success in your SOA project.
(June 23, Tech Talk)
Ari Zilka, CTO of Terracotta, Inc., talks about the new features in Terracotta 3.1, announced during JavaOne and available now.
(June 15, Tech Talk)
In this Tech Talk, Josh Long explores an integration challenge using Spring Integration and walks through the implementation, employing and expanding on the basic patterns of Enterprise Application Integration to tie together components into a function integration solution, and then demonstrates how Spring Integration helps address the integration requirements.
(June 15, Tech Talk)
In this Tech Talk, David Geary teaches you: The basics of Google Web Toolkit; How to implement Ajax-enabled applications in Java; Internationalization; Hooking into the browser history mechanism; Remote procedure calls.
(June 4, Tech Talk)
Jon Kern discusses the best architecture/technical solutions and ensure that they are repeated by all developers. By tackling the architecture up-front in a serial manner, subsequent parallel development will be much more manageable and predictable.
(May 28, Tech Talk)
This keynote describes the frustrations of modern knowledge workers in their quest to actually get some work done, and solutions for how to guard yourself against all those distractions. Neal Ford talks about environments, coding, acceleration, automation, and avoiding repetition as ways to defeat the misguided attempts to sap your ability to produce good work.
(May 26, Tech Talk)
Gil demonstrates how new, aggressive uses of already abundant compute capacity by common applications offer competitive value for application designers.
(May 21, Tech Talk)
Chris Keene introduces WaveMaker as a new way to automate the ability to generate Hibernate classes in order to more quickly bring OR mapping into an application.
(May 19, Article)
In this session Nati Shalom demonstrates how to take a standard Java EE web application and scale it out or down dynamically without changes to the application code. Seeing as most web applications are over-provisioned to meet infrequent peak loads, this is a dramatic change because it enables growing your application as needed, when needed, without paying for unutilized resources.
(May 19, Tech Talk)
Mastering EJB was one of the original and most influential EJB books in the industry. Mastering EJB III now returns with two new expert co-authors, updated for EJB 2.1 and 30% new chapters including security, integration, best practices, open source, and more.
(Book PDF Download)
The Application Server Matrix is a detailed listing of J2EE vendors and their application server products, with information on latest version numbers, J2EE spec support and licensing, pricing, platform support, and links to product downloads and reviews.
(Application Server Comparison Matrix)
|
|
