We are mapping out strategies for designing a large J2EE application. This application has two categories of use-cases. One category handles use-cases where clients make requests on the server. The server executes the request and returns a response. The other category is monitoring. This provides live monitoring of activity within the server.
Thinking about the latter category I immediately think "event bus" or "subject/observer". There are a number of articles about using these types of pattern for notification and decoupling within J2EE. However, I haven't found any that have solutions that work in a clustered/load balanced environment. Has anyone practical experience/ideas relating to this? I am considering using MOM via JMS but I'm concerned that it maybe a sledge hammer to crack a nut. Performance is a major issue, we expect to handle between 500,000,000 and 1000,000,000 requests a day.
Also, our operations have a transactional and non-transactional part. We were considering executing the transactional part of the request and then delegating to the non-transactional part to a "in-container" RMI server. The calls would be async (active object pattern). This would allow the server to return back to the client when the transactional work is done but before the "optional" work is complete. We anticipate this would improve our throughput. Has anyone used any techniques like this (or similar).
Can I just clarify something.
0.5 - 1 BILLION requests a day?
I may be mistaken, but to be honest I wouldn't be doing this in Java if you want any chance of doing that kind of volume!
That's a sustained rate of 11,574 requests per second, if it is perfectly evenly distributed over the 24 hour period, which I suspect it won't be.
I don't think JMS would be a sledgehammer to crack a nut here, It sounds like what you are trying to crack is somewhat larger than a nut! Or did I mis-read your numbers?
The only way I can see that working out is if your operations are such that they can be pretty much perfectly paralelled. In other words you just keep adding servers because the requests are totally unrelated to each other.
JMS would work quite well for what you want, assuming that "live monitoring" means "everything our JMS consumers have already pulled off the queues and processed.
I still think the numbers you mention sound incredibly high for something written in Java (And I love Java!) :-)
Sorry - put the wrong numbers through calctool! We expect a sustained rate of between 500 and 2000 requests per second. By "sledgehammer to crack a nut" I'm referring to using JMS for event distribution in a tightly coupled app (rather than as loosly coupled system).
That sounds a bit more like it, I was going to ask you what color the sky was on your planet! :-)
I still think JMS is the right tool here. For the following reasons / assumptions.
1) You do not want to lose observations (events.) JMS can provide persistent queues / topics to ensure delivery.
2) 2000 requests a second is still a high throughput (150 million a day.) so you need something pretty scaleable to do that.
3) You want to distribute events in an asynchronous manner. Exactly what JMS was designed for.
The main question is, which JMS implementation to use. In my experience.....
WebLogic 5.1.0: Not very good. Supposed to be much better in 6.0 though.
MQ Series: It's IBM, it works, it's quick, but it's a pig to configure.
FioranoMQ: Superb product. The only one I've seen that uses the file system to persist the queues, which is way faster than the implementations that use a database to do it. They have published documents on how to integrate it with WebLogic 6.0.
On a project I worked on we tested Fiorano using a simple test harness. This had a writer and reader thread, in the same VM, on a 256MB Pentium III 700MHZ, using a persistent queue, Java1.2. We didn't do anything with the messages we pulled off, we just threw them away. OK, that's the scenario, so I don't mislead you with the numbers.
2,000 messages per second was easy, the CPU wasn't even breaking sweat. These were JMSObjectMessages, using org.w3c.dom.Element objects, which contained reasonably long XML documents (in other words, they were non trivial to serialize in CPU terms.) We easily hit levels of 5,000 messages a second. Remember though, we weren't doing anything with them, just sucking them off the queue while something else wrote to it.
In the production world, we found that under extremely heavy load on the server (4GB of memory, 4 500MHZ Sparc CPUs, all fully utilized) the JVM for fiorano was using 14MB of memory and next to no CPU time. The only quirk we found was that when the worker applications got bogged down and the queues built up to around the 1 million message mark, then it got a bit upset with life. However, you have more important application issues when that happens IMHO! :-)
I would say that your events will probably be much simpler to pass around and that the throughput should easily be sustainable at a much higher level.
Fiorano have made performance improvements since we ran this test over a year ago. It is also very very VERY simple to set up and use. The admin tool is a bit poor (it could really do with a web interface) but it's easy enough to use and in general, it just does exactly what it says on the tin.
Hope that helps