Discussions

General J2EE: Controlling a cluster from an admin console

  1. Controlling a cluster from an admin console (3 messages)

    I keep running into the same problem, and I haven't seen a pattern that really looks right to me for how to solve it.

    I have a cluster of managed servers, and an admin console webapp. I make changes to the configuration in the admin console, and I want to push out those changes to the managed servers. This could mean that I've changed something in the database, and I want them to reload, or it could mean I've put a new value in for a known JNDI path. Or something else entirely: the upshot is, I want each instance in the cluster to do something on command.

    The first time I needed to do this (quite a long time ago now), I had each managed server periodically touch a row in the database, and the admin server would make an rmi call to each "live" admin server, telling it to reload itself from the database. This worked just fine, and it was easy to tell whether or not things had succeeded, since each RMI call was synchronous and had a return value I could present to the adminstrator.

    I could imagine JMS being useful for this, but it just seems like a nightmare to setup and administer. Perhaps I'm prematurely jumping to conclusions here. Certainly having each managed server subscribe to a topic, and having the admin console publish to it sounds good. JMS just seems like a very heavy-weight solution to configure and administer. Making it "reliable" seems odd, and since I really do want to hear back a "success or fail" message, publish/subscribe seems a little bit like a misfit for this problem.

    JMX also seems like a possibility. My console should be able to get ahold of an MBean on each managed server, and call its API. But, how does the console know which managed servers exist? I'd prefer to have the discovery "just work" without needing to tell it which managed servers are up or not, and I don't quite see how to do that with JMX. Maybe there is an MBean on the admin server, and each managed server finds it when it comes up, and invokes a heartbeat method on it?

    In another app recently I was using a replicated cache, and I had each managed server scribble its identity into the cache periodically, which made it easy to detect who was still around and who wasn't, and the cache provided a nice "message bus", but the whole thing felt like a bit of a hack.

    Recently I've been thinking about building a sort of "lite messaging" system for inter-cluster communication built on top of jgroups. But that just feels like reinventing the wheel.

    I'd love to know how other people deal with this kind of thing.

    Thanks in advance,
    Eric
  2. Hi Eric,
    In another app recently I was using a replicated cache, and I had each managed server scribble its identity into the cache periodically, which made it easy to detect who was still around and who wasn't, and the cache provided a nice "message bus", but the whole thing felt like a bit of a hack.

    You should check out Coherence 3.0 and the upcoming Coherence 3.1. Coherence 3.0 introduced fully clustered JMX support so that you can manage and monitor all the Coherence MBeans from anywhere in the cluster. In Coherence 3.1 we will be providing you the ability to add your own application MBeans and manage them across the cluster.

    Later,
    Rob Misek
    Tangosol, Inc.
    Coherence: Fully clustered JMX for all MBeans.
  3. Firstly I'd say that management of servers should probably be done with JMX; making MBeans for all the things you wish to make configurable/controllable/notifiable.
    I keep running into the same problem, and I haven't seen a pattern that really looks right to me for how to solve it.I have a cluster of managed servers, and an admin console webapp. I make changes to the configuration in the admin console, and I want to push out those changes to the managed servers. This could mean that I've changed something in the database, and I want them to reload, or it could mean I've put a new value in for a known JNDI path. Or something else entirely: the upshot is, I want each instance in the cluster to do something on command.The first time I needed to do this (quite a long time ago now), I had each managed server periodically touch a row in the database, and the admin server would make an rmi call to each "live" admin server, telling it to reload itself from the database. This worked just fine, and it was easy to tell whether or not things had succeeded, since each RMI call was synchronous and had a return value I could present to the adminstrator.I could imagine JMS being useful for this, but it just seems like a nightmare to setup and administer.

    JMS should be no more of a nightmare to setup and administer than something like a RMI naming service etc. e.g. with ActiveMQ you can use multicast discovery so you just run a broker somewhere and start JMS clients and you're all done. No other configuration required.

    Or if you don't want to use multicast its just a matter of running one or two brokers somewhere and configuring one connection String in the ActiveMQConnectionFactory to list the URLs to try and connect by, or to use a JNDI configuration file if you want to use JNDI.

    Though I confess some JMS providers can be a bit painful to setup (requiring the use of an admin tool to add each queue/topic you want to use along with a tool to expose stuff in JNDI). But thats more an issue with some providers and not a general fault with JMS.

    Perhaps I'm prematurely jumping to conclusions here. Certainly having each managed server subscribe to a topic, and having the admin console publish to it sounds good. JMS just seems like a very heavy-weight solution to configure and administer. Making it "reliable" seems odd, and since I really do want to hear back a "success or fail" message, publish/subscribe seems a little bit like a misfit for this problem.

    This is an ideal fit for JMS. The thing you're maybe missing is if you want to know when a consumer in pub/sub has processed a message is they send a message back to the publisher using the Message.getJMSReplyTo() destination.

    The publisher then creates a temporary queue and calls the setJMSReplyTo() method before sending the message.

    JMX also seems like a possibility. My console should be able to get ahold of an MBean on each managed server, and call its API. But, how does the console know which managed servers exist? I'd prefer to have the discovery "just work" without needing to tell it which managed servers are up or not, and I don't quite see how to do that with JMX.

    I've wanted something like this for a while. Basically a JMS transport for JMX. Most of the hooks are all there in JMX; we just need a JMS-JMX bridge so that you can

    * send commands to topics to control many servers in one simple operation
    * use topics as a discovery/heartbeat mechanism to provide a distributed view of mbeans using normal JMX

    I'm not aware of any tool for this yet; all the JMX plugins I"ve seen up to date use RMI or HTTP. Though we'll be building one of these soon in the Lingo project

    http://lingo.codehaus.org/

    If you want to take it for an early spin as it arrives.

    James
    LogicBlaze
  4. Hi James,

    Thanks for your thoughtful reply. This gives me a lot of food for thought.

    I've been looking at JGroups over the last day or so, it seems pretty interesting and straightforward. In particular the "RpcDispatcher" seems powerful, and a close fit for the problem of hitting a JMX interface simultaneously on many machines, not to mention the group membership problem.

    Unfortunately, I'm stuck with one of the more disagreeable containers for JMS. But I really like the idea of a JMS to JMX bridge. Thanks for the idea.

    And Rob, well, you know I think you guys are great. Thanks for the info about the Enterprise version.

    cheers,
    Eric