Discussions

News: Enterprise Service Bus the next big EAI strategy?

  1. According to Garnter, the Enterprise Service Bus (ESB) concept could could bring about a major shift in the enterprise application integration (EAI) software and application server markets. ESB software uses messaging and workflow to transport data between applications and data sources connected to the bus.

    Read Enterprise Service Bus the next big EAI strategy?.

    The article criticizes the 'hub and spoke' model of using an application server as the integration center, however, companies such as BEA (WL Integration) and Novell (exteNd) have been building this in suites that run on top of application servers.

    Threaded Messages (63)

  2. Doesn't surprise me! As mentioned, the Hub and Spoke provides a single point of failure. All clients have to rely on the server to distribute published messages to all recipients thereby providing an extra software hop. For hundreds/thousands of receivers, this doesn't scale at all.

    The Bus model allows individual clients to do their own persistence the load for which should have been borne by the hub(server). Using techniques such as multicasting network load can be greatly reduced.

    On such multicasting based messaging product is XtremeSG from mPulse Technologies Inc. You may check it out at http://www.mpulsetech.com

    Thanks
    Ram Dasari
    mPulse Technologies
  3. <quote>
    ... an application server's hub-and-spoke architecture is a bottleneck for high data volumes and a single point of failure
    </quote>

    Isn't this why we have the ability to cluster an application server? How is a clustered, multi-threaded "messaging server" based hub-and-spoke any worse than the bus architecture?

    Also, what are the drawbacks of using a bus architecture?
  4. Pratap Das quotes:
       Isn't this why we have the ability to cluster an
       application server? How is a clustered, multi-
       threaded "messaging server" based hub-and-spoke any
       worse than the bus architecture?


    A BUS is an extreme form of clustering wherein all end-points (publishers/subscribers) provide the functionality of the application server. If you stretch the limits, a hub (server) however clustered or multi-threaded can never match the throughput of the BUS for pub/sub.
  5. Thanks for your clarifications. Going thru the responses from other posters & some research, I gather that

    A) Hub & Spoke Architecture
    (+) Allows centralized processing/routing of messages.
    (+) All messages go thru this message broker & the broker is routed to the subscribers.
    (+) Transformation/Routing rules are located in one central location.

    (-) Hub is a single point of failure.
    ## Solution: Cluster/Load balance the broker. You could also have distributed hub & spoke architectures.

    (-) For pub/sub only operations, the bandwidth requirements will be higher from broker to subscriber link. i.e. if broker has to send message to 5 subscribers, it will have to pump in 5 separate (duplicate) messages to the network.

    Questions:
    ## Is it possible for broker to use multicast in such cases?
    ## What if there is a mixture of PTP & pub/sub messages being transported in the system?

    B) BUS Architecture:
    (+) Using multicast, sender directly sends messages to the subscribers.
    (+) The central broker's (hub) services is distributed to each publisher / subscriber.
    (+) Single hop delivery using the routers as the bus.

    (-)Unnecessary overhead of messages in case there is significant PTP messages.
    ## How does BUS architecture perform when there is significant PTP?

    (-)Maintenance of Transformation/Routing rules which is now distributed over several nodes.
    ## Could have a central "admin hub" which "pushes" rules to each available node (??)

    (-)Still has a single point of failure (in terms of routers rather than hub)
    ## Hardware is probably more reliable (??)

    Other questions are:
    * Which is less costly in terms of
    a) hardware and / or
    b) maintenance?

    * Do existing MOMs choose exclusively one architecture over the other or do they choose a hybrid that can fit both bills??

    > If you stretch the limits, a hub (server) however
    > clustered or multi-threaded can never match the
    > throughput of the BUS for pub/sub.

    The key phrase here is "for pub/sub". I would think that in real-world scenarios, an organization would have a mixture of pub/sub and PTP (read legacy integration). In such situations, I wonder how the performance compares.

    If any of my previous observations are inaccurate, do clarify.
    --Das
  6. A few questions from Pratap Das:
    ## Is it possible for broker to use multicast in such cases?
    A few reliable multicast protocols, mostly tree based, that support just a single sender and multiple receivers become unsuitable for the BUS architecture. But they are a perfect fit for the hub/spoke with the root of the tree (the sender) being the broker. Ofcourse, you are free to use a reliable multicast protocol that supports multiple senders as well as receivers. However, given similar features of reliability, a single-sender protocol is better optimized and easier to maintain than a multi-sender protocol.

    (-)Still has a single point of failure (in terms of routers rather than hub)
    Routers are needed to forward packets whether for unicast or multicast or BUS or Hub/Spoke or ...
    If you consider a router to be a point of failure, then that will be a factor that has to be considered for all data transfers.

    Ramana
  7. This sure has been a lively discussion. I would like to clear up a few things about the concept of the ESB as a technology trend, and also what it means to Sonic (since that's where this all started ;-).

    - The word &#8220;Bus&#8221; in this context does not mean to imply anything about IP multicast technology. I have enjoyed the discussion on IP multicast and I can appreciate the pros and cons being pointed out. In the context of the ESB, the word &#8220;Bus&#8221; refers to the ease of pluggability, and the abstraction that is provided. An application plugs into the bus. It posts data to the bus. It receives data from the bus. The application need not be concerned with the details of all the other applications and services that need to share the same data, or how they want to be communicated with. The &#8220;Bus&#8221; takes care of getting the data to its intended destinations, in the target data format that the endpoint needs to see it in.

    - Hub and spoke has its problems too, as some have pointed out here in this thread. What&#8217;s necessary is an interconnected network of hubs, capable of traversing firewalls and being deployed between companies or across geographically dispersed divisions of a global corporation, in a secure and reliable fashion with a minimum number of &#8220;moving parts&#8221;.

    - As the original posting has pointed out, Gartner is seeing the Enterprise Service Bus as a general technology trend, and has been talking about it regularly and publicly--both in their published reports (Note Number: DF-18-5857) and at their recent technology shows that they have hosted.

    - As someone else pointed out..yes, Sonic did create the concept, and the term. It&#8217;s the result of working with our customers over the past few years, and listening to what they wanted to do with messaging, and also what they *don&#8217;t* want. The ESB includes a number of technologies that have come to an acceptable maturity level
     o Enterprise Messaging, beyond the basic capabilities of what JMS prescribes.
     o Web services technologies such as SOAP and WSDL
     o Content-based routing and data transformation using, but not limited to, XPATH and XSLT.
     o Connectivity using standards &#8211; From a Sonic perspective, this means being able to connect into the ESB using JMS, JCA, Apache Axis, C/C++ (which mimic the JMS APIs), and direct HTTP using a seamless bi-directional mapping between SOAP-over-HTTP to SOAP-over-messaging.

    When combined together in a service oriented architecture, across a globally connected messaging backbone, you have the basis for a low-cost, standards-based, highly distributed alternative to the typical centralized proprietary monolithic integration broker approach.

     - As was pointed out by the webservices.org article, and by a posting from Nathan Chandler, this can simply be a combination of existing technologies, and not necessarily a particular vendor&#8217;s product suite. The beauty of using standards is that you can choose from best-of-breed implementations and plug them together.

     - Appservers are important, as nodes on the ESB. What we find very often in large corporations is there is almost never just one appserver vendor being deployed globally. There are usually departmental camps using BEA, Borland Appserver, Websphere, etc. What&#8217;s needed is a common way of connecting them all.

     - Sonic&#8217;s approach includes the notion of a highly distributed, lightweight services container. These containers are processes that can house business logic, host a specialized service like a transformation engine, or act as a callout to an external web service or business partner. I use the word &#8220;lightweight&#8221; not to imply that its weak, or lame&#8212;rather it refers to the nimble and agile fashion that a services container can be selectively deployed with only the services that you need, when and where you need them. For example, a transformation service can be selectively deployed at a remote node co-located with an application endpoint that requires a particular set of transformations. Or, this service can also be combined with an adapter that further facilitates communication with the application at the endpoint. The choice is yours. The services container allows either approach. This is in contrast to having to install another Appserver stack or replicate an entire EAI engine at a remote location simply to act as a conduit to an EIS. Also, service containers are abstractly strung together using the concept of a message &#8220;itinerary&#8221; which simply put is a list of destinations that gets carried around with a message as it travels across the ESB. This is in contrast to the idea of a centralized integration broker that has to be constantly referred back to for the next step.

     - Yes, Sonic is working as part of the core Axis developer team. In addition to the JMS transport binding that we donated, we are also working with the other members of the Axis team add asynchronous capabilities to the core Axis engine. This work extends beyond the current binding to JMS, is intended to be work with any protocol, and will be capable of adapting to any reliable asynchronous Web services communications standards as they evolve and mature.

    Any questions about this, feel free....
    Dave
  8. Dave,

    I'm always glad to serve as your buddy to prepare the terrain so that you can pray. It reminds me when I gave you a helping hand against Fiorano in our old JMS-INT list and then got notice that your sales sent my mail around to your prospective leads, saying "Look, my dear lead, if Andreas says Sonic is better than Fiorano, then it must be a great product. Choose it over SwiftMQ!".

    Thanks Dave. Thanks Sonic. I love you. Snifff.

    > As someone else pointed out..yes, Sonic did
    > create the concept, and the term.

    The question I have is how much do you spend to bring it into the Gartner channel? Can I do the same? Please send me some instructions.

    Yours, Andreas
  9. The problem with most clusters now - although I have to admit that I haven't tried all that's out - is managing the cluster.
    With ESB, I can just start/stop/upgrade new "node" (i.e. new instance of the consumer process) w/o worriyng about any other nodes, at runtime if I wish. Again, maybe some app servers can discover that a new node has been added automatically and deploy the cluster's apps to the node & start using it..
    Problem with ESB is that it can be easily swamped by tons of messages that can kill the middleware and/or network hardware. Sometimes it is hard to come with a good semantics of when message should be available to other systems or consumed entirely (it may well depend on your internal business logic). I.e. topics vs. queues and sometimes you may need your topic to behave like queue or vice versa based on the content of the message.
    Regards,
    Vlad
  10. On such multicasting based messaging product is

    > XtremeSG from mPulse Technologies Inc.

    Multicast based messaging is only a niche. If that is your main feature, you might rethink your business model.

    > You may check it out at http://www.mpulsetech.com

    Done. Without going into details, I would suggest to think about some form of quality control... I hate stack dumps on every button click and on every example I try.
  11. Andreas, thanks for you comments!
    1. About multicasting -- it sure is a niche. But thats the only way to go if you need extreme message throughput and real-time performance. Also, lookout for a protocol from mPulse that will provide a combination of Unicast and Subnet-Multicast and can be deployed in all networks (no multicast support needed). Watch out for the throughput on that one!

    2. About stack dumps: Comments well taken! However, we did purposefully put in a lot of debug messages for the evaluation version.
  12. 2. About stack dumps: Comments well taken! However,

    > we did purposefully put in a lot of debug messages
    > for the evaluation version.

    But they look like stack dumps! ;-)
  13. A new approach to enterprise application integration???

    Hardly. For a start Tibco have been doing this for some time. They are a big MOM player but they dont dominate the EAI market by any stretch.

    Message Oriented Middleware is obviously key to enterprise integration. Its asynchronicity is a key reason for this.

    As for MOM flavours, multi-cast messaging suits certain types of problems. Multicast based messaging is well suited for highly pub-sub messaging, and particularly where message sizes are small and you dont mind if you miss a few.

    For most EAI problems, however, the messaging is not highly pub-sub. Moreover, it usually has to be transactional and the message sizes are often large. Here, Queue-based messaging makes more sense.
    Now, it is certainly possible to implement transactional queues using multi-cast messaging, however the benefits of multicast are lost and the cost of multicast mounts.

    Particular problems we have seen with Tibco RV are network load related. As volumes get higher, missed messages increase (as clients are consuming more cpu receiving and discarding messages they have no interest in). Requests for re-transmissions and the re-transimissions themselves add to the volume. Once the spiral becomes unstable, your network goes down (ours has).

    >> An ESB avoids a spaghetti of company application
    >> interconnections because existing applications, services
    >> and other data sources need only to plug into the bus to
    >> communicate

    Not true at all.
    The message formats are as binding as Point-to-point interfaces. While XML loosens the binding, it is no miracle.

    As for the "hub and spoke" model, it is by far an easier beast to maintain and monitor. The distributed nature of messaging (esp P2P) makes it more difficult to monitor and administer.

    I see Appservers as gateways. They support a variety of client interfaces (CORBA, JMS, SOAP, RMI, and now J2EE Connector) and equally can use these same integration means to connect to back-end systems (as well as JDBC). Add some workflow and some transformation and you have your toolkit.

    The J2EE appserver is the EAI platform - not proprietary MOM-based EAI products.

    -Nick
  14. Multicast based messaging is well suited for highly pub->sub messaging, and particularly where message sizes are >small


    The message size is really not a issue. Multicasting can be used for large messages too -- by dividing the message into packets just as is done by unicasting protocols like TCP.

    >Now, it is certainly possible to implement transactional >queues using multi-cast messaging, however the benefits >of multicast are lost and the cost of multicast mounts.
    You are right. P2P or transactional queues are better implemented using unicast. But for Pub/Sub multicasting is still the superior solution.

    >Particular problems we have seen with Tibco RV are >network load related. As volumes get higher, missed >messages increase
    ...
    > Once the spiral becomes unstable, your network goes down (ours has).
    Multicasting is not to blame for this. Infact the real strength of multicasting lies in the fact that it utilizes lower networking resources than multiple unicast connections. A lot of vendors implement multicast-protocols that perform good only if the network is not loaded. Congestion avoidance and Congestion control are implemented the best in TCP and hence its popularity and its ability to be a truly Internet protocol. TCP ensures fairness by sharing available bandwidth with other TCP-like protocols. Any multicast (or unicast) protocol that ignores TCP-friendliness is likely to cause the collapse that you mentioned. In an effort to win the throughput-race, a lot of vendors ignore this key criterion and cause congestive collapse which is a state of the network where the network is fully utilized but no real data transfer takes place.

    >I see Appservers as gateways. They support a variety of >client interfaces (CORBA, JMS, SOAP, RMI, and now J2EE >Connector) and equally can use these same integration >means to connect to back-end systems (as well as JDBC). >Add some workflow and some transformation and you have >your toolkit.
    The EAI problem is not solved by completely forgoing multicasting or unicasting but to use both of them for problems that each solves best.

    -Ramana
  15. But for Pub/Sub multicasting is still the superior

    >> solution.

    No arguments from me.

    >> The message size is really not a issue

    I think it is.
    a) In order to "chunk" messages, the application layer is effectively trying to do what TCP does at the transport layer. However, far less efficiently.
    b) large messages accentuate the losses due to missed messages and also the impact of the "unfriendliness" of multicast.

    >> The EAI problem is not solved by completely forgoing
    >> multicasting or unicasting but to use both of them for
    >> problems that each solves best

    Agreed.

    -Nick
  16. With big messages (and missed messages) you can play all sorts of tricks, like for example storing the data in a common repository and sending just a handle.
    Of course, that makes the repository a SPoF, but then for pretty much all applications the persisten store is SPoF (and you may need to do replication and stuff.. ). And possibly brings in a number of other problems, like concurrency etc.

    The reason why I prefer message bus to app server is that it can give me more flexibility. I can have any applications listening to the bus (as long as there's interface :) of course, but most of them can talk to C functions so it can be solved). We had C++ thin clients on Windows talking to some custom and off-the-shelf C++ apps on Solaris and a bunch of Python apps running all around the place (for example our message logging/debugging tool was put together in Python in a couple of days). I think now it also talks to some Java apps and possibly something on a mainframe.
    Sort of what web services are promising now, except we did it more than 3 years ago :).
    V.
  17. a) In order to "chunk" messages, the application layer is

    > effectively trying to do what TCP does at the transport
    > layer. However, far less efficiently.

    I thought we were discussing the transport layer here. The TCP protocol layer is, for example, a set of C libraries in UNIX, LINUX etc. Since TCP is a standard protocol, OS support is provided by all operating systems. For multicasting, all that the OS provides is unreliable IP level support. For reliability, another protocol layer has to be added on top of IP -- much like TCP provides reliability over IP transport for unicast. As far as message size is concerned, there is really no extra overhead that multicast protocols have to handle that is not handled by unicast. Also remember we are comparing one multicast with multiple unicast connections and NOT one multicast with one unicast connection.

    mPulse Technologies implements multicasting reliability using JAVA mainly to ensure portability across different OS'. The best way to do this would be to program the reliability layer in native languages. But thats an impractical solution. JAVA based protocols don't performly much poorly either. Please refer to JRMS (Java Reliable Multicast Service) for more information on JAVA based multicasting.

    Ramana
  18. mPulse Technologies implements multicasting

    > reliability using JAVA

    Just read that you're using an ACK based protocol. You state that it is 100% reliable. Aren't you bound then to the slowest subscriber?
  19. mPulse Technologies implements multicasting

    > reliability using JAVA

    I would be very interested to know how you do the trick with an ACK-based multicast protocol, 100% reliability and slow subscribers. I mean, you're the one who stated to have the one-fits-all solution with your multicast product. Come on, provide an answer on that simple question, please.
  20. slow subscribers

    Sure! I will answer your answer. I concurred with other people here that multicasting is not a good solution for P2P. "1 solution fits all" -- did I say that?

    First lets look at the extreme condition and the "legal" answer to this question -- a receiver is just too slow and it is causing QoS problems in the entire network. Most protocols NACK(Negative Ack) or ACK have a QOS option wherein a receiver that fail to catch up to a certain minimum rate limit gets a socket error. Essentially, if a receiver doesn't have enough resources to catch up it will be thrown out. That receiver should -- I don't know -- upgrade its network routers or get on to a faster network!

    Now lets look at this in perspective:
    We are talking about slowness at the protocol layer. If you look at the big picture, the applications almost always can't catch up with the rate at which data *can* be passed on to them. So, most of the times the rate at which the slowest reciever gets data is still more than the rate at which the application can consume these messages.

    mPulse tries to alleviate the slow receiver problem by using two different congestion windows: one to determine the rate at which new data can be multicast (fast receivers determine this rate -- they acknowledge received packets and ask for more) and the other window is used to delete old acknowledged data (slow receivers determine this rate). The difference between these two congestion windows is the buffer size of the sender. Given a large enough buffer, temporary congestion conditions and the resulting slow receivers don't effect QoS. Example, a 1mb buffer tolerates about 900 packets of difference. This means, the fast receivers won't even notice the slow ones until the slow receiver falls 900 packets behind. So, most temporary congestion conditions in some parts of the network go unnoticed. Ofcourse, if the whole network is congested, all receivers will slow down.
  21. Essentially, if a receiver doesn't have enough

    > resources to catch up it will be thrown out.

    And the message is lost forever.

    > Example, a 1mb buffer tolerates about 900 packets
    > of difference.

    It doesn't matter how large your window is, an ACK based protocol with "100% reliability" slows down the rate to the slowest receiver. It's only a matter of time when your buffer fills up.
  22. And the message is lost forever.

    Not the message. The receiver will be thrown out. But, a very unlikely case unless you have wild variations in bandwidth. We are talking about the protocol here. There is a whole messaging layer on top of the protocol.

    >slows down the rate to the slowest receiver.
    Thats accurate if the buffer is full. But still far better than multiple unicast connections. One a x kbps LAN, lets say the slowest receiver consumes at 0.5x -- all receivers irrespective of the number of receivers will get the data at 0.5x. Compare this with 10 unicast receivers on the same network. You are talking about a throughput of .1x for each receiver and it degrades as the number of unicast receivers increase. Basic multicasting 101.
  23. Hi Ramana,

    I don't think anyone doubts that multicast-based messaging has its place. Have you had a chance to compare to: Talarian (now Tibco) SmarkSockets, PGM (Pretty Good Multicast), and JavaGroups?

    Peace,

    Cameron Purdy
    Tangosol, Inc.
    Coherence: Easily share live data across a cluster!
  24. Sorry! Didn't mean to belabor my argument any more than the next guy (read Andreas ;-))

    I believe JavaGroups is a framework to develop multicast apps. Like I had mentioned in a previous post, mPulse supports the JRMS API.

    We haven't yet performed quantitative analysis in comparison to SmartPGM or TIBCO's protocol. On the qualitative side, TCP friendliness is a key feature for mPulse and I expect strong adherence to this will make it a bit slower. In addition, using ACKS and ensuring 100% reliability also will add additional load. PGM is a NACK based best-effort delivery protocol. But for enterprise messaging, reliability and congestion control are both important. We will post our findings on the website soon.

    Ramana
  25. Sorry! Didn't mean to belabor my argument any

    > more than the next guy (read Andreas ;-))

    I have no problem with multicast. Since a long time, SwiftMQ provides it as well between SwiftMQ routers as an alternative to routing connections.

    The only remark I have (and that's why I asked you at all) is that you should (IMO) extend you marketing sentence "ACK-based and 100% reliable" with "but bound to the slowest subscriber".

    There's another product I like to add to Cameron's list. It's iBus MessageBus. It's a serverless JMS impl, using it's own multicast protocol. AFAIK it uses a combination of ACK/NACK. It has been on the market since a long time and you might compare your product with it.
  26. "Bound to the slowest receiver" is like saying "the chain is as strong as its weakest link". Thats obvious!!

    The alternative is even worse. If your multicast protocol goes faster than the slowest receiver, there is data loss at the slowest receiver. Hence, any protocol that claims 100% reliability has to be only as fast as its slowest reciever! But, also keep in mind my previous posting about how mPulse absorbs the small bumps in the road without the whole system knowing about it. And that the chain is strong enough even if you keep repeating "the chain is as strong as its weakest link".
  27. The J2EE appserver is the EAI platform - not

    > proprietary MOM-based EAI products.

    Absolutely. A J2EE app server provides you with everything, except the workflow. It's not part of J2EE. You can deploy a [low cost] app server on each node, receive messages from MDBs, call an EIS/WS, whatever, and send a result to a queue where another app server is listening on etc. You only need some kind of JMS MOM in the middle.

    However, to create, deploy, administer, monitor the workflow itself you need proprietary tools.
  28. The problem with the hub and spoke model is, as mentioned by previous posters, is that there _is_ always one point of failure, be it an appserver or a hardware router/load balancer - 90% of today's applications access a system (network port) through it's location aaa.bbb.ccc:xxx, if that point goes down - you're screwed.

    IMHO, only Jini allows proper 'acquaintance clustering' of components - I mean acquaintance, in that applications connect by describing the requirements they need, and matching them to a system that can satisfy those requirements, at runtime, a clusterable component should be redistributable to other machines without requiring reconfiguration, and clients should not need to know network addresses.

    As mentioned the appserver is a gateway however it is also a component of the distributed system, not just an entry point - this effectively creates a conflict of interests.

    Really an application server should not be clusterable as a whole, but an application server should be a 'container' of seperately clusterable components so I could have one appserver in the cluster just running MDB's and one appserver running webcaching, etc, etc.

    Clustering is just a way for vendors to milk their clients. However I do not believe that vendors put in enough effort to allow app servers to be clusterable at the feature level rather than 'here's two app server s in a cluster and each of them does everything that an appserver does' - what if your requirements are that you need lots of Data source integration lots of JCA and JDBC, but really your JSP and servlet needs can be handled by one app server.

    If vendors can already do this then please forgive me, but I always feel disheartened when that gleam gets in eye of the sales person


    Calum
  29. Calum,

    I was wondering whether someone was going to mention Jini. It really does seem to be an incredibly robust way to build distributed applications. Anyone actually used this in an enterprise-class production setting?

    Of course, if the point of this article is discussing EAI strategies, Jini probably isn't going to help too much, eh...?
  30. Bus rules EAI[ Go to top ]

    Hi,
        I have experienced integration solutions based on Bus architecure as well as Hub & spoke.
       
        "Hardly. For a start Tibco have been doing this for some time. They are a
         big MOM player but they dont dominate the EAI market by any stretch"

        >The reason for TIBCO not dominating EAI market is different.
         That cannot be a best example to prove Hub is better than Bus.
         Even today if you take the completeness of products,TIBCO ranks on top.
         
         Here is a converse of your thought,
         IBM websphere Integration suite is reffered as a top most player in EAI
         Arena.That's a complete false statement.Its because IBM has good
         marketing strength he is able to project his product.

    As long as Event driven is going to be in demand, Bus based products will lead the EAI market.
  31. That's a marketing article from Sonic. They created that ESB term as well. Thanks, Floyd, for this valuable information.

    IMO this is not more than another attempt to establish huge frameworks to lock customers into products and to have cash cows like in the good'ol times with proprietary MOMs. Luckily, this time is over and such huge frameworks with $10K and more per node are a dead end. But it's certainly the only way for oversized MOM vendors to argument their existence and to get further money from their investors.

    However, the core technology, JMS, will survive.
  32. Hi Andreas,
    "
    lock customers into products and to have cash cows like in the good'ol times with proprietary MOMs
    "
    "
    JMS, will survive.
    "

    I liken this to database technology and JDBC.....
    where these MOMs charge the $$ for reliability, scalability charesteristics (unless a jboss-like team can do this but that to the best of my knowledge does not exist in the MOM space - SwiftMQ has turned to a $$$ product also.)

    and JMS provides platform independant access to all such MOMS..

    So if you are against proprietary MOMS what other options do you see that will provide reliable messaging, routing to large-scale organizations? I am not talking about ESB, just messaging ..
  33. SwiftMQ has turned to a $$$ product also.)


    I'm a bit disappointed that you use 3 of the $ signs here. SwiftMQ is not expensive.

    > So if you are against proprietary MOMS

    I'm not against proprietary MOMs. I only said their time is over.

    > what other options do you see that will provide reliable > messaging, routing to large-scale organizations? I am not > talking about ESB, just messaging ..

    Plain JMS, what else? ;-)
  34. Plain JMS, what else? ;-)


    JMS is only an API. The actual messaging services are
    supplied by JMS-compatible providers. With high end
    features most of these cost $$$'s


    Also, I recently read an article by chappell (Sonic) on
    webservices.org that introduces the ESB mainly outside of
    the context of Sonic products. If of interest see:

    "Asynchronous Web Services and the Enterprise Service Bus"
    http://www.webservices.org/index.php/article/articleview/352/
  35. JMS is only an API. The actual messaging

    > services are supplied by JMS-compatible
    > providers. With high end features most
    > of these cost $$$'s

    The question is whether I need to use proprietary stuff at my client or not. My position is that JMS is enough. A service of that proprietary ESB is not more than a JMS sender/receiver plus highly proprietary code around which will lock you into that product and converts yourself into a cash cow. Not only the software is very expensive. There is also much consulting required before that stuff works.

    And with the high end features. There are solution which have that built-in and doesn't cost you an arm and a leg.

    > Also, I recently read an article by chappell
    > (Sonic) on webservices.org that introduces the
    > ESB mainly outside of the context of Sonic products.

    Come on. Do you believe that? Dave Chappell is an Evangelist. His task is to make noise by writing technical articles to promote Sonic products and he's doing a good job. Look at the pictures in the article. Looks like SonicXQ, huh? He has recently contributed a JMS provider interface to Apache Axis. That's a 1-day-job. Other products like GLUE have this SOAP-over-JMS since a year. That's nothing new. However, Dave's marketing people pushed article over article (like the one you mentioned) about this pille-palle and it sounds like rocket science. Good marketing, but no meat.

    The truth is that companies starting as standalone JMS vendors have problems to get enough revenue out of their JMS products. The problem is just the portability of the JMS clients and the high competion in this area. Next time a customer should pay $$$ for maintenance, it changes the JMS provider and pays less for a production license of the new product than for the maintenance of the old one. Up and away. The best way to make the real money is to have proprietary frameworks which requires consulting. That's what they do. With the help of Gartner.
  36. It seems to me as if hub-and-spoke and ESB are topologically isomorph. Or, in other words, if you take the logical diagram of a hub-and-spoke and elongate the central dot that represents the hub into a line you get an ESB.
    This transformation of 'elongation' of the hub could be seen as represented by replacing a single app server by a cluster. As Pratap Das remarked.
    So that leaves (a-)synchronicity as the most significant difference.

    Or am I missing something?

    Sincerely,
    Joost
  37. "It seems to me as if hub-and-spoke and ESB are topologically isomorph. Or, in other words, if you take the logical diagram of a hub-and-spoke and elongate the central dot that represents the hub into a line you get an ESB."


    I loved this assessment as it stimulated me to think about what is really different about what Gartner is proposing [other than a diagramming style :-) ]

    Here what I concluded about the difference between ESB and H & S as I cycled to work thru the rain this morning:

    1. Not asynchronicity. All of the mainstream EAI vendors are classified by Gartner as using H & S yet they all employ messaging.

    2. Not clustering. Again all of the EAI old guard support this. For me this just represents a bigger hub.

    3. Cluster to Cluster Technology. If you have the ability to merge the destination name space over many clusters I think that you could start to credibly argue for the elongation of the hub into a bus. If distributed services can communicate with services connected to other clusters and to be unaware of location or topology then we are getting somewhere.

    4. Further distribute the contents of the hub. The EAI vendors generally stuff the hub with all the other parts of their stack, ie content based routers, transformation, business process engines and adaptor engines. An ESB would allow you to bring these on to a bus and distribute to wherever made sense.
  38. "It seems to me as if hub-and-spoke and ESB are topologically isomorph." -- Joost

    The ordering of messages between a given sender and a given reciever is deterministic with a bus and non-deterministic with a cluster.
  39. Brian: "The ordering of messages between a given sender and a given reciever is deterministic with a bus and non-deterministic with a cluster."

    Hi Brian, where does that definition or assumption originate? Our cluster software has always provided ordered messages from a given sender to a given receiver. Maybe you meant from a set of given senders to a given receiver? (That is often referred to as a "total ordering" implementation.)

    Peace,

    Cameron Purdy
    Tangosol, Inc.
    Coherence: Easily share live data across a cluster!
  40. "Our cluster software has always provided ordered messages from a given sender to a given receiver." -- Cameron

    My understanding of clustering is that two messages from the same client dispatched to the cluster in sequence may be processed by two different servers in the cluster. If those two servers in the cluster must forward these messages to the same external subscriber, then this is surely a total ordering problem. I don't see how ordering can be maintained through the cluster.
  41. I'm curious to know how an ESB would support integration at the business process level (BPI) compared to the application level (classic EAI).

    My concern is that in the BPI case, even if you replicate the business process execution model (which control how messages are routed and transformed) on every node, you still have to maintain and share the context/state of your business processes as it will be used to determine the appropriate routing rules and messages transformation.

    What happen if this state is large (containing several big business documents that were retrieved from a mainframe)? Also how could the process execution be fully transactionnal?

    Could the lightweight service containers described in the ESB solution support all these functionalities?

    Regards,
    Jerome.
  42. A very informative white paper from Promenix which gives compelling arguments (mostly using network considerations) in favor of unicast-based message transport for EAI:

    "The pitfalls of using multicast Publish-Subscribe for EAI"
    http://www.ibm.com/software/ts/mqseries/library/whitepapers/pitfalls.pdf

    Regards,
    Jerome.
  43. Come on. Do you believe that? Dave Chappell is an

    >Evangelist. His task is to make noise by writing technical
    >articles to promote Sonic products and he's doing a good
    >job. Look at the pictures in the article. Looks like
    >SonicXQ, huh? He has recently contributed a JMS provider
    >interface to Apache Axis. That's a 1-day-job. Other
    >products like GLUE have this SOAP-over-JMS since a year.
    >That's nothing new. However, Dave's marketing people
    >pushed article over article (like the one you mentioned)
    >about this pille-palle and it sounds like rocket science.
    >Good marketing, but no meat.

    If you wish to take such a cynical view then of course you are more than free to do so. However, you only know the picture looks like XQ because you have seen XQ - show me anywhere in the article that any reference or link is made to sonic. Taking a non-cynical view my spin on this article is that it is indeed evangelistic but more about the merits of asynchronous messaging and its relation to building reliable workflow systems. There is nothing to stop one taking the architecture described and implementing it using freely available technologies.

    Also, you question the AXIS submission - would you rather commercial companies did not provide contributions? Why didn't The Mind Electric do their bit for the reference implementation if they are so ahead of the game?
  44. Also, you question the AXIS submission -

    > would you rather commercial companies did
    > not provide contributions?

    Have I said that? My whole point was that the initial post of this thread and your article are marketing articles.
  45. I apologise if my reading was wrong but your previous posting seemed to introduce the AXIS submission in a way that simply served to belittle it.

    Just for the record, in the context of marketing (and indeed anti-marketing) are you the same Andreas Mueller that founded IIT software?

    http://www.swiftmq.com/about/
  46. are you the same Andreas Mueller that founded

    > IIT software?

    Sure, I am. But my opinions are my own, not related to my employer.
  47. actually, we have already "contributed" our JMS integration to the "reference" implementation - it's called "GLUE" ;-)

    cheers,
    graham
  48. I might be biased, but I think you are off the mark.
    If you take the relative costs of an application server license to 'most' jms vendor licenses its more like buying a scooter compared to buying lexus. And btw, all vendors try to push consulting through their professional services organizations, but that is never where the real money is. In consulting, through pso's or through pure consulting companies, there exists a large amount of overhead, and the margin's never compare equitably to the kinds of margins you can achieve by distributing the costs of r&d for a product against cheap cd copying in mexico or china. Real $$ in software has always been in volume distribution for those who do this stuff well (m$ft for example).

    Jin
  49. And btw, all vendors try to push consulting

    > through their professional services organizations,
    > but that is never where the real money is.

    Hmm. You might check where IBM makes the money with, for example.
  50. ok, call, where does IBM make their money? From your response, one would assume that your position is through their global services division. Is that where IBM (in particular) is making their margins? in professional services?

    I think that your point, no offense, is countered by what has appeared in the economy as a whole. Very few organizations can dictate the kind of consulting fees that were around in the bubble era, and if you go do a search on all of the pure consulting companies that have bellied up, you will find quite a few. Add to that the number which are in dire straits, and it grows bigger.

    Part of the problem with consulting as a revenue generator lies in the fact that it does not scale very well, IMHO. Your margins are always trimmed by the overhead in salaries, turnover, and general hr expenses. As consulting fees plummet, the margins grow slimmer.

    If your position were accurate, I would think that vendors would get out of the product space and focus on consulting, or is the product just a lead loss for the consulting?

    Jin
  51. You are right in that salaries, g&a cut into consulting margins when compared to product margins. However, IBM has a huge consulting force which it (very effectively) uses to sell gazillion dollar contracts to Fortune 500 companies. By selling the whole solution, they sell more product licenses, which as you stated, have higher margins. With consulting, product, training, etc. you have the whole enchilada and therefore gazillion dollar contracts. It is probably why IBM recently bought PWC for a fraction of the amount HP *tried* to buy them for. ($3.5B vs. $18B) IBM had better timing. :) It is also one of the reasons why IBM is gaining strength over Sun, BEA, etc. since they can offer one stop shopping.
  52. I think somewhere I saw that 47% of this year's revenue is from IBM GS. When I used to work for IBM about 5 years back, it was less than 10% (and the whole rev no was about the same).
    V.
  53. I think that your point, no offense, is countered

    > by what has appeared in the economy as a whole.

    No. I'm talking about products like this ESB stuff. This requires integration. It doesn't run out of the box. Thus, you sell a package consisting of licenses and consulting.
  54. Andreas and Vaper, my apologies, I had no idea the kind of $ IBM was making from consulting. However, what if it does run out of the box? or with tenable ramp up times for the developers and architects? The point I am trying to make is that not all companies and developers need hand holding from the consulting organization of the vendors in order to make their products work. Agreed, most it managers will want to get the vendor pso initially if this is the first time the product is being used in house, but only if it is within budget, which is becoming increasingly less common given the economy.

    Personally, as a developer, I want what I think everyone else wants, products that are simple to install, simple to configure, with costs that allow bringing more developers on board instead of spending the $ on the licenses. Open source ? :-)

    Jin
  55. I was in a team that successfully implemented this few years back in C++ for a large fixed interest trading system.
    The advantages were quite numerous, such as linear scalability,
    high availability, ease of deployment/upgrades..
    But..
    The major thing here though is that all the applications must still agree on message format etc. Saying that message brokers (that do transformations) will solve it is not enough - you can transform only what you have.
    EAI won't be solved by any silver bulets. Moving information from one place to another is now easy, and can be done in a lot of different ways. _What_ information should be moved is the important question, and to me it seems like most people ignore it.
    Regards,
    Vlad
  56. The major thing here though is that all the

    > applications must still agree on message format etc.

    Yup. But don't confuse multicast with that ESB term. What they mean is just a form of workflow, based on P2P. A service is listening on an input queue, processes a message, does some generic or specific transformation, calls an EIS or a Web Service, generates an output message and sends it to another queue which is input of another service and so on. So in fact, this is just plain workflow, based on messaging. The 'bus' is a logical term of a messaging infrastructure, connected by routing connections. It's just the ability to address queues, located wherever.
  57. how can you call something P2P bus? :) Marketing?
    V.
  58. Novice question here[ Go to top ]

    Ummm I did a little bit of research into multicast technology a little while back and when I read in this thread about how this approach is sooo much better than a 'hub-n-spokes' messaging architecture, I couldn't help but think 'but multicast IS a hub-n-spokes architecture'. The 'hub' in this case is the router. You 'subscrbe' and 'publish' to a 'topic' (i.e. multicast IP) through communication with the router through IGMP.

    Am I a missing something here? The routers must maintain information about if anyone is subscribed so they know whether to pass the message on or not. They can talk to other 'hubs' oh I mean routers to pass this information up the router tree just MOM hubs can.

    There is a single point of failure, i.e. the router, but just like the software hubs the goal is to make the hub (software or hardare) so reliable that we don't have to worry about it, i.e. we can rely on it. I think many software hubs have reached this as well as hardware routers.

    With a software hub we can get transactions, garunteed delivery, monitoring, security, etc. With a hardware hub (router) we get ... well routing, which is great and all, but that means the burden of all the other stuff rests on the each end-point's shoulders.

    As the topic of this message suggests I'm no expert in this area, but again am missing something? I don't see the advancement. The architectures are fundamentally the same as far as I can tell.

    Cheers.
  59. While I am not sure as to how much can a Technical Architect can rely on analysts such as Gartner who have been proven incorrect repeatedly, Message Bus architectures are definitely an evolution in the complex Integration landscape. I think that by using a message bus one can provide a clearer abstraction to the complex back end systems that exist in an enterprise. Also, the message bus will only be a delivery and interfacing mechanism, application servers will continue to serve the backend processing. So, I don't see how message bus can replace application servers. Also, evolution of standards based messaging is neccessary and it will take at least two more years for this to evolve. I refuse to believe any predictions as far as 2008, it is simply too far to predict anything in this market and dynamic environment.
  60. While I am not sure as to how much can a Technical Architect can rely on analysts such as Gartner who have been proven incorrect repeatedly, Message Bus architectures are definitely an evolution in the complex Integration landscape. I think that by using a message bus one can provide a clearer abstraction to the complex back end systems that exist in an enterprise. Also, the message bus will only be a delivery and interfacing mechanism, application servers will continue to serve the backend processing. So, I don't see how message bus can replace application servers. Also, evolution of standards based messaging is neccessary and it will take at least two more years for this to evolve. I refuse to believe any predictions as far as 2008, it is simply too far to predict anything in this market and dynamic environment.
  61. I have not been able to locate the Gartner document that the referenced article apparently alludes to. (That enough qualifiers for everybody?)

    I surmise that someone has tacitly equated the terms "Enterprise Service Bus" and "Enterprise Nervous System", the latter being a Gartnerism for a somewhat more inclusive concept, embracing not only the "smart wire" that automatically routes "smart messages", but also the accumulation and mining of the message payloads.

    We all need to step back and remind ourselves of the meaning of the word "infrastructure". There are all kinds of reasons why we don't really want a wire to be "smart". We want it to be fast and reliable, and ignorant, so that it reamins plug-replaceable with something faster or more reliable when that comes along.
  62. I have not been able to locate the Gartner document that >the referenced article apparently alludes to. (That enough >qualifiers for everybody?)


    Go to http://www.gartner.com and enter DF-18-5857 in the "Search Research" box at the top of the page. You will get a link to the article. You have to be a Gartner client to actually get access to it.
    Dave
  63. Dave,

    I am highly grateful to you. My boss will never believe me when I tell him how poor Gartner's search facility is, even when I let him watch over my shoulder and see that I get no match, or maybe a 18% match, on a term that we both know is in a specific document. Even the search on the Gartner document number shows only a 32% relevance! What more do they want?

    ;-)
  64. I have not seem in your post mention to the most important key point, IMHO, for the promissing ESB technology.

    I would not see ESB as a framework to make EAI easier, despite it intends that, and probably would. I find ESB as a overall framework to recognize, in a market scope, BUSINESS STANDARDS, trying to parametrize business. Thats the actual power from my point of view. Technology is just below.

    What about of doing plug-and-play software removing the huge burden of ETL and integration? Do not watch if would scale better, if would be clustered... probably it will everything. But the strong point are the open market for business tools. Opinions?