Discussions

News: JMS Server Improvements: JBoss MQ HA, Sonic MQ

  1. Some of our JMS solutions are getting more scalable and enterprise ready. Sonic MQ 6.0 contains faught-tolerance or "continuous availability" in their words. Their new solution uses stateful replication between servers in a cluster. JBoss 3.2.4 has added more to HA JMS, bringing features such as automatic fail-over, lossless recovery, and more.

    Sonic MQ 6.0
    "The major new feature in 6.0 of MQ is fault tolerance," said Gordon Van Huizen, Sonic chief technology officer, in an interview on Tuesday.

    "We'll be labeling it, 'continuous availability,'" Van Huizen said.

    Rather than relying on clustering or RAID, version 6.0 uses 'stateful replication' between servers in a cluster. The feature ensures availability, overcoming failures that have plagued messaging environments, according to Van Huizen.
    Read more in Sonic readies fault-tolerant messaging

    JBoss MQ HA

    List of HA features offered by JBoss JMS:
    * Automatic server fail-over
    * Remote client connectivity to HAJMS
    * Lossless recovery after fail-over for messages targeted to durable subscibers
    * Client notification via connection ExceptionListener on fail-over

    See the full document on JBoss MQ HA

    The 3.2.4 release is not publicly available yet. It is past RC1 and is targeted for release shortly.
    Feel free to test and post your comments once it is released.

    Threaded Messages (30)

  2. The passive server can't be used. It's just standby. So the title of this thread is misleading. Scalability goes down, not up.

    Scalability and fault tolerance is either/or in messaging. It is nice that Sonic will provide stateful replication, however, it requires that replication is done synchronously (clients have to wait until their state has been replicated) and, since at least 2 servers are involved, it must be done with 2PC. This is dog-slow and the reason why we, SwiftMQ, don't provide it [it's not rocket science to implement that].

    The alternative to software replication is to use HA software, IP sharing, shared RAID or a clustered database. For Linux check out www.linux-ha.org and look for the free "Heart Beat" software. Advantage is that there is no scalability lost per single server as with software replication. Disadvantage is that clients have to reconnect by itself [to the same IP due to IP sharing].

    -- Andreas
  3. Good point on the title. I have changed it :)

    Cheers,

    Dion
  4. Some of our JMS solutions are getting more scalable and enterprise ready. Sonic MQ 6.0 contains faught-tolerance or "continuous availability" in their words. Their new solution uses stateful replication between servers in a cluster. JBoss 3.2.4 has added more to HA JMS, bringing features such as automatic fail-over, lossless recovery, and more.Sonic MQ 6.0
    "The major new feature in 6.0 of MQ is fault tolerance," said Gordon Van Huizen, Sonic chief technology officer, in an interview on Tuesday. "We'll be labeling it, 'continuous availability,'" Van Huizen said.Rather than relying on clustering or RAID, version 6.0 uses 'stateful replication' between servers in a cluster. The feature ensures availability, overcoming failures that have plagued messaging environments, according to Van Huizen.
    Read more in Sonic readies fault-tolerant messagingJBoss MQ HAList of HA features offered by JBoss JMS:* Automatic server fail-over* Remote client connectivity to HAJMS* Lossless recovery after fail-over for messages targeted to durable subscibers* Client notification via connection ExceptionListener on fail-overSee the full document on JBoss MQ HAThe 3.2.4 release is not publicly available yet. It is past RC1 and is targeted for release shortly.Feel free to test and post your comments once it is released.
    Andreas, I have no idea what Sonic has done, but data replication protocols are often virtually synchronous, in which case the outcome is the same as if it was actually synchronous. Also, this approach does *not* require 2pc, but careful group membership management. You may have good reasons for not clustering, but your description of the problems aren't accurate and I would be careful about knocking a competitor without getting your facts straight. You can bone up on this stuff with Birman's latest book on reliable network applications.

    Greg
  5. Some of our JMS solutions are getting more scalable and enterprise ready. Sonic MQ 6.0 contains faught-tolerance or "continuous availability" in their words. Their new solution uses stateful replication between servers in a cluster. JBoss 3.2.4 has added more to HA JMS, bringing features such as automatic fail-over, lossless recovery, and more.Sonic MQ 6.0
    "The major new feature in 6.0 of MQ is fault tolerance," said Gordon Van Huizen, Sonic chief technology officer, in an interview on Tuesday. "We'll be labeling it, 'continuous availability,'" Van Huizen said.Rather than relying on clustering or RAID, version 6.0 uses 'stateful replication' between servers in a cluster. The feature ensures availability, overcoming failures that have plagued messaging environments, according to Van Huizen.
    Read more in Sonic readies fault-tolerant messagingJBoss MQ HAList of HA features offered by JBoss JMS:* Automatic server fail-over* Remote client connectivity to HAJMS* Lossless recovery after fail-over for messages targeted to durable subscibers* Client notification via connection ExceptionListener on fail-overSee the full document on JBoss MQ HAThe 3.2.4 release is not publicly available yet. It is past RC1 and is targeted for release shortly.Feel free to test and post your comments once it is released.
    Andreas, I have no idea what Sonic has done, but data replication protocols are often virtually synchronous, in which case the outcome is the same as if it was actually synchronous. Also, this approach does *not* require 2pc, but careful group membership management. You may have good reasons for not clustering, but your description of the problems aren't accurate and I would be careful about knocking a competitor without getting your facts straight. You can bone up on this stuff with Birman's latest book on reliable network applications.Greg
    Let's not forget the epidemic approach to replication that has been around for almost a decade. There's been a lot of work on the duality (or not) of transaction systems that use 2PC and replication protocols (ranging in scope from strong consistency to weak consistency, epidemic and virtually synchronous). You can find some of it at arjuna.ncl.ac.uk/publications

    The important thing is that although you could knock up a replication protocol that used 2PC and blocked while disseminating data, why would you? Perhaps you'd do this as a prototype, or if you really hadn't done your homework. But that's no reason to assume all implementations have to be that way.

    Mark.
  6. Andreas, I have no idea what Sonic has done, but data replication protocols are often virtually synchronous, in which case the outcome is the same as if it was actually synchronous. Also, this approach does *not* require 2pc, but careful group membership management. You may have good reasons for not clustering, but your description of the problems aren't accurate and I would be careful about knocking a competitor without getting your facts straight. You can bone up on this stuff with Birman's latest book on reliable network applications.Greg
    Thanks for your suggestions but I simply meant that a client has to wait until the replication is done. For example, if a client sends a persistent message, it must be in stable store on both sides (source and sink).

    With 2PC. You're right. I was too much into message routing where this is required.

    -- Andreas
  7. Andreas, I have no idea what Sonic has done, but data replication protocols are often virtually synchronous, in which case the outcome is the same as if it was actually synchronous. Also, this approach does *not* require 2pc, but careful group membership management. You may have good reasons for not clustering, but your description of the problems aren't accurate and I would be careful about knocking a competitor without getting your facts straight. You can bone up on this stuff with Birman's latest book on reliable network applications.Greg
    Thanks for your suggestions but I simply meant that a client has to wait until the replication is done. For example, if a client sends a persistent message, it must be in stable store on both sides (source and sink).With 2PC. You're right. I was too much into message routing where this is required. -- Andreas
    Andreas, do you mean in a specific implementation (SwiftMQ?) the client has to block, because that's the only case where your statement would perhaps be true. There are replication protocols (such as the ones Greg and I mentioned) where it is not the case that the client has to block waiting for the data to be flushed/sync-ed to the replicas. And this goes for persistent data replication too.

    Mark.
  8. Andreas, do you mean in a specific implementation (SwiftMQ?) the client has to block, because that's the only case where your statement would perhaps be true. There are replication protocols (such as the ones Greg and I mentioned) where it is not the case that the client has to block waiting for the data to be flushed/sync-ed to the replicas. And this goes for persistent data replication too.Mark.
    No I did not meant SwiftMQ because SwiftMQ doesn't provide state replication.

    I would be very interested to hear about a replication protocol which allows a client to continue before the replica is in sync (at least in-memory-sync).

    -- Andreas
  9. I would be very interested to hear about a replication protocol which allows a client to continue before the replica is in sync (at least in-memory-sync). -- Andreas
    I think what they are referring to here is virtual synchrony. The app consists of a group of processes which is formed at some initial time. Then when a failure is detected the processes do a checkpoint, and do a flush of all the messages they have. Then you just starts over.

    This just guarantees that all live processes receive the same messages.

    I think this is called "agreed delivery". Then there is "safe delivery", which means that before the process gives each message to the application it already knows that all the other processes have received.

    The thing is that OLTP applications are not process groups. If you build a replicated JMS server, then that is a process group by itself, and then you don't want agreed delivery, you want safe delivery.

    For example, if you used three replicated JMS processes, then your client would "send" the message, wait for the server to deliver 3 copies safely, and then get an acknowledgement.

    This actually gets pretty good throughput, for a pretty low latency.

    After all, the alternative is to write to disk ..
  10. If you build a replicated JMS server, then that is a process group by itself, and then you don't want agreed delivery, you want safe delivery.
    That's what I said.

    -- Andreas
  11. If you build a replicated JMS server, then that is a process group by itself, and then you don't want agreed delivery, you want safe delivery.
    That's what I said.-- Andreas
    Hmmm, I don't think that's what you said at all. I hate to get pedantic, but when you make sweeping statements that aren't based on accurate information, some clarification is necessary. You said:
    however, it requires that replication is done synchronously (clients have to wait until their state has been replicated)
    and as has been pointed out here already, that isn't the case. There are replication protocols that do not require synchronous blocking techniques at the client or server/replica, and to give the impress that this is not the case is wrong. You (or anyone else) *may* implement replication to be blocking and synchronous, but that's an implementation choice, not a choice that is imposed by any replication fundamentals.

    I have no idea what JBoss MQ or Sonic MQ do for high availability. They may be synchronous, but they don't have to be. If your original post is speaking as an authority on what they do, then you are saying they are synchronous and use 2PC. Maybe someone who helped implement them can confirm this?

    However, if you are saying that in order to do this all implementations must be synchronous and use 2PC, then you are most definitely wrong.

    Mark.
  12. There are replication protocols that do not require synchronous blocking techniques at the client or server/replica, and to give the impress that this is not the case is wrong.
    Well, I have 2 JMS severs where one replicates the state to the other. I want to have hot failover without loss of anything (neither session states nor persistent messages). A JMS client consumes 10 persistent messages in a transaction.

    I would assume that the client must block until the whole tx has been committed at the replica, otherwise it might consume the same messages again after a failover.

    You state that the client can immediately return while the replictaion is still in progress.

    Simple question: How do you avoid that the messages are consumed twice after a failover?

    -- Andreas
  13. Well, I have 2 JMS severs where one replicates the state to the other. I want to have hot failover without loss of anything [..] You state that the client can immediately return while the replictaion is still in progress
    I think he probably just means that the protocol is asynchronous. That doesn't mean much. It just means that the operations involved don't block the calling thread. You still have to go back and check that the operation finished.
  14. I think he probably just means that the protocol is asynchronous. That doesn't mean much. It just means that the operations involved don't block the calling thread. You still have to go back and check that the operation finished.
    No problem with that, e.g. running the commit on the primary and backup server in parallel and have a rendezvous point from where the call returns to the client.

    But he definitely states that the client can savely return before the replica commits. Look here:
    There are replication protocols (such as the ones Greg and I mentioned) where it is not the case that the client has to block waiting for the data to be flushed/sync-ed to the replicas. And this goes for persistent data replication too.
    IMHO the replica has to commit the changes and has to report it back before the client operation returns otherwise it couldn't do a fail over since the replica might be inconsistent. The only exception I could imagine is to run a rollforward (recovery) on failover at the new primary server. However, this has nothing to do with the replication protocol but it is just *what* is replicated, in this case a log stream which has to be applied to the actual data before the server could be used.

    Anyway, I was wrong in one assumption (and I did already admitted that) and that was that 2PC is *required*. As I said I was too much in a message routing world where messages are moved from one hop to the other. Here it is required to use 2PC (optimized, anyway). Instead replication just work with views and updates and doesn't require to use 2PC.

    I don't see that I'm wrong in the assumption that a client has to wait until primary and backup (replica) both have committed the tx. At least Mr. Little wasn't able to show me how that works then with hot failover.

    -- Andreas
  15. I think he probably just means that the protocol is asynchronous. That doesn't mean much. It just means that the operations involved don't block the calling thread. You still have to go back and check that the operation finished.
    No problem with that, e.g. running the commit on the primary and backup server in parallel and have a rendezvous point from where the call returns to the client. But he definitely states that the client can savely return before the replica commits. Look here:
    There are replication protocols (such as the ones Greg and I mentioned) where it is not the case that the client has to block waiting for the data to be flushed/sync-ed to the replicas. And this goes for persistent data replication too.
    IMHO the replica has to commit the changes and has to report it back before the client operation returns otherwise it couldn't do a fail over since the replica might be inconsistent. The only exception I could imagine is to run a rollforward (recovery) on failover at the new primary server. However, this has nothing to do with the replication protocol but it is just *what* is replicated, in this case a log stream which has to be applied to the actual data before the server could be used.Anyway, I was wrong in one assumption (and I did already admitted that) and that was that 2PC is *required*. As I said I was too much in a message routing world where messages are moved from one hop to the other. Here it is required to use 2PC (optimized, anyway). Instead replication just work with views and updates and doesn't require to use 2PC.I don't see that I'm wrong in the assumption that a client has to wait until primary and backup (replica) both have committed the tx. At least Mr. Little wasn't able to show me how that works then with hot failover. -- Andreas
    Andreas, if you go an read the references then we can talk. Having a strong basis on which to build the discussion is important or we'll just go round in circles. Anyway, it seems from your previous replies that you're not really interested in what is possible, only in what you can't do. Good luck with that approach!
  16. I think he probably just means that the protocol is asynchronous. That doesn't mean much. It just means that the operations involved don't block the calling thread. You still have to go back and check that the operation finished.
    No problem with that, e.g. running the commit on the primary and backup server in parallel and have a rendezvous point from where the call returns to the client. But he definitely states that the client can savely return before the replica commits. Look here:
    There are replication protocols (such as the ones Greg and I mentioned) where it is not the case that the client has to block waiting for the data to be flushed/sync-ed to the replicas. And this goes for persistent data replication too.
    IMHO the replica has to commit the changes and has to report it back before the client operation returns otherwise it couldn't do a fail over since the replica might be inconsistent. The only exception I could imagine is to run a rollforward (recovery) on failover at the new primary server. However, this has nothing to do with the replication protocol but it is just *what* is replicated, in this case a log stream which has to be applied to the actual data before the server could be used.Anyway, I was wrong in one assumption (and I did already admitted that) and that was that 2PC is *required*. As I said I was too much in a message routing world where messages are moved from one hop to the other. Here it is required to use 2PC (optimized, anyway). Instead replication just work with views and updates and doesn't require to use 2PC.I don't see that I'm wrong in the assumption that a client has to wait until primary and backup (replica) both have committed the tx. At least Mr. Little wasn't able to show me how that works then with hot failover. -- Andreas
    Andreas, if you go an read the references then we can talk. Having a strong basis on which to build the discussion is important or we'll just go round in circles. Anyway, it seems from your previous replies that you're not really interested in what is possible, only in what you can't do. Good luck with that approach!
    Come on. I asked a very simply question how you do the trick to hot failover to a possible inconsistent replica.

    It just seems that you are unable to provide a simple answer on that.

    Or should I just conclude that you work for a SwiftMQ competitor, Arjuna, and that your actual intention was to embarrass me? Schieb ab, du Heini!
  17. Andreas, if you go an read the references then we can talk. Having a strong basis on which to build the discussion is important or we'll just go round in circles. Anyway, it seems from your previous replies that you're not really interested in what is possible, only in what you can't do. Good luck with that approach!
    Come on. I asked a very simply question how you do the trick to hot failover to a possible inconsistent replica. It just seems that you are unable to provide a simple answer on that. Or should I just conclude that you work for a SwiftMQ competitor, Arjuna, and that your actual intention was to embarrass me? Schieb ab, du Heini!If I get a spare moment I'll try to write something to explain it to you. However, as to your comment about embarrassing you, that's not the case: you do a good enough job of that yourself.
    "Schieb ab, du Heini"?
    Are you serious? Please act your age and not your shoe-size, or you're going to find it very difficult to get anyone to talk to you about anything!
  18. Well, I have 2 JMS severs where one replicates the state to the other. I want to have hot failover without loss of anything [..] You state that the client can immediately return while the replictaion is still in progress
    I think he probably just means that the protocol is asynchronous. That doesn't mean much. It just means that the operations involved don't block the calling thread. You still have to go back and check that the operation finished.
    Asynchronony is useful, but it doesn't give you the same failure semantics. If you want high availability (with guaranteed delivery) then there's more to it than asynchronous invocations. Take a look as Isis or Horus, for example, from Cornell University (Ken Birman again).
  19. There are replication protocols that do not require synchronous blocking techniques at the client or server/replica, and to give the impress that this is not the case is wrong.
    Well, I have 2 JMS severs where one replicates the state to the other. I want to have hot failover without loss of anything (neither session states nor persistent messages). A JMS client consumes 10 persistent messages in a transaction.I would assume that the client must block until the whole tx has been committed at the replica, otherwise it might consume the same messages again after a failover.You state that the client can immediately return while the replictaion is still in progress.Simple question: How do you avoid that the messages are consumed twice after a failover?-- Andreas
    Andreas, it's not possible to do this subject justice via this medium. I can certainly point out more references (maybe even dig out a copy of my 10+ year old PhD thesis). In principle what you want is a protocol that guarantees delivery of individual messages, so that if either end (sender or receiver(s)) fails, those messages aren't resent, or aren't re-enacted (BTW, RPC mechanisms of the 80's and 90's did the latter using something called retained results). The problem is surmountable and it doesn't require blocking for a N replicas.

    Mark.
  20. [..] it's not possible to do this subject justice via this medium. I can certainly point out more references (maybe even dig out a copy of my 10+ year old PhD thesis). In principle what you want is a protocol that guarantees delivery of individual messages, so that if either end (sender or receiver(s)) fails, those messages aren't resent, or aren't re-enacted (BTW, RPC mechanisms of the 80's and 90's did the latter using something called retained results). The problem is surmountable and it doesn't require blocking for a N replicas.Mark.
    Fair enough. Please post the references. I took a look at the Arjuna publications page, but I haven't found anything about epidemic replication (I don't know where to look,) only something about symmetric replication.

    I look forward to reading these references. Incidentally, I did read much of Birman's book.

    But I think the issue in this thread is a requirements issue. What I was talking about above is supporting JMS transactions. A transaction requires durability, i.e. you really should write to stable storage - that's the D in A.C.I.D. I believe. If you make replicas and write to stable storage later, that's probably good enough durability depending on the lag and the degree of replication.

    If you try to do epidemic replication, then until N copies are made then you don't have the durability required, whatever that is.

    If you are _not_ talking about transactional JMS, then you don't need to write to stable storage. Then you can do something else. Then it just requires some communication between the client and the server to ensure correct failover.

    Again, I am talking durability here.
  21. Lots of stuff deleted
    Again, I am talking durability here.
    Hi again (and probably for the last time on this thread). Yes, I understand you're talking about durability, but you need to think about why you need durability. It's a requirement for transactions to ensure that they can recover from failures: ignoring participants, the durability aspect is intended for the coordinator to write its transaction log (aka intentions list) to durable storage, so that, in the event of a failure and subsequent recovery, the failure recovery subsystem can complete the transaction. Now, durability doesn't necessarily require a hard disk: NVRam would do in some cases, and active replication with an "appropriate" number of replicas may also be used. As with anything (including hard disks) you can never get a 100% guarantee of recoverability: the D is only a good chance to survive failures excepting a catastrophic disk failure.

    As I mentioned to Andreas, I made time to write down some basic thoughts on the subject. It's a little long, but it's still no way sufficient to do this topic justic. Hopefully it'll wet your appetite to do some exploring: it really is an interesting topic, and although perhaps not "rocket science" it's not trivial either.

    First thing to remember is that availability isn't necessarily proportional to the number of replicas: throwing more replicas at a problem can actually reduce availability and performance. Availability depends on the number of replicas, but also critically on the replication protocol you are using. For example, if we use an Available Copies protocol (active replication, where there's no primary), then you can read from any one replica, but you must write to all available replicas. So, increasing the number of replicas *may* improve availability, but it will certainly decrease performance for write operations.

    In fact, the number of replicas and their placement is critical across all protocols. There's been some work in this area over the past decade or so, but replica placement is important. Take another example: if all of my replicas reside on machines that draw power from the same source, then I'm not improving availability to power failures. In some cases, I may actually be able to improve availability and performance by reducing the number of replicas but placing them on highly reliable machines. But this isn't a hard-and-fast rule and anyone using replication needs to consider their personal environment and the types (and number of) failures they want to tolerate. You never get something for nothing, so you're often trading off performance for availability.

    As with transactions (where you trade off performance for fault tolerance and isolation), availability is probabalistic. Even if you took an Available Copies protocol (AC) and threw 10 replicas into it, there's still no 100% guarantee that they won't all fail at the same time: all you can do is try to reduce the likelihood. But anyone who knows their probability theory and/or statistical mechanics, knows that the universe always loads the dice, and given enough time anything is possible (like all the air molecules in the room you're sitting in suddently deciding to go and spend time together in one corner).

    After all this, what I'm trying to get at is that saying 4 replicas is better than 2, or that 10 is better than 4 isn't necessarily true, and if it is, it may be by tiny fractions of a probability percentage (e.g., 99.95% availability for 4 versus 99.96% availability for 10, but 10 write operations per second versus 5 write operations per second - I use these figures as examples only, as they taken from some work we did back in the 90s and are not representative of all protocols and deployments).

    Next there's the protocol type: active replication, single copy passive replication, coordinator-cohort (somewhere inbetween), strong consistency, weak consistency. And within these there are different flavours (AC, weighted voting, epidemic, leader-follower, lazy replication, viewstamped, psync, etc.) The reason there are so many is that they each make assumptions about the environment in which they'll operate (e.g., it can't be partitioned), and the objects/states that they'll be used to replicate (e.g., active replication assumes that all replicas are deterministic). Once again, there are tradeoffs to be made. For example, if you take a particular active replication protocol, you may get really good performance for read operations and perhaps even for some write operations, but it may not be able to tolerate partitions and does require you to ensure all replicas are deterministic (if they start in the same state and receive the same set of messages in the same order then they'll end up in the same final state); that's actually harder to ensure that you might expect, especially when you throw non-realtime operating systems into the equation. Happily passive replication doesn't have this deterministic limitation since only one replica (the primary) receives requests and then checkpoints its state to the other replicas (the backups); a passive protocol may even be able to tolerate network partitions; the drawback is that performance in the event of primary failure is poorer than for active replication, because a new primary has to be elected by the backups.

    So once again there are tradeoffs to be made, and these tradeoffs need to take into account the specifics of your requirements and environment.

    With all of the above in mind (and it does gloss over a lot of issues), we can turn to the problem in hand, which is replicating a specific type of object or service (a JMS). All I can describe here is *one* specific way in which it *could* happen, and this shouldn't be taken as definitive. I hope I've shown that it's just not possible (or safe) to say to all users that N replicas are what they need to achieve a certain level of availability; it needs to be done on a per user basis and only after careful consultation with that user to determine their deployment environment.

    Let's assume we pick passive replication (and for simplicity ignore how the primary is chosen, but we'll assume it's the same primary for all clients); we'll stick each JMS server on a different machine as well. Furthermore let's assume that we want to tolerate 2 simultaneous machine failures (maybe even prioritise this on the type of message) and still be able to gain access to the latest data. In this case we might decide that we want to have 5 replicas, and the rule for the primary is that when it gets an update it has to sync that update to its local disk and to one of the backups before it can return to the client. In the background (after the return to the client), the primary or the backup can then disseminate the updates to the remaining backups, which are out of date.

    BTW, this assumes we're not using multicast protocols (we'll stick with point-to-point). However, if we were, then there are obviously advantages to this that we could capitalise on.

    Now you may ask what 5 replicas gives me over 2, since it's only 2 that are brought up-to-date initially. The answer is that in the event of more than 2 failures (though not simultaneous failures), the recovery time for the entire replica group can be better than having to start up a fresh replica and get it to obtain all of the state. Trickling updates to warm-standbys is often better than starting from cold.

    And then there's "tricks" we could play with heartbeats, that are along the lines of epidemic (gossip-driven) protocols: in many replication protocols (e.g., virtual synchrony) the replicas need to keep track of who's up (in the group) and who's down (not in the group). They do this by heartbeat messages (pings) exchanged between each other (or this may be entirely driven by a master/slave). Obviously there's no way to tell with surety that a replica/machine has failed (network partitions or loss of messages may give the same appearance) until that replica/machine recovers and tells you, so all of this is actually failure suspicions. But I digress - back to heartbeats. Most of the time these messages will just be "are you alive?" or "what's your crash count?". However, because they are exchanged at regular intervals, you could piggyback more information on them, such as the changes in state that have occurred between the last message and this one.

    I've tried to give a flavour of what is possible using replication protocols and techniques, but it's really difficult to do the subject justice in such a short space (it took me 200 pages in my thesis back in 1991 and things have moved on since then). There's no single protocol (and its rules) that has to be used to replicate all types of object/state. Choices are made based on the characteristics of the environment and the thing that is being replicated, but in order to make the right choices, you need to understand what the ramifications of them are (e.g., active replication *requires* determinism). And it's important to remember that just throwing more replicas at a situation isn't necessarily going to improve availability.

    BTW, another option might be to say use a virtually synchronous reliable causal broadcast mechanism, so that only related messages are seen in the same order by everyone. The advantages to this should be fairly obvious.

    All the best,

    Mark.
  22. We are talking about HA JMS with hot failover on base of state replication (don't confuse that with Fiorano's heart beat with a shared store).

    Let's say we have a primary/backup configuration. A JMS client should be able to switch transparently from the primary to the backup and continue with the next operation (or repeat the last op in transit under the covers) because the state is replicated to the backup. For example, if the client has consumed messages 1 to 10 from primary and it fails, it should simply connect to the backup and continue with message 11.

    This requires that the client has to wait on each operation it performs until it is replicated to the backup. Moreover, the backup has to report the result of the operation back to the primary before the operation is finished and returns to the client. So a request/reply is necessary between primary/backup.

    Whatever protocol you chose for your replication - the client has to wait IMHO. Otherwise you cannot failover.

    You state a client doesn't need to wait but you are unable to describe how you do that magic.
  23. Yes, I understand you're talking about durability, but you need to think about why you need durability. It's a requirement for transactions to ensure that they can recover from failures: ignoring participants, the durability aspect is intended for the coordinator to write its transaction log (aka intentions list) to durable storage, so that, in the event of a failure and subsequent recovery, the failure recovery subsystem can complete the transaction. Now, durability doesn't necessarily require a hard disk: NVRam would do in some cases, and active replication with an "appropriate" number of replicas may also be used. As with anything (including hard disks) you can never get a 100% guarantee of recoverability: the D is only a good chance to survive failures excepting a catastrophic disk failure.
    I understand everything you are saying, but I disagree with you on why you need the durability. You are saying that it's a means to recoverability. I know that if I build an application from scratch using process groups then I can get much better performance and availability than if I use traditional means. But this is irrelevant if you are selling a JMS server that participates in an XA transaction. I think most customers would find it very difficult to buy a JMS server that doesn't write to disk. It's a problem of requirements, not technology. I am not even sure that you could claim to be XA compliant if you don't actually write to disk.

    This issue is really at the core of an old article by Ken Birman where he looks back at group communication and assesses why it didn't pick up - this was after ISIS. The problem is that ACID transactions and process groups kind of fight each other.
    BTW, another option might be to say use a virtually synchronous reliable causal broadcast mechanism, so that only related messages are seen in the same order by everyone. The advantages to this should be fairly obvious.All the best,Mark.
    I am glad I finally found someone who values total ordering. Check out my implementation of TOTEM: http://www.evs4j.org
  24. I would be very interested to hear about a replication protocol which allows a client to continue before the replica is in sync (at least in-memory-sync). -- Andreas
    I think what they are referring to here is virtual synchrony. The app consists of a group of processes which is formed at some initial time. Then when a failure is detected the processes do a checkpoint, and do a flush of all the messages they have. Then you just starts over.This just guarantees that all live processes receive the same messages.I think this is called "agreed delivery". Then there is "safe delivery", which means that before the process gives each message to the application it already knows that all the other processes have received.The thing is that OLTP applications are not process groups. If you build a replicated JMS server, then that is a process group by itself, and then you don't want agreed delivery, you want safe delivery.For example, if you used three replicated JMS processes, then your client would "send" the message, wait for the server to deliver 3 copies safely, and then get an acknowledgement.This actually gets pretty good throughput, for a pretty low latency.After all, the alternative is to write to disk ..
    There is certainly a distinction between the message being received by one member of the group and a message being received by all of them. However, that distinction does not mean that the sender is blocked until all recipients have acknoweldged that fact. If that were the case then the state of the art in replication would be in a very poor condition. It isn't, and hasn't been since the early 1990's.

    Mark.
  25. Andreas, do you mean in a specific implementation (SwiftMQ?) the client has to block, because that's the only case where your statement would perhaps be true. There are replication protocols (such as the ones Greg and I mentioned) where it is not the case that the client has to block waiting for the data to be flushed/sync-ed to the replicas. And this goes for persistent data replication too.Mark.
    No I did not meant SwiftMQ because SwiftMQ doesn't provide state replication.I would be very interested to hear about a replication protocol which allows a client to continue before the replica is in sync (at least in-memory-sync). -- Andreas
    Andreas, this isn't the venue to do a Master Class on replication techniques, but if you'd like one, then we can do int via email. I'd suggest reading up on the references that have already been supplied though.

    Mark.
  26. Andreas, this isn't the venue to do a Master Class on replication techniques, but if you'd like one, then we can do int via email.
    Sorry, I don't take offers from assholes. And you are one as you state above.

    -- Andreas
  27. Andreas, this isn't the venue to do a Master Class on replication techniques, but if you'd like one, then we can do int via email.
    Sorry, I don't take offers from assholes. And you are one as you state above.-- Andreas
    That's an interesting argument style. Kind of glad I haven't had time to get back into this discussion all of a sudden.

    Greg
  28. Andreas, this isn't the venue to do a Master Class on replication techniques, but if you'd like one, then we can do int via email.
    Sorry, I don't take offers from assholes. And you are one as you state above.-- Andreas
    Very adult of you. Good debating tactics too. Now are you going to stamp your feet and walk off in a tantrum too?
  29. SpiritSoft has been offering stateful 2PC replication between servers in a cluster for well over a year. Our solution has been successfully deployed in the intensity of banking environments. What exactly is new about what Sonic is offering ?

    Rob Davies

    SpiritSoft
    www.spiritsoft.com
  30. Geronimo, JMS, JORAM[ Go to top ]

    I looked at the Geronimo CVS repository. I did not find
    a JMS implementation. What does the Geronimo team have planned
    for JMS?
  31. Some of our JMS solutions are getting more scalable and enterprise ready. Sonic MQ 6.0 contains faught-tolerance or "continuous availability" in their words. Their new solution....
    Nothing surprising.. SonicMQ has a long history of following FioranoMQ!

    FioranoMQ has multiple commercial deployments of HA already in production since last year. I won't be surprised hearing about a Peer to Peer (Grid) story from Sonic after a year.

    - Amit Gupta
    Fiorano Software Inc.