I have messages in a MQ queue that needs to be processed in the same sequence as it was received. The solution needs to be highly available...
What solution can I use to process the message in a sequence and still provide the capability of automatic failover.
We have been involved in similar scenarios in the past specifically with MQ. One of them was with one of the largest exchanges organisation. The requirements was to take messages from MQ which was delivered by another part of the organization parse them, validate them and store them in an oracle 10G data warehouse for further processing.
We used our space based messaging
approach combined with master/worker
pattern to achieve that. The master/worker on spaces are transactional and ensure that a message that that was retrieved from MQ and stored in the space will not be lost until it was successfully processed. Processed in this case means stored in the oracle 10G. The messages can be retrieved in the same order they were written.
With this approach we were able to push 300-400 million rows processed in 1:45 hours using 2x4-Way Itanium processors.
You can find an interesting presentation
that was given to Gartner by one of the group members of this organisation.
Write Once Scale Anywhere