Alex - stock.adobe.com
Event sourcing is an architectural software pattern that's useful to design complex and distributed systems, particularly those that run many processes concurrently. The pattern captures and stores all changes to an application's state as a sequence of immutable events.
Events are represented in a software system as messages or signals. Typically, messages are controlled by a message management or broker technology such as Redis, RabbitMQ, Apache Kafka or Amazon Simple Queue Service (SQS).
The event sourcing design pattern is particularly well-suited to temporal applications. A temporal application is one in which the state of the application evolves over time toward a given outcome.
Under event sourcing, messages that describe a particular event are submitted to a message manager. The message manager saves the event data to storage and forwards the message onto a message queue. An interested component listens for messages coming into the queue. The component consumes the message and executes logic based on the content of that message. The consuming component's logic varies according to the type of event that the received message describes.
Should the message manager or consuming application fail, all that's needed to replenish the overall state of the application is to replay all the event messages that have been captured and stored.
Figure 1 below illustrates the concepts of capture, storage and replay.
A real-world analogy for event sourcing: A chess match
A chess game is a good analogy for a temporal application.
In chess, players take turns moving pieces according to the rules of the game, to meet the game's objective which is to capture the opponent's king. Often, both players write down each move in the game for recordkeeping and replay later.
Each move of a piece is considered an event. However, no single event describes the entire game accurately. Rather, the game is best described in terms of the execution of all the moves (events) in the game.
For example, aspiring players who want to learn how a Grandmaster played a particular game read a book that describes each move in the game, and they repeat each move on a chessboard in the sequence in which the moves were made. Only by experiencing the entire sequence of events can the reader understand the Grandmaster's strategy.
We can apply the chess analogy to event sourcing as follows:
- When a player makes a move, this is an event sent into the system.
- A player recording that move on a piece of paper is event persistence.
- The opposing player who observes (listens for) the move is the event emitted from the system.
- When the opponent responds with a move, that move is considered another event sent into the system.
- Replay is recreating the game by executing all the recorded moves.
A demonstration project example of event sourcing
The following demonstration project emulates orders processing for a restaurant chain called Terrific Tacos. In this application, each order processing step is described as an event. Here is the sequence of steps in the order process:
Steps are instigated via signals (events) that are submitted to the WebServer, which acts as a very primitive message manager. Also, the WebServer is the sole component that does the work of event persistence, by storing signal data in the signals.log file.
A component called a RestaurantManager sends a signal that describes an Order for a particular restaurant to a component called a WorkflowController. The signal has a property called name which indicates the step to which the signal applies.
The WorkflowController passes the signal onto the system's WebServer which, performing the role of a message manager, passes the signal on to a component called a Workflow. The Workflow processes the given order processing step according to the name property of the signal. (See Figure 2.)
The code below is an example of the signal that starts the Workflow process. Notice that the value of the name property of the signal is orderSubmitted.
"description": "Cheese Quesadilla",
"description": "Breakfast Burrito",
"email": "[email protected]"
"restaurant": "Terrific Taco Number 1"
The Workflow has a set of handler functions that correspond to the various signals. A handler function takes a signal as a parameter. When a handler completes its processing, it returns the next signal to be used in the workflow process.
This is very similar to the use of HATEOAS in a RESTful API, in that the nextSignal response makes the workflow progress self-descriptive.
The nextSignal response is then returned by the WebController to the WebServer, which returns the nextSignal to the calling request as an HTTP response. The nextSignal can then be resubmitted to the WebServer to continue the logic of the workflow process.
In our demonstration application, the RestaurantManager component does the work of creating three orders and submits each to a distinct workflow by way of an HTTP POST request to the WebServer. Each order is dedicated to a specific customer created at random, and to a specific restaurant that is part of the restaurant chain.
A separate component called WorkflowPlayer provides the capability to replay the workflow based on the event data stored in the file named signals.log.
The application has two bash scripts: One named runOrders.sh invokes the RestaurantManager to create three orders and submit them to the WebServer, and the other named replay.sh uses the WorkflowPlayer to replay the orders issued by the RestaurantManager using the information stored in signals.log.
Putting it all together
The event sourcing design pattern is particularly useful for businesses that operate large-scale applications that use many processes asynchronously.
Event signals are consumed in a non-blocking manner, so processes can run simultaneously. This is a big benefit for applications that support thousands or even millions of users, all running at the same time. Examples of this include order management systems, shopping carts and streaming services.
Also, with event sourcing's replay capability, one can create systems that are fault-tolerant and replicable at scale.
It's not easy to get event sourcing to work in production; there are a lot of moving parts, and it requires significant time and effort. Hopefully the concepts and code presented in this article provide the foundation a company needs to move toward adoption in an informed manner.
Get the source code for the event sourcing example application
The source code for this demonstration application is hosted on GitHub. The application was created specifically for the article and demonstrates the basics of event sourcing as described herein.
The demonstration application is named Terrific Tacos, an ordering system for a fictitious restaurant chain that sells a variety of food products. This GitHub project shows how to implement both event sourcing and replay using a simplified architecture. The demonstration project is written in TypeScript.
Bob Reselman is a software developer, system architect and writer. His expertise ranges from software development technologies to techniques and culture.