Event sourcing is an architectural software pattern that's useful to design complex and distributed systems, particularly those that run many processes concurrently. The pattern captures and stores all changes to an application's state as a sequence of immutable events.

An event records a fact that has already happened, such as OrderSubmitted or PaymentCompleted. In an event-sourced system, these events are appended to a durable event store and become the authoritative history from which application state can be derived.

Event sourcing is often used alongside messaging technologies such as RabbitMQ, Apache Kafka, Amazon Simple Queue Service (SQS) or Redis, but a message broker is not what defines event sourcing. The defining characteristic is that state changes are persisted as an ordered sequence of immutable events rather than storing only the latest state.

Consumers can build projections, read models and other derived state by processing those events. Messaging infrastructure may publish events to interested components, but the durable event history and the transport mechanism are separate architectural concerns.

If a projection is lost or must be changed, it can often be rebuilt by replaying the relevant event history. Replaying events is powerful, but production systems must also consider event schema evolution, side effects, idempotency and snapshots when histories become large.

Figure 1 below illustrates the concepts of capture, storage and replay.

Capture, storage and replay in an event sourcing architecture.
An event sourcing architecture incorporates capture, storage and replay.

Event sourcing vs. event-driven architecture

Event sourcing and event-driven architecture are related but different ideas. An event-driven system uses events to communicate between components. An event-sourced system uses the event history as the source of truth for state.

A system can therefore be event-driven without being event-sourced. It can also persist events as its source of truth and then publish selected events through a broker for asynchronous processing.

Current state vs. event history

In a conventional persistence model, an order record might simply contain status = "CLOSED". That tells us the current state but not necessarily how it got there. An event-sourced model stores the sequence of facts:

OrderSubmitted
OrderStarted
OrderReady
OrderServed
PaymentStarted
PaymentCompleted
OrderClosed

Replaying that sequence allows a projection to reconstruct the order's current state. Different projections can use the same events to create different views, such as an operations dashboard, customer order history or accounting report.

A real-world analogy for event sourcing: A chess match

A chess game is a good analogy for a temporal application.

In chess, players take turns moving pieces according to the rules of the game, with the objective of checkmating the opponent's king. Often, both players write down each move in the game for recordkeeping and replay later.

Each move of a piece is considered an event. However, no single event describes the entire game accurately. Rather, the game is best described in terms of the execution of all the moves (events) in the game.

For example, aspiring players who want to learn how a Grandmaster played a particular game read a book that describes each move in the game, and they repeat each move on a chessboard in the sequence in which the moves were made. Only by experiencing the entire sequence of events can the reader understand the Grandmaster's strategy.

We can apply the chess analogy to event sourcing as follows:

  • When a player makes a move, this is an event sent into the system.
  • A player recording that move on a piece of paper is event persistence.
  • The recorded move becomes part of the event history, and observers can react to that event.
  • When the opponent responds with a move, that move is considered another event sent into the system.
  • Replay is recreating the game by executing all the recorded moves.

A demonstration project example of event sourcing

The following demonstration project emulates orders processing for a restaurant chain called Terrific Tacos. In this application, each order processing step is described as an event. Here is the sequence of steps in the order process:

  • orderSubmitted
  • orderStarted
  • orderReady
  • orderServed
  • paymentStarted
  • paymentComplete
  • orderClosed

In this simplified demonstration, steps are represented as signals submitted to the WebServer. The server also persists those signals in signals.log. This combines responsibilities that a production architecture would normally separate, but it keeps the example small enough to demonstrate event capture and replay.

A component called a RestaurantManager sends a signal that describes an Order for a particular restaurant to a component called a WorkflowController. The signal has a property called name which indicates the step to which the signal applies.

The WorkflowController passes the signal onto the system's WebServer which, performing the role of a message manager, passes the signal on to a component called a Workflow. The Workflow processes the given order processing step according to the name property of the signal. (See Figure 2.)

Event sourcing architecture for a demonstration application called 'Terrific Tacos.'
The process flow for a demonstration application based on an event sourcing architecture.

The code below is an example of the signal that starts the Workflow process. Notice that the value of the name property of the signal is orderSubmitted.

{
  "id": null,
  "name": "orderSubmitted",
  "timeStamp": "2023-09-02T23:36:54.709Z",
  "order": {
    "orderItems": [
      {
        "description": "Cheese Quesadilla",
        "price": 6.99,
        "quantity": 7
      },
      {
        "description": "Breakfast Burrito",
        "price": 9.99,
        "quantity": 3
      }
    ],
    "customer": {
      "firstName": "Eriberto",
      "lastName": "Runte",
      "email": "Eriberto.Runte@email.com"
    },
    "creditCard": {
      "firstName": "Eriberto",
      "lastName": "Runte",
      "number": "4541511651043665"
    },
    "id": "d86f3d21-c14e-4fd6-9253-fd13f6b0bb29"
  },
  "restaurant": "Terrific Taco Number 1"
}

The Workflow has a set of handler functions that correspond to the various signals. A handler function takes a signal as a parameter. When a handler completes its processing, it returns the next signal to be used in the workflow process.

The returned nextSignal makes the demonstration workflow explicit, but this should not be confused with HATEOAS. HATEOAS communicates available application actions through hypermedia controls in a REST representation; returning the next event name is a workflow technique rather than HATEOAS by itself.

The nextSignal response is then returned by the WebController to the WebServer, which returns the nextSignal to the calling request as an HTTP response. The nextSignal can then be resubmitted to the WebServer to continue the logic of the workflow process.

In our demonstration application, the RestaurantManager component does the work of creating three orders and submits each to a distinct workflow by way of an HTTP POST request to the WebServer. Each order is dedicated to a specific customer created at random, and to a specific restaurant that is part of the restaurant chain.

A separate component called WorkflowPlayer provides the capability to replay the workflow based on the event data stored in the file named signals.log.

The application has two bash scripts: One named runOrders.sh invokes the RestaurantManager to create three orders and submit them to the WebServer, and the other named replay.sh uses the WorkflowPlayer to replay the orders issued by the RestaurantManager using the information stored in signals.log.

Event sourcing challenges in production

Event sourcing provides capabilities that are difficult to achieve with ordinary CRUD persistence, but it also introduces architectural complexity. Teams should plan for several issues:

  • Event immutability: Published historical events should not simply be edited when requirements change.
  • Schema evolution: Consumers may need to understand events written by older versions of the application.
  • Idempotency: Replay and message redelivery must not accidentally repeat external side effects such as charging a credit card.
  • Snapshots: Long event histories may benefit from periodic snapshots so state does not always need to be rebuilt from the first event.
  • Ordering and concurrency: Systems need clear rules for event ordering and conflicting updates to the same aggregate.
  • Read models: Applications commonly maintain projections optimized for queries rather than querying the raw event stream directly.

Putting it all together

Event sourcing is useful when a system benefits from a complete audit history, temporal queries, reproducible state transitions or multiple projections of the same underlying facts. It is frequently seen in domains such as order processing, financial systems and complex workflows.

Event sourcing does not inherently make processing asynchronous or non-blocking. Those properties come from the surrounding architecture and messaging model. Likewise, event sourcing alone does not guarantee fault tolerance or scalability, although a durable event history can make recovery, replication and rebuilding derived state significantly easier.

Event sourcing is not a default replacement for CRUD. Its additional complexity is justified when the history of change is itself valuable, when state must be reconstructed or audited, or when multiple projections of the same sequence of business events provide a meaningful architectural advantage.

The Terrific Tacos demonstration intentionally simplifies several production concerns, but it illustrates the core idea: preserve the facts that changed the system, derive state from those facts, and retain the ability to replay the history when another representation of state must be rebuilt.

Get the source code for the event sourcing example application

The source code for this demonstration application is hosted on GitHub. The application was created specifically for the article and demonstrates the basics of event sourcing as described herein.

The demonstration application is named Terrific Tacos, an ordering system for a fictitious restaurant chain that sells a variety of food products. This GitHub project shows how to implement both event sourcing and replay using a simplified architecture. The demonstration project is written in TypeScript.

Bob Reselman is a software developer, system architect and writer. His expertise ranges from software development technologies to techniques and culture.