Transactional Outbox Pattern: Diagram and Implementation Guide

Every event-driven service eventually writes code like this: save the order, then publish OrderPlaced. It works in testing. In production, the process crashes between the two lines, and the order exists without the event. Or the publish succeeds and the database transaction rolls back, and the event announces an order that does not exist.

That is the dual-write problem, and the transactional outbox pattern is the standard fix. This post explains the problem, the pattern, the two ways to build the relay, and the details that decide whether it actually works: ordering, duplicates and cleanup.

The dual-write problem

A database and a message broker are two systems with no shared transaction. Any code that writes to both has a window where one write succeeded and the other did not:

Retries do not close the window; they move it. Ordering tricks, such as publishing from an after-commit hook, only shrink it: a crash between the commit and the hook still loses the event. Distributed transactions across the database and the broker are rarely available, and slow when they are.

The pattern

Write the event to the database, in the same transaction as the business data, and publish it later from there:

1. The service inserts the order row and an outbox row (event ID, aggregate ID, type, payload) in one local transaction. Both commit or neither does.
2. A separate relay reads new outbox rows and publishes them to the broker.
3. The relay marks rows as published, or deletes them, once the broker acknowledges.

The guarantee is now simple: if the order exists, its event will be published, at least once. Chris Richardson's description of the pattern uses the same structure.

CREATE TABLE outbox (
  id            uuid PRIMARY KEY,
  aggregate_id  bigint      NOT NULL,
  type          text        NOT NULL,
  payload       jsonb       NOT NULL,
  created_at    timestamptz NOT NULL DEFAULT now()
);

Two ways to build the relay

Polling publisher. A loop selects unpublished rows in batches, publishes them and deletes them. In Postgres, SELECT ... FOR UPDATE SKIP LOCKED lets several relay instances share the work without blocking each other; see the locking clause documentation. It is easy to build and easy to understand. The costs are polling latency and load, and an ordering subtlety: rows with lower IDs can commit after rows with higher IDs, so a poller ordered by ID can skip past a row that is not yet visible.

Log tailing (change data capture). A CDC tool reads the database's transaction log and turns committed outbox inserts into messages. Debezium's outbox event router does exactly this for outbox tables, routing each row to a topic based on its columns. Events come out in commit order with low latency and no polling queries, at the cost of running and operating the CDC pipeline.

At least once means duplicates

The relay can publish a row and crash before marking it published. On restart it publishes the row again. That is unavoidable, so consumers must be idempotent:

• Every event carries a unique ID (the outbox row's ID).
• Consumers record the IDs they have processed, ideally in the same transaction as the effect of processing (an inbox table), and skip IDs they have seen.
• Or make the effect naturally idempotent, such as an upsert keyed on the order ID.

Ordering

Consumers usually care about order per aggregate: OrderPlaced before OrderCancelled for the same order. Use the aggregate ID as the message key so all events for one order go to the same partition, and publish in commit order (CDC does this; a poller should order by a sequence assigned at commit time, or process one aggregate's events serially). Global order across all orders is rarely needed and expensive to provide.

Cleanup

An outbox table that is never emptied grows without bound and slows the poller. Delete rows after publishing, or keep them for a short audit window and drop old partitions. With CDC, rows can be deleted right after insertion in the same transaction, since the connector reads the insert from the log, not from the table.

Walking through the failures

Check the design against each crash point:

• Crash before commit. Neither the order nor the event exists. The client gets an error and can retry.
• Crash after commit, before the relay runs. Both rows exist; the relay publishes the event when it starts.
• Relay crash after publishing, before deleting the row. The event is published again on restart, and consumers drop the duplicate by ID.
• Broker down. Rows accumulate in the outbox; the service keeps taking orders, and the relay catches up when the broker returns. Alert on the age of the oldest unpublished row.

In every case the database stays the source of truth, which is the property the pattern exists to provide.

The architecture diagram

How to read it:

• Left: the order service writes two rows, in the orders and outbox tables, inside one Postgres transaction. The frame marks the transaction boundary.
• Middle: the outbox relay, by polling or CDC, reads new rows and publishes them to the order-events topic.
• Right: the shipping and email services consume the events. The email service checks a table of processed event IDs before acting, which is how duplicates are absorbed.
• The SQL block shows the transaction, so nobody has to guess what "atomically" means here.

What to put on the diagram

• The transaction boundary. Draw both tables inside it; that is the whole pattern.
• The relay and its type. Polling or CDC changes latency and ordering.
• Delivery semantics. Write "at least once" on the relay and "idempotent" on every consumer.
• The message key. Note that events are keyed by aggregate ID.

The event-driven architecture post covers the consumer side in more depth, and the saga pattern diagram shows where an outbox is needed in each step of a multi-service workflow. The database replication post explains the write-ahead log that CDC reads from.

Build it in Archboard

1. Sign in with Google or GitHub and create a diagram.
2. Add a Container from the C4 Model pack for the order service and another for the outbox relay.
3. Add a Frame from the Core pack labeled with the database and the transaction, and put two Entity nodes from the Entity Relation pack inside it for the orders and outbox tables.
4. Add Kafka from the Data pack for the topic, Container nodes for the consumers, and a Database for processed event IDs.
5. Add a Markdown block from the Core pack with the transaction's SQL.

The example uses 10 diagram nodes, within the Free plan.

FAQ

What problem does the transactional outbox solve? Losing or inventing events when a service writes to its database and publishes a message as two separate steps. The outbox makes the event part of the same database transaction.

Is the outbox pattern exactly once? No. It guarantees at least once delivery. Consumers must handle duplicates, usually by recording processed event IDs.

Polling or CDC: which relay should I use? Polling is simpler and fine for moderate volumes. CDC gives lower latency and commit-order delivery, and is worth it at higher volume or when many services use outboxes.

Do I still need the outbox if I use event sourcing? Not when the event store itself is the source that consumers subscribe to. You need it whenever a service stores state in tables and must also publish events.

How big does the outbox table get? As big as you let it. Delete published rows promptly or partition the table by time and drop old partitions.

Start free — one seat, no credit card.