Change Data Capture Architecture: Log-Based CDC with Debezium and Kafka

The orders table lives in Postgres, and now five other places need it: the search index, a cache, the data warehouse, a fraud service and a reporting read model. Writing to all of them from the application fails halfway sooner or later. Polling the table every minute misses deletes and adds load. Log-based change data capture (CDC) takes a different route: it reads the database's own transaction log, the same record the database uses for crash recovery and replication, and turns every committed change into an event.

This post explains log-based CDC with Debezium and Kafka: the approaches compared, how the pipeline works, what a change event contains, snapshots, ordering and delivery guarantees, schema changes, the operational trap that takes down primaries, and when to use the outbox pattern instead, with an architecture diagram.

Four ways to capture changes

Log-based CDC is the only approach that sees every committed change, including deletes, in commit order, without touching the application or slowing down writes. The rest of this post is about it.

How log-based CDC works

Databases already write every change to a log before they apply it: the write-ahead log in Postgres, the binary log in MySQL, the transaction log in SQL Server, the oplog behind MongoDB change streams. A CDC connector reads that log the way a replica would.

For Postgres, that means logical decoding:

-- postgresql.conf: wal_level = logical (requires a restart)
CREATE PUBLICATION orders_pub FOR TABLE orders, order_items;

-- Include all old column values in update and delete events
ALTER TABLE orders REPLICA IDENTITY FULL;

The connector creates a replication slot, which remembers how far it has read, and decodes changes with the pgoutput plugin built into Postgres. By default, update and delete events carry only the old primary key; REPLICA IDENTITY FULL adds the full old row at the cost of a larger log. For MySQL, the equivalent is row-based binary logging, the default in current versions, with full row images.

Debezium on Kafka Connect

Debezium is a widely used open-source CDC engine. Its connectors run inside Kafka Connect, read the log, and publish one event per row change to a Kafka topic per table, keyed by the row's primary key. A minimal Postgres connector configuration:

{
  "name": "orders-cdc",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "plugin.name": "pgoutput",
    "database.hostname": "orders-db",
    "database.port": "5432",
    "database.user": "debezium",
    "database.dbname": "orders",
    "topic.prefix": "shop",
    "table.include.list": "public.orders,public.order_items",
    "publication.name": "orders_pub",
    "slot.name": "orders_cdc",
    "heartbeat.interval.ms": "10000"
  }
}

Changes to public.orders land in the topic shop.public.orders. Keep the password out of the file with a secrets provider. If you do not run Kafka, Debezium Server streams the same events to other brokers and cloud queues.

Anatomy of a change event

Trimmed to its payload, an update event looks like this:

{
  "before": { "id": 1042, "status": "pending", "total": 59.90 },
  "after":  { "id": 1042, "status": "paid",    "total": 59.90 },
  "source": { "db": "orders", "table": "orders", "lsn": 24023128, "txId": 5521 },
  "op": "u",
  "ts_ms": 1793606400123
}

The op field is c for create, u for update, d for delete and r for rows read during a snapshot. A delete is followed by a tombstone, a message with the same key and a null value, so Kafka log compaction can eventually remove the key entirely. The source block records exactly where in the log the change came from, which is what makes deduplication possible downstream.

Snapshots: the rows that already exist

A new connector starts with a table that already has millions of rows, and the log only contains recent changes. The connector therefore takes an initial snapshot, emitting every existing row as an r event, then streams from the log position it recorded when the snapshot began, so nothing falls in the gap.

For large tables, a long blocking snapshot is painful. Incremental snapshots copy a table in chunks while streaming continues, using the watermark technique from Netflix's DBLog paper, as the Debezium team explains. You trigger them through a signaling table, and they resume after a restart instead of starting over.

Ordering and delivery guarantees

• Per row, events stay in order. All events for one primary key share a key, so they land in the same Kafka partition.
• Across rows and tables, there is no global order. Two tables changed in one transaction become events in two topics. If you need transaction boundaries, Debezium can emit transaction metadata events that let consumers regroup them.
• Delivery is at least once. After a crash or restart, the connector may resend some events. Debezium's documentation says so plainly.

The consequence for consumers: make them idempotent. Upsert by key, and ignore an event whose log position is not newer than the last one applied for that key.

Schema changes

Once a table is captured, its schema becomes a contract with every consumer. Adding a nullable column is safe; renaming a column or changing a type breaks someone. Use a schema registry with Avro, Protobuf or JSON Schema and enforce compatibility rules, so a breaking change fails in CI rather than in production. Better still, decide which tables are public. Internal tables can change freely if nothing outside the service consumes them.

The replication slot trap

This is the operational lesson most teams learn the hard way. A Postgres replication slot keeps every WAL segment the connector has not confirmed. If the connector stops, or falls far behind, WAL accumulates on the primary until the disk fills and the database stops accepting writes. The Postgres default for max_slot_wal_keep_size is -1, which means slots may retain an unlimited amount of WAL.

A quieter version of the same problem: the captured tables are idle while other tables are busy. The connector has nothing to confirm, so the slot never advances. Debezium's heartbeat setting exists for exactly this case.

Defenses:

-- How much WAL each slot is holding back
SELECT slot_name, active,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)) AS retained
FROM pg_replication_slots;

• Alert on retained WAL per slot, not just on disk usage.
• Set max_slot_wal_keep_size to a size the disk can afford. A slot that exceeds it is invalidated and the connector must snapshot again, which is far better than a full disk.
• Enable heartbeats, and add the heartbeat table to the publication.
• Drop the slot when you decommission a connector.
• Plan failover. PostgreSQL 17 can synchronize logical slots to a physical standby; check that your setup and connector version use it.

CDC or the outbox pattern?

Capturing tables directly exposes your internal schema as events. The alternative is the outbox pattern: the service writes explicit events to an outbox table in the same transaction as its data, and CDC publishes only that table. Debezium includes an outbox event router for this.

The transactional outbox pattern covers the second column in detail.

The CDC pipeline diagram

How to read it:

• The orders service writes to Postgres as usual; it knows nothing about CDC.
• Debezium in the Kafka Connect frame reads the write-ahead log through its replication slot and publishes change events to the shop.public.orders topic, registering schemas in the registry.
• Four consumers read the topic independently: the search index, the Redis cache, the data warehouse and a read-model service.
• The dashed edge back to Postgres is the slot confirmation, the link that keeps WAL from piling up.
• The Markdown block shows a trimmed change event and the slot lag query.

Drawing CDC pipelines

• Draw the log, not just the database. The slot and the WAL are where CDC succeeds or fails.
• Show the topic names and keys. "Keyed by order ID" explains ordering in four words.
• Draw every consumer separately. Each has its own lag, failure mode and idempotency logic.
• Mark the delivery guarantee on the diagram: at least once.

The Kafka side is covered in the Kafka architecture diagram. CDC is the usual way to feed the raw layer of a lakehouse, shown in the data pipeline architecture diagram, and to build read models for the CQRS and event sourcing diagram.

Build it in Archboard

1. Sign in with Google or GitHub and create a diagram.
2. Add a Compute node from the Cloud pack for the orders service and a Postgres node from the Data pack.
3. Add a Core Frame for Kafka Connect with a Compute node inside for the Debezium connector.
4. Add a Kafka node from the Data pack for the topic and a Core Box for the schema registry.
5. Add the consumers: a Managed DB from the Cloud pack for the search index, a Redis node, a Data Warehouse node and a Compute node for the read-model service.
6. Draw a dashed edge from the connector to Postgres for slot confirmations, label the topic with its key, and add a Markdown block from the Core pack with the event example.

The example uses 11 diagram nodes, within the Free plan.

FAQ

What is log-based change data capture? Reading a database's transaction log, such as the Postgres write-ahead log or the MySQL binary log, and turning each committed change into an event. It captures inserts, updates and deletes in order without changing the application.

Does Debezium guarantee exactly-once delivery? No. During normal operation each change is delivered once, but after failures some events can be delivered again, so Debezium documents at-least-once delivery. Consumers should be idempotent.

Why did my Postgres disk fill up after setting up CDC? A replication slot retains WAL until the connector confirms it. A stopped connector, or captured tables that stay idle while others change, lets WAL accumulate. Monitor slot lag, enable heartbeats and set max_slot_wal_keep_size.

How does CDC handle existing data? With a snapshot. The connector first emits every existing row, then streams changes from the log position recorded at the start. Incremental snapshots do this in chunks while streaming continues.

Should I use CDC or the outbox pattern? Table CDC suits replication, search, caches and analytics. For events between services, the outbox pattern publishes a designed contract instead of internal table rows.

Start free — one seat, no credit card.