Kafka Architecture Diagram: Brokers, Partitions, Replicas and Consumer Groups

Kafka is easy to draw as one box labeled "Kafka" with arrows in and out, and that box hides every property people rely on: what is ordered and what is not, how many copies of the data exist, what happens when a broker dies, and how several services can read the same stream without stepping on each other.

This post draws a Kafka cluster with the parts that matter: brokers, topics and partitions, leader and follower replicas, producers with keys, consumer groups and offsets, and the KRaft controllers that manage the cluster.

Topics, partitions and brokers

A topic is a named stream of records, such as orders. Each topic is split into partitions, and each partition is an append-only log where every record gets an increasing offset.

Partitions are the unit of everything:

• Ordering: records are ordered within a partition, never across partitions.
• Parallelism: a topic can be consumed by at most as many consumers in one group as it has partitions.
• Placement: partitions are spread across brokers, the servers of the cluster.

Pick the partition count from the throughput you need per topic and the number of consumers you want to run in parallel, with headroom. Adding partitions later is possible, but it changes which partition a key maps to.

Replication

Each partition has a replication factor, usually 3. One replica is the leader and handles reads and writes; the others are followers that copy the leader's log. Followers that are caught up form the in-sync replica set (ISR). If a leader's broker fails, a caught-up follower becomes the new leader.

The settings that decide durability:

With factor 3, min.insync.replicas=2 and acks=all, an acknowledged write exists on at least two brokers, and the cluster keeps accepting writes with one broker down. The Kafka documentation covers these settings and their interactions.

Producers and keys

Producers choose a partition for every record. With a key, the default partitioner hashes the key, so all records with the same key land in the same partition and stay in order. With no key, records are spread for balance and lose any ordering between them.

Choose keys from the entity whose events must stay in order: an order ID, an account ID. Write the key on the producer in the diagram; it is the single most consequential setting a producer has. Avoid keys with a few huge values, because one hot key overloads one partition no matter how many you have.

Retries can duplicate records. Recent Kafka versions enable idempotent producers by default, which removes duplicates caused by producer retries within a partition; duplicates from your own application retries are still possible, so consumers should be idempotent too.

Consumer groups and offsets

A consumer group is a set of consumers sharing the work for one purpose. Kafka assigns each partition to exactly one consumer in the group, so three partitions can keep at most three consumers busy; a fourth would sit idle. When consumers join or leave, the group rebalances and partitions move.

Each group tracks its own position, the committed offset per partition, stored by Kafka in an internal topic. That is how several services read the same stream independently: fulfillment and analytics each have their own group and their own offsets, and neither affects the other.

Commit offsets after processing, not before, and you get at-least-once processing: after a crash, the consumer reprocesses records since the last commit. Make processing idempotent so the repeats are harmless.

Retention and compaction

Kafka keeps records after they are read. Time- or size-based retention deletes old segments, say after 7 days. Log compaction instead keeps the latest record per key forever, which turns a topic into a changelog of current state. Pick per topic: events usually use retention, state such as "latest address per customer" uses compaction.

A worked sizing example

Say checkout produces 3,000 order events per second at peak, about 1 KB each, so 3 MB/s. One consumer in the fulfillment service can process 400 events per second. You need at least 8 consumers in parallel at peak, so at least 8 partitions; choose 12 to leave room for growth and a failed consumer's partitions. With replication factor 3, the cluster writes 9 MB/s to disk across brokers, and 7 days of retention at that rate would store up to about 5.4 TB of replicated data. Average traffic is lower than peak, so treat that as the ceiling for disk planning. Numbers like these belong next to the diagram, because they decide the broker count and disks, not the box labeled Kafka.

KRaft: no more ZooKeeper

Older clusters stored their metadata in ZooKeeper. Kafka now manages metadata itself with a Raft-based quorum of controllers, called KRaft. Kafka 4.0, released in March 2025, is the first major release that runs entirely without ZooKeeper. On a diagram, draw the controller quorum separately from the brokers, even when small clusters run both roles on the same machines.

The architecture diagram

How to read it:

• Left: two producers send events keyed by order ID. Order 8812 hashes to partition 0, order 9001 to partition 2.
• Middle: three brokers, each the leader for one partition and a follower for the other two. Dashed edges show replication; the KRaft controllers below manage metadata and leader election.
• Right: the fulfillment group has three consumers, one per partition. The analytics consumer belongs to a different group, so it reads every event with its own offsets.
• The Markdown block lists the topic settings that the picture depends on.

What to put on a Kafka diagram

• Partitions per topic and the key, because they define ordering and parallelism.
• Replication factor and min.insync.replicas, because they define durability.
• Each consumer group by name, with its consumers.
• Retention or compaction per topic.
• Who produces and who consumes, across services, so ownership is clear.

For the patterns built on top of Kafka, see the event-driven architecture diagram, the transactional outbox pattern for publishing reliably from a database, and the data pipeline architecture diagram for Kafka feeding analytics.

Build it in Archboard

1. Sign in with Google or GitHub and create a diagram.
2. Add Container nodes from the C4 Model pack for the producers, with their key in the description.
3. Add three Kafka nodes from the Data pack for the brokers, labeled with the partitions each leads and follows, and a Container for the KRaft controllers.
4. Add a Frame from the Core pack for the consumer group with one Container per consumer, plus a separate consumer for the second group.
5. Draw replication as dashed edges and add a Markdown block from the Core pack with the topic settings.

The example uses 12 diagram nodes, within the Free plan.

FAQ

Does Kafka guarantee message order? Within a partition, yes. Across partitions, no. Use a key so that records that must stay in order share a partition.

How many partitions should a topic have? Enough for the throughput you need and the number of consumers you want in parallel, with room to grow. Changing the count later changes how keys map to partitions.

What happens when a broker fails? For each partition it led, a caught-up follower on another broker becomes the leader. With replication factor 3 and min.insync.replicas=2, writes continue.

What is a consumer group? A set of consumers that split a topic's partitions between them. Different groups read the same topic independently, each with its own offsets.

Does Kafka still need ZooKeeper? No. Kafka 4.0 and later run only in KRaft mode, where a quorum of controllers manages cluster metadata.

Start free — one seat, no credit card.