Raft Consensus Explained: Leader Election and Log Replication

Every Kubernetes cluster stores its state in etcd, and etcd agrees on that state with Raft. So do Consul, CockroachDB, TiKV, RabbitMQ quorum queues and, since Kafka 4.0 removed ZooKeeper, Kafka's own controllers. If you run any of them, Raft explains behavior that otherwise looks arbitrary: why a three-node cluster stops accepting writes when two nodes fail, why writes pause for a second during a leader change, and why a fourth node makes a cluster more expensive without making it safer.

This post explains Raft with a diagram of log replication in a five-node cluster: terms, leader election, AppendEntries, the commit rule, failures, cluster sizing, reads and membership changes.

The model: a replicated log

Raft builds a replicated state machine. Every server keeps a log of commands. If all servers apply the same commands in the same order, they end up in the same state. Raft's job is to keep the logs identical, even when servers crash and messages are lost or delayed.

Each server is in one of three states: follower, candidate or leader. Time is divided into terms, numbered 1, 2, 3 and so on; each term starts with an election and has at most one leader. A server that sees a higher term than its own updates its term and becomes a follower, so a stale leader steps down as soon as it hears from the newer one.

Raft needs only two RPCs, plus one for snapshots:

The design comes from the paper In Search of an Understandable Consensus Algorithm by Diego Ongaro and John Ousterhout, published at USENIX ATC 2014.

Leader election

A follower expects regular heartbeats from the leader. If none arrives within its election timeout, it assumes the leader is gone:

1. It increments its term, becomes a candidate and votes for itself.
2. It sends RequestVote to every other server, with the index and term of its last log entry.
3. Each server grants at most one vote per term, and only to a candidate whose log is at least as up to date as its own: a higher last term wins, and for equal terms a longer log wins.
4. A candidate that collects votes from a majority becomes leader and starts sending heartbeats.

If two candidates split the vote, nobody wins, and the term ends without a leader. Raft makes this rare by randomizing the timeouts; the paper suggests values such as 150 to 300 ms, so one server usually times out first and wins before others start.

Production systems pick timeouts from their network. etcd defaults to a 100 ms heartbeat and a 1,000 ms election timeout, and its documentation says the election timeout should be at least ten times the round trip between members. Those numbers are why a Kubernetes control plane can take a second or more to accept writes again after losing its etcd leader.

Log replication

Once elected, the leader handles every write:

1. A client sends a command to the leader, for example SET x=5.
2. The leader appends it to its own log as a new entry with the current term, say index 7 in term 3.
3. The leader sends AppendEntries to every follower with the new entry, plus the index and term of the entry just before it (index 6, term 3).
4. A follower accepts only if its log has an entry at index 6 with term 3. This consistency check guarantees that if two logs agree at one position, they agree on everything before it.
5. When a majority of servers, the leader included, stores the entry, the leader marks it committed, applies it to its state machine and answers the client.
6. Followers learn the new commit index from the next AppendEntries and apply the entry too.

If a follower rejects an entry because its log diverges, the leader steps back one index at a time, or faster with optimizations, until the logs match, then overwrites the follower's conflicting entries with its own. Only uncommitted entries are ever overwritten, which is safe because nobody was told they succeeded.

The commit rule, and the subtle part

"Committed once a majority stores it" has one exception, and it is the most famous detail in the paper. A new leader must not count replicas to commit entries left over from earlier terms. It commits them indirectly, by committing an entry from its own term on top of them. Without this rule, an entry stored on a majority could still be overwritten by a later leader. In practice, a new leader appends an empty no-op entry at the start of its term to commit everything before it quickly.

The election restriction completes the safety argument: because a candidate needs votes from a majority, and every committed entry is on a majority, every possible winner already has every committed entry.

What a cluster survives

A cluster of n servers needs a majority to elect a leader and commit writes, so it tolerates the failure of fewer than half:

Four servers tolerate one failure, like three, but every write now waits for three acknowledgements instead of two. That is why clusters have odd sizes. Five is the usual choice when you want to survive one failure during planned maintenance on another.

Latency follows the same arithmetic. Every write waits for the leader to reach the fastest majority and for those servers to persist the entry to disk. In a cluster spread across three regions, each write costs at least one round trip to the nearest other region. Slow disks hurt as much as slow networks, which is why etcd's documentation spends as much time on fsync latency as on timeouts.

Failures in practice

• The leader crashes. Followers time out, one wins an election, and writes resume. Requests in flight may or may not have been committed, so clients must retry, and the commands they retry should be idempotent.
• A network partition. The side with a majority elects a leader and continues. An old leader on the minority side cannot commit anything; once the partition heals, it sees the higher term and steps down.
• A slow follower. Commits do not wait for it. If it falls far behind, the leader sends a snapshot instead of replaying the whole log.
• A disruptive server. A server cut off from the leader keeps timing out and raising its term, which can force needless elections when it returns. Pre-vote, from Ongaro's dissertation, lets a candidate check that it could win before it increments its term.

Reads are not free

A leader cannot answer reads from its own state blindly: a new leader may have been elected on the other side of a partition, and the old one would return stale data. Linearizable reads need either a round of heartbeats to confirm leadership before answering (Raft's ReadIndex) or a time-based leader lease, which depends on bounded clock drift. Followers can serve reads too, but only after checking the leader's commit index, or by accepting stale results. etcd makes this explicit: etcdctl get is linearizable by default, and --consistency=s asks for a faster, possibly stale serializable read.

Changing membership

Adding or removing servers changes what a majority is, and switching configurations carelessly can create two majorities at once. Raft handles this with joint consensus or by changing one server at a time. Many implementations also add new servers as non-voting learners, as etcd does, so that a fresh node catches up on the log before it counts toward the majority.

The log replication diagram

How to read it:

• The client sends one command to the leader, S1, in term 3. The leader writes it at index 7.
• Each table is a server's log: index, term and command. Matching rows across tables are the same entry.
• AppendEntries edges go from the leader to the four followers. S2 and S3 acknowledge index 7.
• S4 lags at index 6 and S5 is unreachable at index 5. Neither blocks the commit.
• The note shows the commit rule: three of five servers store entry 7, so it is committed and the client gets its answer.

Drawing consensus diagrams that teach

• Show the logs, not just the servers. Raft's guarantees are statements about logs. A diagram with five boxes and arrows hides everything interesting.
• Write the term next to every entry. Terms explain elections, conflicts and the commit rule.
• Draw an unhealthy cluster. A lagging or partitioned follower shows why majorities matter better than a perfect cluster.
• Use one diagram per scenario: an election, normal replication and a conflict repair after a failover each deserve their own picture.

For how replication fits into database architectures more broadly, see the database replication diagram. The Kubernetes architecture diagram shows where etcd sits in a control plane, and the Kafka architecture diagram covers KRaft controllers. Systems such as CockroachDB run one Raft group per range of data, which connects consensus to the ideas in the database sharding diagram.

Build it in Archboard

1. Sign in with Google or GitHub and create a diagram.
2. Add a Core Frame for the cluster and label it with the current term.
3. Add five Core Table nodes inside it, one per server, with columns for index, term and command. Put the leader in the middle.
4. Add a Core Box for the client outside the frame and connect it to the leader with the command.
5. Draw AppendEntries edges from the leader to each follower, with dashed acknowledgement edges back from the followers that stored the entry. Mark the unreachable follower with a dashed edge in a warning color.
6. Add a Markdown block from the Core pack with the commit rule, and a Note next to the leader stating the commit index.

The example uses 8 diagram nodes, within the Free plan.

FAQ

Why do Raft clusters have an odd number of nodes? A cluster needs a majority to work. Adding a fourth node to three raises the majority from two to three without tolerating more failures, so even sizes cost latency and money without adding fault tolerance.

What happens to writes during a leader election? They wait or fail until a new leader is elected, typically within one or two election timeouts. Clients should retry with idempotent requests, because a write that timed out may have been committed.

What is the difference between Raft and Paxos? They solve the same problem with similar guarantees. Raft was designed to be easier to understand and implement: it uses a strong leader, separates election from replication and specifies membership changes.

Can followers serve reads in Raft? Yes, if they confirm the leader's commit index first, or if the application accepts possibly stale data. Reading a follower's local state without a check can return old values.

Does Raft protect against malicious nodes? No. Raft tolerates crashes, restarts and lost or delayed messages, but it assumes servers follow the protocol. Byzantine fault tolerance needs different algorithms and more replicas.

Start free — one seat, no credit card.