Raft Consensus: How Databases Agree on Data

Why this matters: Five Servers, One Source of Truth

You run a distributed key-value store across 5 nodes in 3 data centers. A client writes SET user:123 "Alice" to Node 1. Simultaneously, another client writes SET user:123 "Bob" to Node 3.

Without a consensus protocol, both writes succeed locally. Node 1 thinks user:123 is “Alice.” Node 3 thinks it’s “Bob.” When a read request arrives, the answer depends on which node handles it. You now have inconsistent data across your cluster with no way to determine which value is correct.

This is the fundamental problem of distributed consensus: getting multiple servers to agree on a single sequence of operations, even when servers crash and networks partition. Every distributed database — etcd, CockroachDB, TiKV, Consul — needs a consensus protocol at its core. Without one, you’re running 5 independent databases that happen to share a network.

Raft solves this by electing a single leader that serializes all writes. Every write goes through the leader, which replicates it to a majority of followers before committing. If the leader crashes, the remaining nodes elect a new leader within 150–300ms.

The interactive visualization walks through 16 stages — from initial election to leader failure, network partitions, split-brain pressure, and log rollback — with animated RequestVote and AppendEntries RPCs flowing between 5 nodes.

My rule of thumb Consensus is about choosing availability boundaries deliberately. A cluster that accepts writes everywhere during a partition is not highly available; it is inconsistent.

The mental model: Leader Election and Log Replication

Raft divides consensus into two subproblems: leader election and log replication.

Leader Election: All nodes start as Followers. Each follower sets a randomized election timeout between 150ms and 300ms. If a follower doesn’t receive a heartbeat from the leader before its timeout expires, it becomes a Candidate. The candidate increments its term number (a monotonically increasing epoch), votes for itself, and sends RequestVote RPCs to all other nodes.

Node 3 election timeout expires (200ms with no heartbeat)
Node 3 → Candidate (term 2)
Node 3 → Node 1: RequestVote(term=2, candidateId=3, lastLogIndex=5, lastLogTerm=1)
Node 3 → Node 2: RequestVote(term=2, candidateId=3, lastLogIndex=5, lastLogTerm=1)
Node 3 → Node 4: RequestVote(term=2, candidateId=3, lastLogIndex=5, lastLogTerm=1)
Node 3 → Node 5: RequestVote(term=2, candidateId=3, lastLogIndex=5, lastLogTerm=1)

A node grants its vote if: (1) it hasn’t voted in this term yet, and (2) the candidate’s log is at least as up-to-date as its own. If the candidate receives votes from a majority (3 of 5 nodes), it becomes the Leader and immediately begins sending heartbeats to prevent new elections.

Log Replication: Once elected, the leader handles all client writes. It appends the entry to its local log, then sends AppendEntries RPCs to all followers. When a majority (3 of 5) acknowledge the entry, it’s committed. Committed entries are durable — they survive leader crashes because at least 3 nodes have the entry.

Client → Leader (Node 3): SET user:123 "Alice"
Leader appends to log: {index: 6, term: 2, command: SET user:123 "Alice"}
Leader → Node 1: AppendEntries(entries=[{index:6, term:2, ...}])
Leader → Node 2: AppendEntries(entries=[{index:6, term:2, ...}])
Leader → Node 4: AppendEntries(entries=[{index:6, term:2, ...}])
Leader → Node 5: AppendEntries(entries=[{index:6, term:2, ...}])

Node 1: ACK ✓
Node 2: ACK ✓  → Majority reached (3/5), entry committed
Node 4: ACK ✓
Node 5: (slow, ACK arrives later)

The visualization animates this entire flow: watch the RequestVote and AppendEntries RPCs fly between nodes, see log entries appear and commit across the cluster, and observe the state transitions from Follower to Candidate to Leader.

Hint The visualization’s Stage 5 shows log replication in action. Watch how the leader waits for a majority of ACKs before committing — and what happens when one node is slow to respond.

Decisions that matter

• Election Timeout Range (150–300ms): The randomized range prevents multiple nodes from starting elections simultaneously. Narrow ranges (150–180ms) cause faster elections but increase the chance of collisions (two candidates at once). Wide ranges (150–500ms) reduce collisions but extend the leader-down window. The Raft paper recommends 150–300ms as the sweet spot.

• Heartbeat Interval (50–150ms): How frequently the leader sends empty AppendEntries RPCs to maintain authority. Must be significantly less than the minimum election timeout — typically 10x smaller. At 50ms, the leader sends 20 heartbeats per second per follower (80 RPCs/s for a 5-node cluster). At 150ms, network jitter can occasionally cause a missed heartbeat to look like a leader failure.

• Cluster Size (3, 5, 7 nodes): Determines fault tolerance. A 3-node cluster tolerates 1 failure (needs 2 of 3 for majority). A 5-node cluster tolerates 2 failures (needs 3 of 5). A 7-node cluster tolerates 3 failures (needs 4 of 7). But every additional node increases write latency because the leader must wait for more ACKs. The visualization uses 5 nodes and shows what happens when 1 or 2 fail.

• Log Compaction Threshold (1,000–100,000 entries): After this many log entries accumulate, the node takes a snapshot and discards old log entries. Low thresholds (1,000) keep memory usage minimal but trigger frequent snapshots that consume CPU. High thresholds (100,000) reduce snapshot frequency but require more memory and make new node catch-up slower.

• Pre-Vote Protocol (on/off): Before starting a real election, a candidate sends a pre-vote request to check if it would win. If it can’t get a majority of pre-votes, it doesn’t increment the term number. This prevents disrupted nodes from incrementing terms needlessly and causing unnecessary leader step-downs.

Where it breaks: Election Storms From Unstable Networks

The failure mode: Your 5-node Raft cluster runs across two data centers connected by a WAN link with intermittent 200ms latency spikes. The election timeout is set to 150–300ms.

During a network hiccup, Node 4 and Node 5 miss the leader’s heartbeat and both start elections simultaneously. Node 4 increments to term 3, Node 5 increments to term 3. They split the votes: Node 4 gets votes from Node 1 and Node 4 (itself). Node 5 gets votes from Node 2 and Node 5 (itself). Node 3 (the old leader) hasn’t voted.

Neither candidate reaches majority. Both timeout and restart elections with term 4. The cycle repeats — terms increment rapidly, no leader is elected for seconds or even minutes. This is an election storm.

What you will notice: The term number increments rapidly — you see terms jumping from 3 to 15 to 40 within seconds. etcdctl endpoint status shows no leader. Write operations fail with no leader errors.

Client-side latency spikes as requests queue waiting for a leader. Prometheus metric etcd_server_leader_changes_seen_total shows dozens of changes per minute.

# Check for election storms in etcd
etcdctl endpoint status --write-out=table
# ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | RAFT TERM
# All show IS LEADER = false and rapidly incrementing RAFT TERM

# Check leader change rate
curl -s http://localhost:2379/metrics | grep etcd_server_leader_changes_seen_total
# Value jumping by 10+ per minute = election storm

What I would do: Increase the election timeout range to 300–500ms to account for WAN latency. Enable the pre-vote protocol (--pre-vote=true in etcd) so that nodes don’t increment terms unless they can actually win.

Set heartbeat interval to at least 100ms for WAN clusters. Randomized timeouts are Raft’s primary defense against election storms — the interactive visualization shows how different random values cause one candidate to start slightly earlier and win before the other starts.

Warning Running Raft across data centers with high-latency links requires tuning the election timeout well above the default 150–300ms range. Failing to account for WAN latency is the leading cause of election storms.

Where it breaks: Split-Brain Partitions

The failure mode: A network partition divides your 5-node cluster into two groups: {Node 1, Node 2} and {Node 3, Node 4, Node 5}. Node 1 was the leader. After the partition, Node 1 can still send heartbeats to Node 2 but not to Nodes 3–5.

On the other side, Nodes 3–5 stop receiving heartbeats from Node 1, trigger an election, and Node 3 wins with 3 votes (majority). Now there are two “leaders”: Node 1 (old term, minority partition) and Node 3 (new term, majority partition).

Clients connected to Node 1 continue sending writes. Node 1 accepts them into its log but can never commit them — it can only get acknowledgment from Node 2 (2 of 5, not a majority). These writes are “uncommitted.” Meanwhile, Node 3 accepts writes and commits them normally with 3 nodes.

When the partition heals, Node 1 discovers Node 3’s higher term, steps down, and its uncommitted entries are rolled back and replaced with Node 3’s log.

What you will notice: During the partition, writes to the minority side appear to succeed (the leader accepts them) but never commit. Read-after-write consistency breaks — a client writes to Node 1, then reads from Node 3 and doesn’t see its write.

After partition heals, some writes “disappear” — they were rolled back. The visualization’s network partition stage shows this exact scenario across 5 nodes.

# Detect partition: check commit index progress
etcdctl endpoint status --write-out=json | jq '.[].Status.raftIndex'
# Nodes in minority partition: raftIndex stops advancing
# Nodes in majority partition: raftIndex continues advancing

# After healing: check for log rollback
journalctl -u etcd | grep "raft log rollback"
# Entries that were accepted but not committed are discarded

What I would do: This is actually Raft working correctly — the majority partition maintains availability and consistency while the minority partition correctly refuses to commit. To minimize data loss from uncommitted entries on the minority side, configure clients with linearizable reads that require leader confirmation.

Deploy nodes across 3 availability zones (2+2+1) so a single AZ failure doesn’t create an even split. The interactive visualization walks through the full partition-heal-rollback sequence across stages 12–16.

Where it breaks: Log Divergence During Leader Changes

The failure mode: Leader Node 1 (term 3) appends entries at indices 6, 7, and 8 to its log but crashes before replicating index 7 and 8 to any follower. Only index 6 was replicated to a majority and committed.

Node 3 wins the next election (term 4) and starts appending new entries at indices 7 and 8 with term 4. When Node 1 recovers, its log has entries {6:term3, 7:term3, 8:term3} while Node 3’s log has {6:term3, 7:term4, 8:term4}. The logs have diverged at index 7.

What you will notice: The recovering node fails AppendEntries consistency checks. The new leader’s AppendEntries RPC includes prevLogIndex=6, prevLogTerm=3 — which matches — but the follower already has different entries at indices 7 and 8.

The leader detects the conflict and sends entries progressively further back until it finds the matching point. Logs show "log entries conflict at index 7, overwriting with leader's entries". Data that was written to the old leader but not committed is permanently lost.

# Log divergence scenario
Node 1 (old leader, recovering):
  [1:t1] [2:t1] [3:t1] [4:t2] [5:t2] [6:t3] [7:t3] [8:t3]
                                                ↑ committed  ↑ uncommitted

Node 3 (new leader, term 4):
  [1:t1] [2:t1] [3:t1] [4:t2] [5:t2] [6:t3] [7:t4] [8:t4]
                                                ↑ committed  ↑ committed

# After healing: Node 1's entries at 7:t3 and 8:t3 are replaced with 7:t4 and 8:t4

What I would do: This is Raft’s safety guarantee in action — uncommitted entries are not durable and can be overwritten. The key insight: clients should only consider a write “successful” when the leader confirms it’s committed (replicated to a majority).

If your application treats the leader’s initial acceptance as success, you risk losing data during leader changes. Use learner nodes (non-voting replicas) for read scaling without affecting quorum math. The interactive visualization shows log divergence and rollback in the later stages.

Hint The visualization’s Stages 14–16 walk through log divergence and rollback step by step. Watch how the recovering node’s uncommitted entries are replaced with the new leader’s committed entries.

The tradeoff

Approach	Pros	Cons
3-node cluster	Minimal resource cost, fastest writes (wait for 2 ACKs)	Tolerates only 1 node failure
5-node cluster	Tolerates 2 failures, standard for production	Higher write latency (wait for 3 ACKs)
7-node cluster	Tolerates 3 failures, maximum fault tolerance	Significantly higher write latency, 7 nodes to manage
Short election timeout (150–300ms)	Fast leader election after failure	Election storms on high-latency networks
Long election timeout (500–1000ms)	Stable elections on WAN clusters	Longer unavailability window when leader fails
Pre-vote protocol enabled	Prevents unnecessary term increments, reduces disruption	Slightly more complex election, extra RPC round trip
Pre-vote disabled	Simpler, faster elections	Disrupted nodes can increment terms and force step-downs
Linearizable reads (via leader)	Strongest consistency guarantee	All reads go through leader, higher latency
Serializable reads (any node)	Lower read latency, distribute read load	May return stale data from lagging followers

A practical checklist

Step 1: Check cluster health. Run etcdctl endpoint status to see which node is leader, what term the cluster is on, and whether all nodes are healthy. If no leader exists, check for election storms (rapidly incrementing term numbers).

# Quick cluster health check for etcd
etcdctl endpoint health --endpoints=http://node1:2379,http://node2:2379,http://node3:2379
etcdctl endpoint status --write-out=table

Step 2: Diagnose leader election failures. If the term number is incrementing rapidly, check network connectivity between all pairs of nodes. Run ping and measure latency. If round-trip time exceeds your election timeout minimum, increase the timeout range.

Step 3: Check for split-brain after partition recovery. After a network partition heals, verify all nodes converge to the same leader and term. Check each node’s raftIndex — they should converge within seconds. If any node’s log is being rolled back, that’s expected and correct.

# Verify convergence after partition
for node in node1 node2 node3 node4 node5; do
  echo "$node: $(etcdctl endpoint status --endpoints=http://$node:2379 --write-out=json | jq '.[] | {leader: .Status.leader, term: .Status.raftTerm, index: .Status.raftIndex}')"
done

Step 4: Monitor committed vs applied index lag. If raft_applied_index lags significantly behind raft_committed_index, the state machine is falling behind. This could indicate disk I/O bottlenecks or CPU saturation. Check iostat and top on the affected node.

Step 5: Add nodes safely. Never add multiple nodes simultaneously — each new node changes the quorum requirement. Add one node at a time as a learner (non-voting member) first, let it catch up on the log, then promote it to a voting member. This prevents the new node’s empty log from affecting leader elections.

Try it yourself

The visualization animates the full Raft protocol across 16 stages. Watch RequestVote RPCs fly between nodes during elections. See AppendEntries RPCs replicate log entries from leader to followers.

Kill the leader and watch a new election happen with randomized timeouts. Create a network partition and observe split-brain pressure, uncommitted entries, and log rollback on healing.

Try the interactive Raft Consensus visualization — step through leader election, log replication, network partitions, and split-brain recovery across 5 animated nodes.