Consensus

In plain words

Some things must have exactly one answer across a whole cluster: which node is the leader, whether a lock is held, what the next entry in a shared log is. Consensus is how a group of machines agrees on one answer even when some of them crash, some are slow, and messages get delayed — and, crucially, even when nobody can tell which of those is happening.

The idea is smaller than its reputation

A decision counts only when a majority of nodes has accepted it. Five nodes: three must agree. That one rule does all the work, because:

How Raft makes it concrete

Raft is the protocol you will most likely meet (etcd, Consul, CockroachDB). It turns "majority" into three moving parts: a leader, numbered terms, and a replicated log.

1. One leader per term

   A term is a numbered period with at most one leader. Followers accept writes only from the leader of the current term.

2. A write is committed when a majority has it

   The leader appends the entry to its log and sends it to everyone. Once a majority has written it to disk, it is committed and the client gets an answer. A committed entry can never be lost, because any future majority includes someone who has it.

3. Heartbeats, then an election

   The leader sends heartbeats. If a follower hears nothing for an election timeout (150–300 ms, randomised), it starts a new term with a higher number and asks for votes. A majority of votes makes it leader. A node votes at most once per term — so two leaders in the same term are impossible.

4. An old leader steps down

   A leader that was partitioned away still thinks it leads an old term. When it reconnects it hears the higher term number, steps down, and discards anything it accepted in isolation — nothing it did there was ever committed, because it could not reach a majority.

func (r *Raft) propose(entry Entry) error {
    entry.Term = r.currentTerm
    r.log = append(r.log, entry)
    acks := 1                                   // the leader has it
    for _, peer := range r.peers {
        if r.sendAppend(peer, entry) {          // follower wrote it to disk
            acks++
        }
        if acks > len(r.peers)/2 {              // majority (peers excludes self)
            r.commitIndex = len(r.log) - 1
            return nil                          // safe to answer the client
        }
    }
    return ErrNoMajority                        // could not commit; do not answer OK
}

What it costs, and so where it is used

Every committed write is a round trip from the leader to a majority. Across a region that is a millisecond; across continents it is a hundred. So consensus is used for the small, critical things and not for the bulk data path:

Where it goes wrong

• Losing the majority. Three of five down and the cluster stops — correctly. It cannot tell whether the missing three are dead or off deciding on their own. Add capacity before you lose it.
• Slow disks. A commit waits for a majority to write to disk, so one slow disk in a three-node cluster is on the critical path half the time. Consensus clusters get the fast disks.
• Reading from a follower. It may be behind. A read that must be fresh (linearisable) goes through the leader, or through a lease — a time-bounded promise from a majority that this node is still the leader.
• Running it across regions without thinking. A 100 ms commit is fine for a config change and ruinous for a request path.

Take this with you

• The one idea: a decision counts only when a majority accepts it, and two majorities always overlap — so there can never be two answers.
• In an interview, explain majorities, odd cluster sizes, and why consensus is used for small things (leader, locks, config) rather than bulk data.
• At work, keep the consensus cluster small, odd, and on fast disks — and never run one across a slow link without knowing what a commit costs.