Replication

In plain words

Replication keeps more than one copy of your data, on more than one machine. It buys three things: durability (a disk dying does not lose the data), read capacity (several copies can answer reads), and staying up when a machine goes away. The cost is that copies can disagree, and every replication design is an answer to when they are allowed to.

Leader and followers

One node — the leader (or primary) — accepts all writes and streams its change log to the followers, which apply it in the same order. Reads can go anywhere. This is the shape of Postgres, MySQL, Redis and most managed databases.

The one decision that matters is whether the leader waits for followers before telling the client "done".

-- Default: async. Acknowledge as soon as the leader has it on disk.
SET synchronous_commit = local;

-- For this one transaction (a payment), wait for a standby to have it too.
BEGIN;
SET LOCAL synchronous_commit = on;    -- with synchronous_standby_names set
INSERT INTO payments (order_id, amount) VALUES (42, 50.00);
COMMIT;

Replication lag, and the bugs that look like the database lying

Followers are behind by an amount that varies with load: milliseconds usually, seconds under a heavy write burst. This produces a family of bugs that are really routing bugs.

async function query(userId: string, sql: string) {
  const lastWrite = await redis.get(`last-write:${userId}`);   // set on every write
  const recent = lastWrite && Date.now() - Number(lastWrite) < ;
  const conn = recent ? leader : pickFollower();                // 5 s of leader reads after a write
  return conn.query(sql);
}

Failover

The leader dies. Someone has to notice, choose a new leader, and redirect writes. Every step has a failure mode.

1. Detect

   A heartbeat stops. But a slow leader and a dead one look identical from outside — wait too little and you fail over on a GC pause; wait too long and writes are down for a minute.

2. Elect

   Pick the follower with the most of the log. Needs a majority to agree (see consensus) or two followers may each think they won.

3. Redirect and fence

   Point clients at the new leader — and make sure the old one, if it was only slow, cannot keep accepting writes when it wakes up.

Other shapes

• Multi-leader. A writable node in each region: fast local writes, and write conflicts become routine rather than an emergency. Needs a merge rule for every table.
• Leaderless (Dynamo, Cassandra). Every write goes to N nodes and waits for W of them; every read asks R. No failover because there was no leader; consistency becomes a dial per request. The key-value store problem builds one.

Take this with you

• The one idea: copies can disagree. Every replication setup is a decision about when that is allowed and what a reader is promised.
• In an interview, say leader-follower, async by default, and name the two lag bugs (read-your-writes, monotonic reads) and how routing fixes them.
• At work, find out how far behind your replicas run at peak, and whether anything reads its own write from a follower.