Load Balancing

Spreading requests across a fleet. The strategies only diverge once requests stop costing the same.

Distributing requests evenly is not the same as distributing work evenly.

round robinleast connectionsweighted

Try it

Move the dials — the sentence under the picture changes.

Strategy

Request cost spreadidentical

With every request costing the same, all four strategies land in the same place — the choice is invisible. Raise the cost spread to make requests unequal, which is what real traffic looks like.

In plain words

One server can only do so much. When you run several copies of a service, a load balancer stands in front and decides which copy handles each request. It is the receptionist at a clinic with six doctors: every patient walks up to one desk, and the desk sends them to whoever is free.

Round robin sends the same share to every server whatever state it is in — including the one that is stuck.

The strategies

The simple demos all look the same because they use identical, instant requests. The strategies only differ when requests are uneven or servers are unequal — which is always, in production.

Round robin

simplest

Requests in turn: 1, 2, 3, 4, 1, 2… No state, nothing needed from the servers, perfectly even request counts. Blind to a request that takes 2 ms versus 4 seconds.

Least connections

usual pick

Send to whichever server currently holds the fewest open requests. The only common strategy that reacts to what is happening: a server that drew the slow report stops receiving new work until it catches up.

Weighted

"Server A is twice the machine server D is." Fixes mixed hardware, not mixed requests. Often combined with the two above.

Random

Round robin without the counter. Same distribution over enough requests, lumpier over few — but needs no shared state, which matters when the balancer is itself a fleet.

Consistent hash

sticky

Hash a key (user id, session) so the same key always lands on the same server. Good for local caches; bad when a server dies and its users all move at once.

The same choice in a real config (nginx)YAML

upstream api {
  least_conn;                      # react to what servers are actually doing
  server 10.0.0.11:8080 weight=2;  # twice the machine
  server 10.0.0.12:8080;
  server 10.0.0.13:8080;
  server 10.0.0.14:8080 max_fails=3 fail_timeout=;
}

Two kinds of balancer

	Layer 4 (transport)	Layer 7 (application)
Sees	TCP connections — bytes in, bytes out	The HTTP request itself
Can	Forward fast and cheap	Route by path or header, retry a safe request, split traffic 90/10 for a canary, terminate TLS
Costs	Almost nothing per request	Real work per request; must hold the TLS certificates
Examples	AWS NLB, HAProxy in TCP mode	AWS ALB, nginx, Envoy

Most services sit behind an L7 balancer because routing by path (/api here, /static there) is worth the cost. L4 is for raw throughput or protocols that are not HTTP.

Health checks are the real feature

Spreading load is table stakes. The most valuable thing a balancer does is stop sending traffic to a server that is broken. That makes the health check the important decision.

Too shallow

GET /ping returns 200 as long as the process is up. A server whose database connection pool is exhausted still says "fine" and keeps getting traffic — and failing every request.

Too deep

GET /health checks the database, the cache and two downstream services. One shared dependency blips and every server fails its check at once. The balancer pulls the whole fleet and serves nothing.

A health check that reports on the server, not the worldTypeScript

app.get("/health", async (req, res) => {
  const poolOk = db.pool.idleCount + db.pool.waitingCount < db.pool.max; // not exhausted
  const diskOk = (await checkDiskFree("/tmp")) > 100 * MB;
  if (poolOk && diskOk) return res.status(200).send("ok");
  return res.status(503).send("unhealthy");   // balancer stops routing here
});

Where it goes wrong

Sticky sessions. Pinning a user to one server (because the session lives in that server's memory) makes that server's death a user-visible logout and makes the pool impossible to drain for a deploy. Move session state to a shared store and let any server serve anyone.
Thundering herd on recovery. A server rejoins with an empty cache and immediately gets its full share. It is slower than the others, so its queue grows, so it fails its health check, so it is removed — and the cycle repeats. Slow start ramps a returning server from 0% to its share over a minute.
The balancer is the new single point of failure. Run at least two, and put DNS or anycast (one address announced from several places, so the network routes each client to the nearest) in front of them.
Retries at the balancer. Retrying a POST that timed out may run it twice. Retry only requests that are safe to repeat.

Take this with you

The one idea: spreading requests is the easy part. Health checks and how a recovering server rejoins decide whether a bad hour becomes an outage.
In an interview, name round robin vs least connections and say when the difference shows. Mention L4 vs L7 and what a health check should test.
At work, read your health check. If it only opens a port, a server with a dead database connection still gets traffic.