Track 2 · Products people use
mediumChat
Messages between people who are online now, and delivery for the ones who are not. The interesting part is that a connection is a resource you hold, not a request you serve.
Suggested architecture
Scenario
Messages are cheap; connections and fan-out are not. A gateway that holds 50,000 idle sockets is normal — the question is what happens when a fraction of them all send at once, and how many deliveries each message becomes.
Messages arriving from all connected users per second.
Conversations opened per second — each a range read from the store.
Recipients per message. 1 is a direct chat; a 50-person group is 50 deliveries — and 50 push sends when they are offline.
Each holds up to 50,000 sockets. Size for connections first, messages second.
Click a component for its role, common technology choices and tradeoffs, and what it is carrying at this scale. Hover a connection to see what flows along it. Drag to rearrange — layout changes are local and reset on reload.
Every figure here is a rough estimate from simple capacity arithmetic, not a benchmark. Each part carries its own assumption about what one copy can do — real numbers depend on your hardware, payloads and access pattern. The point is which component moves first as you turn the dials, not the digits themselves.
In plain words
A chat system delivers a message from one phone to another in under a second, whether or not the other app is open. The unusual part is that the server has to push — it cannot wait to be asked — which means holding a connection open for every user. Holding a million connections is a different job from answering a million requests.
The numbers
Assume 10 M daily users, each sending 40 messages a day, and 1 M connected at any moment.
- messages, average — 4× in the evening
- 5,000 /s
- open sockets, idle almost all the time
- 1 M
- deliveries: a 50-person group is 50 sends
- × recipients
- sockets one gateway can hold
- 50 K
Worked the way back-of-envelope estimation describes: a few facts, a few multiplications, and the big number tells you the shape of the problem.
The connection number is the one that sizes the gateways. Five thousand messages a second is a modest service; a million open sockets is not, and it costs memory and file descriptors (the operating system's per-connection bookkeeping, which has a hard cap) whether or not anyone types.
How a message flows
Decision 1: connections are held, not served
An HTTP request is served and forgotten. A WebSocket is held: the gateway keeps state per socket (which user, which device, a send buffer) for as long as the connection lives. That changes the sizing unit. A gateway is limited by open connections — call it 50,000 — long before message throughput matters, and a restart drops every one of them at once.
Two consequences: gateways must be stateless (holding nothing a user would miss if the instance vanished) except the socket map, so any of them can take any reconnecting user. And clients must reconnect with backoff and jitter: a deploy that restarts eight gateways in sequence is eight small thundering herds (everyone reconnecting at the same instant), and without jitter they arrive as one big one.
let attempt = 0;
function connect() {
const ws = new WebSocket(GATEWAY_URL);
ws.onopen = () => { attempt = 0; resync(); }; // fetch what was missed
ws.onclose = () => {
const wait = Math.random() * Math.min(, 500 * 2 ** attempt++); // full jitter
setTimeout(connect, wait);
};
}Decision 2: fan out through a queue, not gateway to gateway
The tempting design is for the gateway that receives a message to find the recipient's gateway and forward directly. It works for one-to-one chat with one device per user, and falls apart the moment either stops being true.
Instead the chat service stores the message, publishes it to a queue split into partitions (independent lanes) by conversation, and a fan-out worker does the delivery: look up each recipient in the presence cache, write to the right socket on the right gateway, or hand off to push for anyone not connected.
"Sent" means "stored and queued", which takes a few milliseconds. The sender is never waiting on the slowest recipient.
A gateway that is mid-restart means a failed write; the worker retries. The sender's app knows nothing about it.
All of group 9's messages go through one lane, so they arrive in the order they were stored. Different groups are independent.
Decision 3: presence is a guess with a timer
"Online" cannot be known; it can only be assumed until proven otherwise.
Each connected client heartbeats every 30 s and the gateway refreshes a Redis
key, user → gateway, with a 60 s TTL (time-to-live: the key deletes itself
unless refreshed). Presence is whatever keys have not expired.
SET presence:user:4471 "gateway-7" EX 60 -- on connect, and on every heartbeat
GET presence:user:4471 -- fan-out: online, and where?
-- (nil) after 60 s of silence: treat as offline, send a push insteadThat means a phone that lost signal stays "online" for up to a minute, and a
message sent in that window is written to a socket that no longer exists.
The fix is not better presence; it is that delivery is not the source of
truth. The message is in the store, the client fetches what it missed on
reconnect (resync() above), and the push notification covers the gap.
Where this design breaks
- Big groups. A 10,000-member channel makes every message 10,000 deliveries and 10,000 push sends. Past a few hundred members, switch to pull: notify "new messages", let clients fetch.
- Hot conversations. One partition per conversation means one consumer per conversation; a single viral thread is capped at what one worker can do.
- The push provider. APNs and FCM (Apple's and Google's push services) rate-limit per app. A large offline audience at peak is a third-party limit you cannot raise on the day.
- History reads. "Last 50 messages" is one partition scan. "Search my messages" is not, and needs a separate index.
Take this with you
- The one idea: connections are held, not served. Size gateways by open sockets, keep them dumb, and put the logic one hop back where it scales by messages.
- In an interview, explain WebSocket vs push notification, why fan-out goes through a queue, and why presence is a guess with a TTL.
- At work, test what a rolling restart of the gateways does to reconnects. Without jitter, a deploy is a self-inflicted traffic spike.