Notification System

In plain words

A notification system takes "something happened" from any service and turns it into a push, an email or a text — the right one, in the right language, not too often, and never twice. Its real constraints are outside your control: the providers that actually deliver have quotas and latencies you cannot change, so the design is built around waiting for them gracefully.

The numbers

Assume 3,000 events/s average across all services, five times that when a campaign fires, and a mix of roughly 70% push, 25% email, 5% SMS.

events, average — 15 K in a campaign

3,000 /s

a push provider call

200 ms

an email provider call

300 ms

an SMS provider call

500 ms

Worked the way back-of-envelope estimation describes: a few facts, a few multiplications, and the big number tells you the shape of the problem.

Those provider numbers are the design. Push is fast and nearly free. Email has a sending quota per account. SMS is slow, expensive, and capped per phone number. Nothing you own is the bottleneck; the third parties are.

Decision 1: accept fast, send later

The API validates the event, checks its idempotency key (a caller-chosen id that makes a repeated request harmless) against a short-lived set in Redis, and puts it on a queue. Everything slow — loading preferences, rendering the template, checking the per-user rate limit, calling the provider, recording the attempt — happens in the delivery workers.

Decision 2: never twice

Duplicates come from two places: the caller retrying the API call, and the worker retrying the provider call. Both are handled the same way — an idempotency key that travels the whole way.

1. The caller supplies a key

   order-123-shipped. The API records it with a 24-hour TTL; a second call with the same key is acknowledged and dropped.

2. The worker passes it to the provider

   As the provider's client-side message id. Good providers deduplicate on it, so a retry after a timeout does not send twice.

3. The delivery log is the last line

   When a provider offers no such id, check for a successful attempt before sending — and accept that a crash between "sent" and "logged" can still produce one duplicate.

app.post("/events", async (req, res) => {
  const { key, userId, type, data } = req.body;
  // SET NX = only if absent. Returns null when the key already existed.
  const fresh = await redis.set(`idem:${key}`, "1", { NX: true, EX:  });
  if (fresh === null) return res.status(202).json({ deduplicated: true });

  await queue.publish(channelFor(userId, type), { key, userId, type, data });
  res.status(202).json({ queued: true });
});

Decision 3: workers wait, they do not work

A delivery worker spends almost all of its time waiting on a provider: 200 ms for push, 300 for email, 500 for SMS. Its capacity is therefore not CPU but concurrent calls — how many provider requests it can have in flight at once. Six workers holding 2,000 calls each is 12,000 in flight, which at 300 ms a call is 40,000 sends per second of headroom.

Slow the email provider down with the slider and watch: the rate does not change, but the workers fill up. This is the failure that surprises people, because CPU looks fine the whole way down.

import asyncio

MAX_IN_FLIGHT = 2000                       # per worker: the real capacity number
slots = asyncio.Semaphore(MAX_IN_FLIGHT)

async def deliver(event):
    async with slots:                      # blocks when 2,000 calls are already waiting on providers
        prefs = await prefs_cache.get(event.user_id)
        if not prefs.allows(event.channel): return
        body = templates.render(event, prefs.locale)
        await provider(event.channel).send(body, client_id=event.key)
        await delivery_log.record(event.key, "sent")

Where this design breaks

• Per-user rate limits across workers. "No more than 5 an hour" needs a shared counter, which is one more Redis round trip per event. Fold it into the preferences read.
• Fan-out events. "Everyone in this group" is one event and ten thousand sends. Expand at enqueue time, not in the worker, or one event blocks a partition (one lane of the queue) for everyone behind it.
• Provider quotas at peak. An email account allowed 2,000/s does not care that it is Black Friday. Warm up limits ahead of campaigns, or spread across accounts.
• Templates in the hot path. Rendering with a locale (language and region) lookup per event is fine at 3,000/s and not at 100,000/s; cache compiled templates per locale.

Take this with you

• The one idea: accept fast, send later. A queue turns a slow or broken provider into a delay instead of an outage.
• In an interview, cover the idempotency key end to end, one queue per channel, and why workers are sized by calls in flight rather than CPU.
• At work, alarm on queue depth, not on worker CPU — that graph looks fine all the way down.