Message Queues

Putting a buffer between a producer and a consumer, and what happens when they disagree about pace.

A queue converts an overload into a delay — it never creates capacity.

backpressurequeue depthconsumer lag

Try it

Move the dials — the sentence under the picture changes.

Producer6 msg/tick

Consumer6 msg/tick

Producer and consumer are matched and the queue is empty, so it adds nothing but a hop. Push the producer above the consumer to see backpressure build.

In plain words

A message queue sits between the part of a system that creates work and the part that does it. The producer drops a message in and moves on; the consumer picks it up when it can. It is the order rail in a restaurant kitchen: the waiter clips the ticket and goes back to the floor, and the cooks work through the tickets at their own pace.

The producer's job ends when the message is accepted. Consumers take from the queue at their own rate.

Three things this buys you:

The producer stops waiting on slow work. Checkout is fast regardless of the email provider.
Bursts are absorbed. A flash sale creates 10,000 orders in a minute; the consumers work through them over ten. Nothing is rejected.
Consumers can restart — deploy, crash, scale — without the producer noticing. The messages wait.

What it does not buy: capacity

Backpressure is the polite word for telling whoever is upstream to slow down. When the buffer is full something has to give: drop new messages, drop old ones, or push back on the producer. Pushing back is usually the right answer, and usually the option nobody built.

What "delivered" means

The producer's message is in the queue. Did the consumer get it? Did it act on it? Every queue makes one of three promises.

At most once

fire and forget

Send and forget. Fast, and a message is lost if the consumer crashes before handling it. Fine for metrics; wrong for orders.

At least once

the default

The consumer acknowledges after it has done the work; anything not acknowledged is redelivered. Nothing is lost — and duplicates are guaranteed: a consumer that crashes after doing the work but before acking will do it again.

Exactly once

you build it

Not something the network delivers. You get it by making the consumer idempotent: doing the work twice has the same effect as once. Systems that advertise it are doing this for you within a boundary.

An at-least-once consumer made safe with idempotencyTypeScript

for await (const msg of queue.consume("orders")) {
  const { orderId } = JSON.parse(msg.body);

  // Have we handled this order already? (a crash after work, before ack)
  const done = await redis.set(`email-sent:${orderId}`, "1", { NX: true, EX:  });
  if (done === null) {            // key existed: duplicate delivery
    await msg.ack();
    continue;
  }

  await sendConfirmationEmail(orderId);
  await msg.ack();                // only now is the message gone
}

At-least-once means a redelivery after a crash. The consumer's memory of what it has done is what stops the second email.

Queue or log?

Two different tools share the word "queue", and picking the wrong one hurts.

	Queue (SQS, RabbitMQ)	Log (Kafka, Kinesis)
A message is	Deleted once consumed	Kept, in order, for days
Consumers	Share the work; each message handled once	Each group reads the whole stream at its own pace (its offset)
Replay	No	Yes — rewind after a bug and reprocess
Ordering	Best effort	Strict within a partition
Use it for	Tasks: send this email, resize this image	Events: everything that happened, for anyone who cares

Partitions are lanes inside a topic. Messages with the same key (say a user id) go to the same lane and stay in order; different lanes are independent, which is what lets many consumers work in parallel. Your choice of key is your choice of what "in order" means.

Same key → same partition → in orderPython

producer.send("user-events", key=user_id, value=event)   # all of one user's events stay ordered

Where it goes wrong

Poison messages. One message that fails every time is redelivered forever, blocking its lane. After N attempts move it to a dead-letter queue — a side queue for the ones that keep failing — and alert.
Ordering assumptions. Most queues promise ordering far more narrowly than people assume, and a retry reorders things by definition. If two messages must be handled in order, give them the same key.
Scaling consumers to hide a bug. Adding consumers to drain a growing backlog works right up until the database they all write to becomes the new bottleneck. The queue was hiding that the work itself was too slow.
Unbounded queues. "It can hold a million messages" means an hour-long delay looks like health. Bound the queue and alert well before the bound.

Take this with you

The one idea: a queue turns overload into delay. It does not add capacity; it buys time and absorbs bursts.
In an interview, say at-least-once plus an idempotent consumer, name the partition key as what defines ordering, and mention a dead-letter queue.
At work, alarm on queue depth and on whether it is growing. A queue that is slowly filling is an outage with a delay on it.