Message Queues

12:00 noon, and a phone sale opens on a shopping app. In the first minute, 60,000 buyers tap Pay. The order service calls the stock, payment, receipt and warehouse services in turn and waits for all four. The receipt service, sized for an ordinary day, slows to three seconds a call. Every checkout now waits on it, request threads run out, and buyers who were already charged see an error page. Nothing was wrong with orders or payments. The app needed to accept each order in milliseconds and do the slower work just after: a message queue.

Think of the token counter at a bank: you take a token and sit down, and clerks call the next number when free. In a message queue, the order service, the producer, writes a small message such as "order 881 placed" and moves on. Consumers, such as a receipt worker, read messages and do the work at their own pace. The broker is the server that stores the queue and hands messages out.

On a bad day the two designs behave very differently:

The queue frees checkout from every slower service; the last row is the price, because the result comes later.

The slow receipt service was one problem. What happens when orders arrive faster than the workers can finish them? In the sale's first minute, 1,000 orders a second arrive, and the workers that check stock and take payment finish about 400 a second. Without a queue, the other 600 a second time out. With one, they wait in line like token holders: the backlog grows by 600 a second, then shrinks by a net 300 a second once orders fall back to 100 a second.

The queue turned a spike the workers could not handle into a delay they could. The last order of the rush waited about 90 seconds for confirmation, instead of 36,000 orders failing. A queue trades waiting time for not dropping work, so it fits jobs where a short delay is acceptable.

The arithmetic behind the scene fits in three lines:

A backlog grows at the arrival rate minus the service rate and clears at the reverse, so a queue helps only when the spike ends.

A queue that holds 36,000 orders must also survive its own workers. If worker 1 takes order 881 and crashes halfway through, the order must not vanish. So how does the broker know that a message is really done?

The way a courier does: it waits for a signature. Consumers acknowledge (ack) a message after finishing it, and only then does the broker delete it; a message never acked is delivered again. Workers sharing one queue each take different messages, a pattern called competing consumers, so adding capacity means starting another worker.

That hiding period is the visibility timeout. Log-based brokers, which Part 4 opens up, keep messages after they are read: each consumer group, the set of workers sharing the reading of one stream, saves an offset, its position in the log, and resumes from it after a crash. Either way, work counts as done only when the consumer says so.

The ack closes one gap and opens another, because the moment you ack decides what a crash costs. Ack before the work and a crash loses the message. Ack after the work and a crash between finishing and acking brings the message back, so the work runs twice. Each choice has a name:

Most business work picks the middle row and then upgrades it to the last, because a lost order costs far more than a repeat you can make harmless.

Why not deliver exactly once? When no ack arrives, the broker cannot tell whether the work never happened, or happened and then the consumer died or the ack was lost. The cases look identical, so the only safe move is to deliver again. What you can build is an exactly-once effect: deliver at least once, and make the second delivery do nothing. The next page shows one.

Here is the classic case. Worker A takes order 881, charges the buyer ₹450 and crashes before it can ack. It passed the order id as an idempotency key, the abstractions chapter's tool for safe retries: the payment service remembers each key and the result it gave.

The queue did its half: a message whose worker died was not lost. The payment service did the other half: it remembered key 881 and refused to charge twice. At-least-once delivery is only safe with idempotent consumers, so design for the repeat from the start, not after the first double charge.

The payment service in the scene was already idempotent. Your own consumers usually are not, so how do you get them there? An operation is idempotent if doing it twice has the same effect as doing it once, like pressing a lift button twice: one lift comes. "Set order 881's status to shipped" is idempotent. "Add the sale's ₹50 cashback to the wallet" is not: run it twice and the buyer gets ₹100.

Four techniques cover most consumers:

• Record processed ids. Insert each message id into a table in the same transaction as the work. A repeat fails on the unique key and changes nothing.
• Use conditional updates. UPDATE orders SET status = 'paid' WHERE id = 881 AND status = 'pending' does nothing the second time.
• Pass an idempotency key downstream, as in the scene: the only option when the work happens in another company's system.
• Send absolute values, not changes. "Stock for item 7 is 41, version 12" survives repeats and stale copies; "stock minus 1" does not.

BEGIN;
-- fails on the primary key if msg-881 was already handled
INSERT INTO processed_messages (message_id) VALUES ('msg-881');
UPDATE wallets SET balance = balance + 50 WHERE user_id = 42;
COMMIT;

Both statements commit together or not at all. The consumer treats the duplicate-key error as already done and acks.

The first technique works only when the dedupe table and the data share one database; split across two, a crash between the writes brings the double credit back.

Idempotency makes repeats harmless. Order is a separate problem: what if two messages about one order finish in the wrong sequence? Many high-volume queues keep order only roughly, and with ten consumers in parallel, "order 881 cancelled" might finish before "order 881 placed", so the cancel finds nothing to cancel.

Picture railway reservation counters: each serves its own line in order, the lines move independently, and a family always queues at the same counter. One total order for everything means one counter, so throughput is one consumer's speed. The usual middle ground is per-key order. The queue is split into partitions, every message carries a key such as the order id, and all messages with one key go to one partition, read by one consumer at a time. Log-based brokers call such a partitioned stream a topic.

With partition = hash(order_id) % 12, up to 12 consumers work in parallel, and every event for order 881 stays in the order the broker received it. The three choices compare like this:

Ask for the weakest order that keeps the data correct: receipts need none, so a plain queue with competing consumers scales with no partitions to plan.

Per-key order has costs. A slow or failing message blocks every message behind it in its partition: head-of-line blocking. A very busy key, such as one seller with half the sale's orders, makes its partition hot. And retries must not jump the line: skip a failed message 5 to run message 6, and the order is already broken.

Head-of-line blocking turns one failing message into a stalled partition, so how should a consumer retry? Some failures fix themselves: the payment gateway times out during the sale, and a retry seconds later works. Others never will: an order with a missing address fails every time. That is a poison message. Sort failures into three kinds:

Retrying a permanent failure only burns workers, so the first question is always which row you are in.

Retries should wait longer each time, exponential backoff, plus randomness, jitter, so a thousand failed orders do not retry in the same second. After, say, 5 attempts, the message moves to a dead-letter queue (DLQ), named after the post office's dead letter office: a separate queue where failures wait to be inspected, fixed and replayed.

import random

def retry_delay(attempt, base=1.0, cap=300.0):
    # attempt 1: up to 1 s, attempt 2: up to 2 s, attempt 3: up to 4 s ... at most 5 min
    delay = min(cap, base * 2 ** (attempt - 1))
    return delay / 2 + random.uniform(0, delay / 2)   # jitter

With jitter, retries spread over a window instead of arriving as one wave.

Retries handle a few bad messages. What if every message is slow, for hours? If producers stay faster than consumers, the backlog grows until disks fill or the work is too old to matter: a confirmation an hour late has already lost the buyer. Either drain faster or take in less. Back-pressure is the second, like a restaurant that stops taking names when its waiting list is full: a full system signals upstream, so producers slow down or tell users to try again. Four levers help:

• Add consumers. Autoscale on the age of the oldest message, not on CPU.
• Bound the queue. When it is full, reject new messages, so the app can say "try again shortly" now instead of failing silently an hour later.
• Expire stale work. Give messages a time to live, so an hour-old message is dropped rather than acted on.
• Shed low-value work. Sample analytics events during the sale and keep the orders.

More consumers help only up to a point. A consumer group uses at most one consumer per partition, and workers writing to a database that takes 5,000 writes a second go no faster, however many you add. Say each order worker handles 50 messages a second and the sale needs 2,000: that is 40 workers. With 32 partitions only 32 get work, so throughput tops out at 1,600 a second. Choose 64, because raising the count later moves keys to new partitions and can break per-key order during the change.

A backlog of 36,000 orders is useful only if a broker restart does not wipe it, so how does the broker keep a confirmed message? Like a bank writing each entry into two ledgers, a durable broker stores each message on disk, copies it to other broker nodes, and confirms only once enough copies exist. With 3 copies and a rule that 2 must hold the message before the producer hears "ok", any one broker can die without losing a confirmed message, the replication idea from the databases chapter.

The producer chooses how long to wait, and each choice loses something different:

For orders those few milliseconds are cheap; fire and forget suits only data you can lose, such as metrics.

The producer side has its own trap, the dual write. The order service saves order 881 to its database, then crashes before publishing the message, or publishes and then fails to save. Either way the two disagree. The outbox pattern fixes it: write the message into an outbox table in the same transaction as the order, and let a separate relay publish from the outbox, retrying until the broker confirms. Delivery is at least once again, so consumers still dedupe.

A classic queue deletes a message once it is acked. A broker that takes hundreds of thousands of messages a second and serves several services is usually built differently: around an append-only log. Think of a shop's ledger: entries are only added at the end, and each reader keeps a bookmark. How does that change the broker?

Pick a classic queue for jobs that one worker does once; pick a log when several services read the same stream or you may need to replay it.

In a log, each partition is a series of segment files, say 1 GB each. New messages are only appended to the newest segment, and each gets the next offset. A small index per segment maps offsets to byte positions, and another maps timestamps, so a consumer can jump to offset 5,000,000 or to "yesterday 9 pm" without reading the whole partition.

Retention works on whole segments: when the newest message in a segment is older than the retention period, or the partition passes its size limit, the broker deletes that file in one cheap step. So a log cannot delete one bad message from the middle; consumers skip it or move it to a retry topic. A log keeps messages for everyone and lets each reader move its own bookmark, which is what makes replay and many independent readers possible.

Keeping every message for days sounds expensive. Why can one partition still take about 10 MB a second? Because a log uses the access patterns disks handle best:

• Sequential writes. Appending to the end of a file is far faster than writing to scattered places on disk.
• Batches. Producers send hundreds of messages per request, often compressed together, so one request and one disk write carry them all.
• The page cache. Consumers that keep up read recent messages from the memory the operating system uses for recently used file data, so they rarely touch the disk.
• No unpacking. The broker stores each batch as it arrived and sends it on in the same form.

That rough figure gives a second rule for counting partitions: the byte rate sets one floor and the number of parallel consumers another; take the larger. The sale's order topic carries 2,000 orders a second at about 2 KB each, only 4 MB a second, so bytes need a single partition, but 40 workers need at least 40 partitions. A clickstream topic is the opposite: millions of small events, where bytes decide. Each partition also costs files, memory and replication traffic, so size for need plus headroom, not thousands "to be safe".

Speed is half the story; the other half is surviving a broker's death. Each partition is copied to several brokers, usually 3. Producers write to the leader and consumers read from it; followers fetch new messages from it and append them in the same order. One write goes like this:

The committed point, which some brokers call the high-water mark, is the offset up to which enough replicas hold the data. What counts as enough depends on the broker: a majority in some, every replica that is keeping up in others. Consumers are only given messages below the committed point, so they never act on a message that a leader crash could still take back.

Consumer progress lives in the broker too. Each group commits its offsets to a small internal topic, and after a crash or a rebalance the new owner of a partition resumes from the last committed offset. Commit after processing, and a crash repeats some work: at-least-once again, seen from the inside.

A controller decides which broker leads each partition, and is itself chosen by a small consensus group, as the coordination chapter explains. The next page asks what happens when a leader dies holding messages no follower has yet.

During the sale, broker 1 leads the partition that holds order 881. It has just confirmed offsets up to 1,499 when the next batch, offsets 1,500 to 1,999, reaches it and nobody else. Then it crashes. Every change of leader raises a number, the leader epoch, from 7 to 8 here; brokers reject anything stamped with an older epoch.

Broker 1 had appended the second batch, but no follower held it, so it was never committed: the producer got no ok and the consumer could not read it. The producer's retry put the batch safely on the new leader, and the epoch told the returning broker exactly where its log parted from the leader's. Only what enough replicas hold counts as written, and that rule is what makes failover lose nothing confirmed.

Had broker 2 copied the batch before broker 1 died without replying, the retry would add a duplicate. Brokers with idempotent producers number each producer's batches, so the new leader drops the repeat. The price of this safety is latency: every confirmed write waits for a follower, a few milliseconds inside one datacenter.

When a design needs a queue, say what it absorbs, what it guarantees and what happens when a consumer or a broker dies, with a number or a trade-off in each line.