Queues and Async Processing

7 pages · ~5 min✓ Reviewed

Builds on Databases, Replication and Sharding. Next up: CAP, Consistency and Reliability.

System DesignPage 1

Queues vs Pub/Sub Mental Model

1. Queue vs Pub/Sub

	Queue	Pub/sub
Each message goes to	one consumer in a competing group	every subscriber
Use for	work distribution	fan-out
Examples	SQS, RabbitMQ queue	SNS, Redis pub/sub, Kafka groups (full copy each)

2. Broker and Retention

Producer
Broker
Consumer

Decoupled in time, speed and failure
Producer does not wait for the consumer
Queue deletes a message after ack
Log (Kafka) retains by time/size, so it can be replayed

3. Command vs Event

	Command	Event
Says	'do X'	'X happened'
Handled by	one handler	any number of listeners
Example	`ChargeCard`	`OrderPlaced`

Remember

Queue = one consumer per message. Pub/sub = every subscriber. A log keeps messages for replay.

System DesignPage 2

Kafka Basics: Topics, Partitions, Offsets

1. Topic, partition, offset

A topic is split into partitions
Each partition is an append-only ordered log
Ordering holds only within a partition, never across
Offset = a message's position in its partition
Consumer commits offsets to resume where it left off

m00

m11

m22

m33↑ committed

m44

Partition 0: append at the end

2. Keys and consumer groups

Key is hashed to pick the partition
Same key → same partition → order kept
Each partition is read by one consumer per group
Max parallelism = number of partitions

Partitions	Consumers	Result
3	1	one reads all 3
3	3	one each
3	4	1 sits idle

3. Replication

Each partition has a leader and followers
acks=all waits for every in-sync replica
With min.insync.replicas=2, losing one broker loses no data

properties

acks=all                # producer
min.insync.replicas=2   # topic/broker

the durable write setup

Remember

Order needs the same key. Parallelism stops at the partition count: extra consumers sit idle.

System DesignPage 3

Delivery Guarantees and Idempotent Consumers

1. Commit Order Sets the Guarantee

	Offset commit	Crash means
At-most-once	before processing	message lost, no duplicates
At-least-once	after processing	redelivery, duplicates possible

At-least-once is the usual default
Pair it with an idempotent consumer

2. Exactly-Once Is Not Free

Kafka: idempotent producer (enable.idempotence=true) plus transactions
Holds only within Kafka
External side effects (email, payment API) still need idempotency

3. Idempotent Consumer

Store the processed message ID (unique key or dedupe table)
Skip repeats: the duplicate insert fails
Dedupe check and business write in one transaction

sql

BEGIN;
INSERT INTO processed(msg_id) VALUES ('m42');
UPDATE orders SET status = 'paid' WHERE id = 7;
COMMIT;

unique key on msg_id

Operation	Safe to repeat?
upsert	yes
`set status=paid`	yes
`balance += 10`	no

Remember

Default to at-least-once + idempotent consumer. Dedupe outside the write transaction can still double-apply.

System DesignPage 4

Failures: Retries, Dead-Letter Queues and Backpressure

1. Retry Safely

Retry only transient errors: timeouts, 503, throttling
Delay grows exponentially, plus random jitter
Set a max attempt cap, then stop retrying
Never retry in a tight loop

python

delay = min(cap, base * 2 ** attempt)
time.sleep(random.uniform(0, delay))

full jitter avoids synchronized retry storms

2. Dead-Letter Queue

Receiveconsumer
Failsretry again
N attemptsmaxReceiveCount
DLQmoved aside

A poison message always fails and blocks the queue
DLQ needs an alarm on its depth
Keep the original payload plus error metadata
Have a tested redrive path back to the queue

3. Backpressure and Lag

Lever	Action
Slow the producer	throttle, block, return 429
Bound the queue	fixed max size
Shed or buffer	drop low priority, spill to disk

Consumer lag = latest offset - committed offset
Also watch queue age
Scale consumers up to the partition count, no further

Unbounded queue

A queue that grows without limit only hides overload. Bound it and slow the producer.

System DesignPage 5

Event-Driven Design Pitfalls

1. Event Shape

	Notification	State transfer
Carries	`order placed` + ID	full order data
Consumer	calls back for details	no callback

2. Coordination

	Choreography	Orchestration
Control	services react to events	central coordinator
Trade-off	loose, hard to trace	explicit flow

3. Pitfalls

DB + outbox rowone transaction
Relayreads outbox
Brokerpublish

Dual write: DB and broker written separately, one can be lost
Fix: transactional outbox
Order is per key only, never global
A rebalance mid-processing can reorder handling
Version events: add optional fields only
Old messages stay in the log; consumers upgrade at different times

4. Interview Point

Why not write the DB, then publish? One can fail; use an outbox.
Is order global? No, only per key.
How to evolve schemas? Add optional fields only.

System DesignPage 6

Messaging Cheat Sheet

1. Pick the Model

Model	Delivered to	Example
Queue	one consumer per message	SQS, RabbitMQ
Pub/sub	every subscriber	SNS, Redis
Kafka	each group gets a full copy; replayable	partitioned log

Order holds only within a partition (same key)
Parallelism ≤ partition count

2. Process Safely

Receive
Dedupe + writeone transaction
Commit offsetafter processing

Default: at-least-once + idempotent consumer
Retry with backoff + jitter, then DLQ with an alarm
Monitor consumer lag: latest minus committed offset

3. Design Traps

DB write + publish can lose one: use the outbox pattern
Version event schemas: add optional fields only

Remember

Commit the offset before processing and a crash loses the message: at-most-once.

System DesignPage 7

Check yourself

A consumer charges a card via an API, crashes before committing the offset, then restarts. What happens, and what is the fix?

Offset uncommitted, so the message is redelivered and the card is charged twice
Pass the message ID as the payment API's idempotency key
Optionally also record the ID in a dedupe table

A topic has 4 partitions and the group has 6 consumers. What runs in parallel?

Only 4 consumers get a partition; 2 sit idle
Max parallelism = partition count

Code saves the order to the DB, then publishes an event; it crashes between the two. What breaks?

Dual write: order saved but event never sent
Fix: write an outbox row in the same transaction; a relay publishes it

A poison message fails every time and is retried in a tight loop. What is wrong?

It blocks the queue and burns CPU
Back off with jitter, cap attempts, then dead-letter it
Alarm on DLQ depth