Handbooks / System Design Case Studies / Chapter 5

Design a Notification System

12 pages · ~18 min✓ Reviewed

Builds on Design Search Autocomplete. Next up: Design a Chat App like WhatsApp.

System Design Case StudiesPage 1

Problem Statement, Scope and Requirements

1. Scope

Running example: a multi-channel Notification System for an app with 10M daily active users.

Channels: push (APNs/FCM), SMS, email, in-app
Triggers: service call, scheduled, bulk campaign

Out of scope: marketing UI
Out of scope: content authoring

2. Functional Requirements

Send a single or a bulk notification
Template rendering with variables
Per-user opt-in/out by channel and by category
Scheduling for a future time
Delivery status query by notification id

3. Estimates and Latency Targets

Metric	Estimate	Working
Volume	50M/day	10M DAU x 5 per user
Average QPS	about 580	50M / 86,400 s
Peak QPS	about 5k

System Design Case StudiesPage 2

High-Level Architecture (Notification System)

1. End-to-End Flow

Producerservice
Notification APIreturns 202
Validate + dedup + prefsintake checks
Kafka topicschannel x priority
Channel workersrender, limit, send
Provider adaptersone per vendor
ProviderFCM, Twilio, SES

System Design Case StudiesPage 3

Notification API and Intake Service

1. Responsibilities and API

Intake for the Notification System: ~5k QPS peak, accept is synchronous, delivery is asynchronous.

Authenticate the producer service
Validate the payload: user_id, template_id, channel[], priority
Assign the notification_id on the server
Enqueue durably, then return 202 with the id
GET /v1/notifications/{id} returns the current status

json

POST /v1/notifications
{"user_id": "u_42", "template_id": "otp_login",
"channel": ["sms", "push"], "priority": "critical",
"idempotency_key": "login-9f3a",
"data": {"code": "481516"}}

202 Accepted -> {"notification_id": "..."}

System Design Case StudiesPage 4

Templates and Localization Service

1. Responsibilities and Render Path

Store versioned templates per channel and locale
Render variables into subject and body
Enforce size limits: SMS 160 GSM-7 chars, push payload 4 KB
Render at worker time, not at intake: the API only stores template_id + data
Worker input: template_id + version + data

Workertemplate_id + version + data
In-memory cachecompiled, TTL 5 min
Compileon miss only
Render + size check160 chars / 4 KB

System Design Case StudiesPage 5

User Preferences and Opt-Out

1. Responsibilities

Per-user, per-channel, per-category opt-in (push, SMS, email, in-app)
Quiet hours stored with a timezone (tz), not a fixed UTC offset
Global unsubscribe: one switch that stops every non-critical message
Legal opt-out: TCPA (SMS STOP) and CAN-SPAM (email unsubscribe) must be honored

2. Check Order and Failure

Check order (first failure suppresses)

System Design Case StudiesPage 6

Priority Queues and Channel Workers

1. Priority Isolation

Split traffic into 4 priorities: critical, high, normal, bulk
Goal: a campaign never delays an OTP
One Kafka topic per channel x priority, e.g. sms.critical, push.bulk
Separate topics also stop a slow SMS provider from blocking push

Priority	Example

System Design Case StudiesPage 7

Retries, Dead Letters and Deduplication

1. What to Retry

Failure	Class	Action
5xx from provider	transient	retry
Timeout	transient	retry
429 (rate limited)	transient	retry after the backoff delay

System Design Case StudiesPage 8

Rate Limiting and Throttling

1. Three Limits, Three Jobs

Limit	Example	Protects
Per-user	5 marketing push/day, 3 SMS/hour	The user from spam
Critical abuse	5 OTPs / 10 min per user	The SMS bill and the user from OTP floods
Per-provider	SES 14 emails/sec, Twilio 1 msg/sec per number

System Design Case StudiesPage 9

Third-Party Providers and Failover

1. Adapter Interface

Provider layer of the Notification System: one adapter hides each vendor, so workers never see vendor APIs.

Every adapter exposes send(msg) and returns {provider_id, status}
One adapter each: FCM, APNs, Twilio, SES, SendGrid
Adapter maps vendor errors to transient or permanent
Swapping or adding a vendor touches only its adapter

python

class Adapter(Protocol):
    def send(self, msg: dict) -> dict:
        """Return {'provider_id': str, 'status': str}"""
        ...

same contract for FCM, APNs, Twilio, SES, SendGrid

System Design Case StudiesPage 10

Delivery Tracking and In-App Inbox

Running example: a Food Delivery App tracks an order-shipped notification from queued to clicked, and shows it in the inbox.

1. Status Lifecycle and Sources

queuedaccepted
sentto provider
deliveredon device or inbox
openeduser saw it
clickedlink tapped

Terminal states: failed, suppressed (pref or limit), bounced (bad address)
Each status gets a rank: queued 1, sent 2, delivered 3, opened 4, clicked 5

System Design Case StudiesPage 11

Scaling, Failure Handling, Trade-offs and Interview Questions

1. Scale at 5k QPS Peak (Notification System)

Layer	Sizing at 5k QPS peak	Rule
API pods	about 20 pods	stateless, add pods behind the LB
Channel workers	about 50 workers	autoscale on consumer lag
Kafka	100 partitions per hot topic	key = `user_id`, keeps per-user order

System Design Case StudiesPage 12

Check yourself

A producer POSTs a notification with idempotency_key K, times out, and retries with the same K. What stops the user getting two pushes, and what is the UNIQUE column on idempotency_key for?

The Redis SET NX EX 24h gate finds the existing key for the first request
The retry gets the original notification_id back with 202, and nothing new is enqueued
The UNIQUE column is only a per-partition backstop: the table is partitioned by created_at, so a retry in another partition would not collide
So the Redis gate is the dedup mechanism, not the UNIQUE index

A 2M-message marketing campaign is running and users report OTP texts arriving minutes late. All SMS traffic shares one queue. What is the bug and the fix?

Tier	End-to-end target	Example
Critical	under 2 s	OTP, security alert
Transactional	under 10 s	Order or payment receipt
Marketing	within minutes	Campaign, digest

Component	Job	Backing
Template Service	versioned templates per channel and locale; workers render	DB + 5 min cache
Preference Service	opt-in/out, quiet hours, device tokens	DB + Redis cache
Rate Limiter	per-user caps and per-provider caps	Redis
Scheduler	holds future-dated sends, releases them to Kafka at due time	DB + timer
Tracking Service	consumes status events and webhooks, serves status query	append-only store
Dead Letter Queue	parks messages that exhausted retries, with error payload	Kafka topic

Component

Job

Backing

Template Service

versioned templates per channel and locale; workers render

DB + 5 min cache

Preference Service

opt-in/out, quiet hours, device tokens

DB + Redis cache

Rate Limiter

per-user caps and per-provider caps

Redis

Scheduler

holds future-dated sends, releases them to Kafka at due time

DB + timer

Tracking Service

consumes status events and webhooks, serves status query

append-only store

Dead Letter Queue

parks messages that exhausted retries, with error payload

Kafka topic

	Single vendor	Own pipeline + providers
Cost	one price list, no routing by cost	route to cheapest, e.g. SES for email
Failover	vendor outage = our outage	switch to a secondary provider
Control	vendor limits and features	our queues, retries, dedup, tracking
Effort	low	higher: we run the pipeline

Single vendor

Own pipeline + providers

Cost

one price list, no routing by cost

route to cheapest, e.g. SES for email

Failover

vendor outage = our outage

switch to a secondary provider

Control

vendor limits and features

our queues, retries, dedup, tracking

Effort

low

higher: we run the pipeline

Metric	Target
Accept latency	p99 under 50 ms
5xx rate	under 0.1%
Enqueue lag	pending outbox age: alert on growth
Rejected by validation	rate per producer

Column	Type / values
`id`, `version`	template key, one row per version
`channel`	push / sms / email / in-app
`locale`	`en`, `hi`, ...
`subject`	email and push title
`body`	text with `{{name}}` variables
`status`	draft or active

Preference store down	Policy	Why
Critical (OTP)	Fail open	A blocked OTP locks the user out
Marketing	Fail closed	A wrong send breaks legal opt-out

Metric	Target or use
Consumer lag per topic	alert on growth; drives autoscaling
Queue wait p99, critical	under 500 ms
Worker throughput	msgs/sec per pod, per channel
Autoscale events	scale-out count per topic per day

Call	Reply
call 1	OK (accepted)
call 2	(nil) (duplicate suppressed)

critical	OTP, security alert	8	under 2s end to end
high	Ride accepted, payment receipt	4	under 10s
normal	Social activity	2	within minutes
bulk	Marketing campaign	1	within minutes, throttled

Metric	Target
Retry rate	under 5%
DLQ rate	under 0.1%
DLQ depth	alert above 100
Duplicate-suppressed count	tracked; spike means producer retry storm
Time-to-success after retry	tracked per retry topic

	Token bucket	Sliding window
Bursts	Allowed up to bucket size	Strict, no burst past the limit
Memory	O(1) per key (tokens + timestamp)	O(n) per key (one entry per event)
Accuracy	Approximate over a period	Exact over any trailing window
Best for	Provider caps, global limits	Per-user marketing caps

Channel	Providers	Target	Note
Push iOS	APNs	device token	token per device
Push Android/web	FCM	registration token	token per device
SMS	Twilio, Vonage	phone number	secondary = Vonage
Email	SES, SendGrid	email address	SES cheap: ~$0.10 per 1k

Channel	Status source	Gives
Email	SES via SNS webhook	delivered, bounced
SMS	Twilio status callback	sent, delivered, failed
Push	Device ack on open	opened
Email	Tracking pixel	opened
Email	Redirect link `/r/{token}`	clicked

Choice	Pick	Why
Exactly-once vs at-least-once	at-least-once + dedup	exactly-once is impossible across a provider boundary
Push vs pull	push for low latency	send to the device now
Inbox fan-out	pull for reliable fan-out	client reads `inbox` by cursor, nothing is lost

Design a Notification System

1. Scope

2. Functional Requirements

3. Estimates and Latency Targets

1. End-to-End Flow

1. Responsibilities and API

1. Responsibilities and Render Path

1. Responsibilities

2. Check Order and Failure

1. Priority Isolation

1. What to Retry

1. Three Limits, Three Jobs

1. Adapter Interface

1. Status Lifecycle and Sources

1. Scale at 5k QPS Peak (Notification System)

4. Non-Functional Requirements

2. Components

3. Isolation and Scaling

4. Build vs Buy

2. Intake Flow: Two Lanes

3. Data Model

4. Failure and Metrics

2. Data Model and Locale Fallback

3. API

4. Failure and Metrics

3. Data Model and API

4. Metrics

2. Worker Flow and Crash Recovery

3. Bulk Fan-out and Autoscaling

4. Metrics

2. Backoff, Delay Topics and DLQ

3. Deduplication

4. Metrics

2. Token Bucket vs Sliding Window

3. Over Limit and Redis Down

4. Metrics

2. Providers per Channel

3. Failover and Token Pruning

4. Outage and Metrics

2. Event Store and Inbox

3. API

4. Failures and Metrics

2. Trade-offs and Multi-Region

3. Common Mistakes

4. Interview Questions and Cheatsheet