Design a Notification System

Friday, 8:00 pm. A food delivery app sends a “50% off biryani” push to 20 million users. At 8:02, Priya in Pune logs in on her new phone; her OTP arrives at 8:06, after it has expired. She tries again, gets in and orders dinner for her parents. “Your rider has arrived” reaches her at 8:41, two minutes after Imran rang the bell. Nothing was down. The OTP, the order update and 20 million offers simply waited in one queue, in arrival order. Urgent messages needed a lane of their own.

You are building the notification system for that food delivery app, with about 20 million daily users. Any team should be able to send a push notification, an SMS, an email or an in-app message, from login OTPs and order updates to marketing offers. The urgent ones must arrive within seconds, even while a campaign is going out.

Priya’s evening went wrong because every message shared one path. A food delivery app sends many kinds of messages: a login OTP (one-time password) by SMS, “your order is out for delivery” as a push notification, a receipt by email, a weekend offer to millions, and a red dot on the in-app bell. If every team wired up its own SMS provider and push code, users would get duplicate, badly timed messages, and nobody could say what had been sent.

A notification system is one shared service that every team calls. It decides whether to send (preferences, quiet hours, limits), how (which channels) and what (the final text), then hands each message to its channel and tracks what happened. Each channel has its own path and its own trap:

Only in-app is fully ours. Every other channel ends at an outside provider we cannot control, which is why lanes, retries and failover get the deep dives. Push needs one more idea: when the app starts, it gets a device token from the platform, an address for this app on this phone, and sends it to our servers. The requirements, in short:

Now size it. Assume the app has 20 million daily users, 5 million orders a day, and two marketing campaigns a day, each sent as a push to every one of those users.

The average hides the shape. A campaign is a burst we create on purpose, and the design must absorb it without delaying a single OTP behind 20 million offers. Storage and lookups, by contrast, are modest:

Nothing in that table strains a database. The hard number is the 12,000 sends a second of a campaign, and the rule that OTPs and order updates must not feel it.

With the numbers known, define how teams call the service. They call one endpoint and get an answer at once; delivery happens later, and 202 Accepted makes that explicit. Here is Priya’s order update, and the other calls:

POST /v1/notifications
Idempotency-Key: order-81723-out-for-delivery
{
  "user_id": "u_42",
  "type": "order_update",
  "template": "out_for_delivery",
  "data": { "rider": "Imran", "eta_min": 7 },
  "deliver_by": "2026-10-16T20:45:00+05:30"
}
202 Accepted   { "notification_id": "n_9f3a" }

POST /v1/campaigns   { "segment": "inactive_30_days", "template": "weekend_offer", "send_at": "2026-10-18T12:00:00+05:30" }
PUT  /v1/users/u_42/preferences   { "offers": { "push": true, "sms": false, "email": true }, "quiet_hours": ["22:00", "08:00"], "tz": "Asia/Kolkata" }
POST /v1/devices    { "user_id": "u_42", "platform": "android", "token": "..." }
GET  /v1/users/u_42/inbox?cursor=...

The caller says what happened, for whom and which template; the service decides channels, timing and the final text for each channel and language.

Three fields carry the design:

• type maps to rules: channels, lane, whether quiet hours apply. Callers do not pick channels themselves.
• Idempotency-Key lets a caller retry after a timeout without Priya getting two messages.
• deliver_by marks when the message stops being useful. An arrival estimate (ETA) from 40 minutes ago is noise, so it is dropped rather than sent late.

Services can also publish events such as “order picked up” to a topic and let the notification service subscribe, as in the pub-sub chapter. Then a notification outage delays messages instead of failing orders.

Behind the API, five tables hold the state. Four are read on almost every send and change rarely; the fifth, notifications, grows by tens of millions of rows a day.

notifications is written several times per message (queued, sent, delivered, opened), with one row per channel, keyed by notification id and channel. Partition it by user_id so one user’s history is a single range read, which is exactly what the in-app inbox needs, and expire rows after 90 days.

preferences and devices are read on every send, so keep them in a cache in front of the database. When a user turns offers off, delete their cached entry at once (the distributed cache chapter’s rule): one more offer after an opt-out is the kind of mistake users remember.

templates are versioned: an edit creates a new version, and each notification records the version it used, so you can always see the exact text that was sent. devices needs care: the app sends its token on every start, because the platform can replace it at any time; on logout the token is unlinked, so the next person on that phone does not get Priya’s orders; tokens the push service reports as unregistered are deleted at once.

Keep secrets out of storage and logs. Store the template id and its variables, with sensitive ones such as an OTP masked, rather than the rendered text, and never write a message body into logs. Support staff can still see which message went out and when, without seeing the code inside it.

Put the pieces together. The service accepts a request, runs it through a short decision pipeline, and drops one message per channel onto a queue for that channel. Channels that carry both kinds of traffic get two lanes: urgent (OTPs, order updates, receipts) and bulk (offers, digests). In-app messages need no provider, so the service writes them straight to the inbox store. Channel workers pull from the queues and call outside providers at a rate those providers allow.

Workers are pooled per channel because each channel has its own provider, limits and failure modes. Push workers can scale out for a campaign without touching the SMS workers that carry OTPs, and a slow email provider cannot hold up anything else.

For an OTP, our own part of the journey is small: checks and rendering take a few milliseconds, the urgent lane is usually empty, and the provider call takes a few hundred milliseconds. Most of the wait is the telecom network delivering the SMS, often a few seconds. A backed-up queue would be the one delay we caused ourselves.

Two questions remain before the deep dives: what the pipeline decides for each message, and how lanes keep a campaign away from Priya.

Every request passes through the same short pipeline before anything is queued. Each step can stop the message, delay it, or change its channel:

The rules per type live in one table, owned by the notification team:

Urgent types skip quiet hours and caps; bulk types obey both. Fallbacks live in the same table: if an order update cannot go by push, because Priya has no device token or has push switched off, steps such as “your rider has arrived” fall back to SMS. Offers never fall back to SMS. A new kind of notification is a new row, reviewed once, which is what stops a well-meaning team from sending offers at 2 am.

Back to Friday, 8 pm. A queue hands out messages roughly in arrival order, so in one shared queue Priya’s update would wait behind 2 million offers for about 3 minutes. Lanes work like the separate counter at a railway booking office: the long general line never reaches it. Priority must hold at every shared stage, or the line forms at the next one:

• Intake: the campaign job feeds the pipeline in paced batches, so offers never pile up in front of an OTP request.
• Queues: each channel that carries both kinds has an urgent lane and a bulk lane.
• Workers: each lane has its own workers, so offers cannot occupy all of them.
• Provider quota: workers share one rate limit per provider account, and the urgent lane keeps a reserved share.

Lanes make nothing faster; they stop a patient burst from delaying impatient messages. A campaign may delay other offers, never an OTP or an order update.

Size the reserved share from numbers. Say the push provider allows this account 15,000 sends a second (real quotas are agreed with each provider). Cap bulk at 11,000, enough for a 20 M campaign in 30 minutes, and keep 4,000 for urgent, about four times the dinner-peak rate of order updates. The cost is idle quota on quiet evenings.

Lanes keep Priya’s messages from waiting on us. Providers fail in their own ordinary ways: timeouts, 5xx errors, throttling with 429, and permanent rejections such as an invalid number or a device token from an uninstalled app. The worker treats each differently:

Only temporary failures are retried; retrying a permanent rejection wastes money and quota. Each SMS and email provider also sits behind a circuit breaker: if its error rate stays above a threshold, say 20% over a minute, workers stop calling it and use the standby provider.

Two rules made Priya’s OTP fast. An OTP’s retry goes straight to the other provider, because backoff against a failing provider spends a waiting user’s patience. And the breaker spares everyone else the timeouts. Push has no standby, since each platform has one push service: pushes wait until their deliver_by, and critical steps fall back to SMS.

Retries and failover keep messages moving, but every retry risks a second copy. The queue delivers at least once (see the message queue chapter): a worker can call the provider, then crash before acknowledging, and another worker receives the same message. So de-duplication happens in layers, each an atomic check, such as a set-if-absent with a one-day expiry, so two copies arriving together cannot both pass.

The status check turns a redelivery into a no-op. One window stays open: a crash after the provider accepted the message but before the status was saved. The status check is one of four layers:

Each layer catches a repeat the others cannot see.

Most failures here start somewhere else: a provider slows down, a phone platform rejects tokens, a caller floods the API. The design’s job is to keep each failure inside its own lane:

In every row the urgent path survives: failures delay bulk work or cost a rare duplicate, but never strand an OTP behind someone else’s problem.

Each choice above trades something away. Here are the main ones, with the usual pick:

Every pick favours the urgent message over the cheap or simple option. Three signals show whether that is working:

What dominates the bill: SMS, about 3 M a day, each one paid for, while push is free to send and email costs a small fraction per message. That is why OTP templates stay short and offers never fall back to SMS.

Optional: four further topics, starting with a burst that the bulk lane cannot absorb.

Campaigns are big but patient: an offer at 8:25 is as good as one at 8:00, so the bulk lane drains it over half an hour. Some messages lose their value in minutes. During the cricket World Cup final, the app runs an innings-break deal, “free dessert for the next 15 minutes”, and must tell 10 M users within 2 minutes, or the break is over.

That is 10 M ÷ 120 s ≈ 83,000 sends a second, about seven times the campaign rate, and none of it can be smoothed out. At the bulk lane’s 11,000 a second it would take 15 minutes. It needs its own preparations:

The quota is the real limit: workers are easy to add, and 83,000 a second is more than five times the account’s normal quota. Keep the safety rules while you go fast. The urgent lane keeps its workers and reserved quota, so Priya’s order updates still flow; each deal push carries a deliver_by 2 minutes out and a collapse id. Where the platform offers topics, one send can reach every subscribed device, at the cost of per-user status and caps.

The next topic is how users stay in control. Preferences are stored per category and channel: Priya can keep order updates on push but turn offers off everywhere. Some messages ignore preferences because the user asked for them or rules require them, such as OTPs and security alerts. Marketing email must carry a working unsubscribe link: laws in many countries require one, and large email providers expect one-click unsubscribe from bulk senders.

Quiet hours are kept in the user’s own time zone. A bulk message that falls inside them is not dropped straight away: the task scheduler (see the task scheduler chapter) holds it until quiet hours end, unless its deliver_by will have passed by then. Held messages are released over the first hour of the morning, so 8:00 does not become a second spike. Here is Priya’s evening; her quiet hours run from 10 pm to 8 am:

Quiet hours delay only what can wait; anything Priya is actively waiting for goes at once.

Per-user limits reuse the rate limiter chapter’s counters, kept in a shared in-memory store with atomic increments: a daily count per user and category, such as 2 offer pushes a day, and a short window per phone number for OTPs. An offer over its cap is dropped, not saved for later. Limits protect users who still have notifications switched on: someone who disables push because of spam stops getting order updates too.

Next, the words themselves. A template is a message with named gaps that the service fills at send time from the request’s data. Each template has a variant per channel, because a push line, an SMS and an email need different lengths and tone, and a variant per language, falling back to English if one is missing:

template: out_for_delivery    version: 7

push, en:  Your order is on its way. {rider} will reach you in about {eta_min} min.
push, hi:  आपका ऑर्डर रास्ते में है। {rider} लगभग {eta_min} मिनट में पहुँचेंगे।
sms,  en:  Order on its way. {rider} arrives in about {eta_min} min.

data:      { "rider": "Imran", "eta_min": 7 }
push, en → Your order is on its way. Imran will reach you in about 7 min.

One template, with a variant per channel and language. Variables are filled in at send time.

Mind each channel’s limits. One SMS holds 160 characters of the basic GSM alphabet (GSM-7): English letters, digits and common punctuation. One character outside it, such as a Hindi letter, an emoji or the ₹ sign, switches the whole message to UCS-2, a 16-bit encoding that fits only 70. Longer texts are split into parts of 153 or 67 characters, and every part is billed, so keep OTP templates short in every language. Push payloads are capped at about 4 KB, and the lock screen shows a line or two.

Campaigns are many small sends. A campaign job reads the audience in batches of, say, 1,000 users and passes each through the normal pipeline, so their own preferences, caps and quiet hours apply. Workers check the campaign’s kill switch before every send, so stopping a campaign also stops messages still waiting in the lane. Three rules keep templates safe:

• Missing variable? Fail that send and alert. Never deliver Hello {name} to anyone.
• Escape per channel. Data placed into an email must be HTML-escaped, so an odd restaurant name cannot break the layout or slip in a link.
• Test before launch. Preview every language, and give new versions to a small share of users first.

Last, how do you know what happened to each message? Every message moves through statuses, and each status comes from a different place with different reliability. Providers report delivery and bounces later, by calling a webhook of ours:

Treat email opens as a rough signal and SMS delivery reports as the dependable one. Status events go onto their own queue and into an analytics store, away from the hot send path, so dashboards can answer whether OTP delivery is dropping for one operator or yesterday’s campaign raised opt-outs.

Tracking also feeds back into sending. Email addresses that hard-bounce go on a suppression list and dead device tokens are deleted, so we stop paying for messages nobody can receive. The in-app inbox is the simplest channel: write a row for the user, bump an unread counter, and let the app fetch the list with a cursor; opening an item marks it read.

A 45-minute round leaves about 40 minutes of design after introductions. Here is how this design fits that clock:

Lanes carry most of the marks, because they answer the requirement that shapes everything. If they ask what happens when the SMS provider goes down, or whether a user can get two OTPs, go to failover and duplicates; for an innings-break deal that must reach 10 M users in 2 minutes, go to the unspreadable burst.

Interviewers often change one thing and watch how you adapt. Try each before opening the answer.