Handbooks / System Design Case Studies / Chapter 6

Design a Chat App like WhatsApp

13 pages · ~18 min✓ Reviewed

Builds on Design a Notification System. Next up: Design a News Feed like Instagram.

System Design Case StudiesPage 1

Problem, Scope and Requirements

1. Scope (WhatsApp Chat App)

	Features
In	1:1 chat, groups up to 1,024 members, media, receipts, presence, push, E2EE
Out	Calls, status, payments

2. Scale and Storage

Metric	Estimate	Math
Users	2B total, 500M DAU	given
Messages	100B/day	given
Text storage	10 TB/day	100B × ~100 B
Media storage	~1 PB/day	5% of msgs × ~200 KB
Sockets	500M concurrent	one per online user
Gateways	2,500+ servers	500M ÷ ~200K each

100B msgs/daydaily volume
÷ 86,400 sseconds per day
≈ 1.2M msg/saverage
~3M msg/speak

System Design Case StudiesPage 2

High-Level Architecture

1. Sockets and the Gateway

WhatsApp-style Chat App: stateful sockets at the edge, stateless services in the middle, Kafka behind them.

Clientapp, 1 WebSocket
L4 LBTCP, no parsing
Gateway~200K sockets each
Chat servicestateless logic
Kafkaby conversation_id

Each client holds one persistent WebSocket for all chats
L4 load balancer spreads TCP connections; sockets then stay pinned to a gateway

System Design Case StudiesPage 3

WebSocket Connection Gateway

Gateway tier (WhatsApp Chat App): 500M concurrent sockets ÷ ~200K per server ≈ 2,500+ gateways.

1. Responsibilities and Capacity

Hold long-lived WebSockets, one per online device
Authenticate the socket once, at connect
Heartbeat every ~30 s to detect dead peers
Forward frames to the chat service; gateway logic stays stateless

System Design Case StudiesPage 4

One-to-One Message Flow and Ordering

1. Send Flow (WhatsApp)

Senderclient_msg_id
Gateway AWebSocket frame
Chat servicevalidate, persist
Session lookupuser_id → gateway
Gateway Bpush frame
Receiverdevice

Server ACK goes back to the sender only after persist
ACK means stored (one tick), not delivered
Session miss in Redis → inbox queue + push job
Target: online delivery p99 < 200 ms

2. Idempotency and IDs

System Design Case StudiesPage 5

Message Storage

1. Access Pattern and Store

Chat App (WhatsApp): 100B msgs/day × ~100 B ≈ 10 TB/day of text, ~1.2M writes/s average.

Write-heavy: every message is one insert
Append-only: messages are never updated in place
Read recent: newest messages of one conversation
So use a wide-column store (Cassandra / HBase), not SQL

System Design Case StudiesPage 6

Delivery and Read Receipts

1. Three Receipt States

State	Trigger	Ticks
sent	Server persisted the message	one grey tick
delivered	Recipient device received it	two grey ticks
read	Recipient opened the chat	two blue ticks

System Design Case StudiesPage 7

Group Messaging and Fan-out

Running example: the WhatsApp-style chat app. One group message becomes up to 1,024 deliveries at ~3M msg/s peak.

1. Why Fan Out on the Server

	Client sends N copies	Server-side fan-out
Uplink per message	up to 1,024 uploads	1 upload
Mobile bandwidth	drained on cellular	saved

System Design Case StudiesPage 8

Offline Delivery and Push Notifications

1. Offline Path and Push

Session lookup in Redis misses → recipient is offline
Message stays in the recipient's inbox queue
A push job is created for the recipient's devices
Message is already persisted, so push is only a wake-up

Session lookupmiss
Inbox queuekeep message
Push jobKafka
APNs / FCMwake device
App fetchesdrain inbox

System Design Case StudiesPage 9

Presence and Typing Indicators

1. Presence = Heartbeat + TTL

Online = heartbeat seen within ~60 s
Gateway heartbeat every ~30 s refreshes a Redis key with TTL 60 s
Key presence:{user_id} expires on its own: no cleanup job
last_seen is written to the store only on disconnect
Heartbeats are ~500M sockets ÷ 30 s ≈ 17M refreshes/s at peak

python

r.set(f"presence:{uid}", gateway_id, ex=60)  # heartbeat
r.exists(f"presence:{uid}")  # online?
# on disconnect
db.set_last_seen(uid, now())

TTL key for online, durable write for last_seen

2. Who Gets Presence and Typing

System Design Case StudiesPage 10

Media Sharing and End-to-End Encryption

1. Media Flow (WhatsApp Chat App)

Sending a photo

1Encryptrandom key on the client
2Uploadciphertext via pre-signed URL
3Object storeblob lands in S3
4Send messageURL + key + hash, E2EE

json

{ "type": "image",
  "url": "https://cdn.../blob/9f3a",
  "key": "<random 256-bit key>",
  "sha256": "<hash of ciphertext>",
  "thumb": "<tiny blurred preview>" }

travels inside the E2EE message, never in the clear

Server stores ciphertext only; it never holds the key
Thumbnails are made on the client and sent inline
CDN caches ciphertext; the key stays in the chat message

System Design Case StudiesPage 11

Scaling, Failure Handling, Trade-offs and Interview Questions

1. Scale Out and Multi-Region

Running example: WhatsApp-style chat app, 500M concurrent sockets, ~3M msg/s peak, 99.99% availability.

Shard chat service and storage by conversation_id: a chat lives on one partition, so order stays local
Shard sessions (Redis) by user_id with consistent hashing: adding a node moves only ~1/N of keys
Add gateways horizontally: ~200K sockets each, 2,500+ nodes for 500M connections
Home each user to the nearest region; Kafka replicates messages across regions
Region failover: DNS health checks repoint users; clients reconnect with backoff + jitter

System Design Case StudiesPage 12

WhatsApp Design Cheatsheet

1. Numbers (WhatsApp Chat App)

Metric	Estimate	Derived from
Concurrent sockets	500M	500M DAU online
Sockets per gateway	200K	500M / 200K = 2,500+ servers
Message rate	1.2M msg/s avg, ~3M peak	100B msgs/day
Text storage

System Design Case StudiesPage 13

Check yourself

The ACK for a sent message is lost after the server persisted it. The client retries, but its code builds a fresh UUID for every attempt. What does the receiver see, and what is the fix?

The server cannot tell the retry from a new message, so it stores a second copy
The receiver sees the same text twice
Fix: create client_msg_id once per message and reuse it on every retry
The server then finds the ID, skips the write and returns the stored result

# bug: new id on every attempt
for attempt in range(3):
    send(body, client_msg_id=uuid4())

A channel has 1M members and fan-out on write is used as for 1,024-member groups. What happens for one post, and what do you change?

Consumer	Reads	Does
Fan-out	new messages	finds recipients, sends to gateway
Push	offline deliveries	calls APNs / FCM
Storage writer	new messages	writes messages and inbox

Idle sockets per node	200K–1M	Mostly idle, so I/O is cheap
Memory per socket	~10 KB	200K ≈ 2 GB, 1M ≈ 10 GB
File descriptors	`ulimit -n` above 1M	One fd per socket
Heartbeat load	200K ÷ 30 s ≈ 6.7K pings/s	Tiny frames, cheap

Frame	Direction	Purpose
`send`	client → server	New message with client_msg_id
`ack`	both ways	Server stored it, or device received it
`receipt`	both ways	Delivered / read update
`typing`	both ways	Ephemeral, never stored
`presence`	server → client	Contact online / offline

Snowflake field	Bits	Meaning
Sign	1	always 0
Timestamp	41	ms since custom epoch, sorts by time
Shard	10	which chat-service shard minted it
Sequence	12	4,096 IDs per ms per shard

ID	Made by	Job
`client_msg_id`	client	dedupe retries
`message_id`	server (Snowflake)	unique, sortable key
conversation seq	server, per conversation	display order

Write rate	LSM append, scales out	B-tree, shard by hand
Read recent	one partition, sorted	joins and index scans
Growth	add nodes	manual resharding

Metric	Watch for
Write p99	slow commit log or compaction
Partition size	unbounded chats, bucket too wide
Disk usage	~10 TB/day × replication factor 3

	APNs	FCM
Platform	iOS	Android
Payload	small, no plaintext under E2EE	small, no plaintext under E2EE
Carries	wake signal + collapse key	wake signal + collapse key

Failure	Response
Invalid or expired push token	Prune the token on APNs/FCM error
APNs/FCM outage	Degrade to fetch-on-open; inbox still holds messages

	Presence	Typing indicator
Stored in	Redis key + TTL	nowhere
Persisted	`last_seen` on disconnect	never
Recipient offline	Shows last seen	dropped
Path	Gateway → subscribers	Gateway → recipient gateway

Key	Lifetime	Role
Identity key	long-term	Proves who the device is
Signed prekey	rotated periodically	Medium-term key, signed by the identity key
One-time prekeys	used once, then deleted	Batch uploaded; one consumed per new session

Decision	Chosen	Why
Delivery	At-least-once + dedupe	`client_msg_id` makes retries safe; exactly-once is costly
Fan-out	Write for groups ≤1,024	One copy stored, 1,024 deliveries via Kafka
Fan-out	Read for 1M channels	Pull by sequence; no 1M writes per post
Transport	WebSocket	p99 < 200 ms push; long poll only on bad networks

Key	Solves	How
`client_msg_id`	idempotency	UUID from client; retry returns stored result
per-conversation sequence	order	monotonic per `conversation_id`, no global order
`last_read_id`	receipts	one pointer per user per conversation

user_id	conversation_id	last_delivered_id	last_read_id
u7	c42	7391822015	7391822011
u9	c42	7391822015	7391821990

Design a Chat App like WhatsApp

1. Scope (WhatsApp Chat App)

2. Scale and Storage

1. Sockets and the Gateway

1. Responsibilities and Capacity

1. Send Flow (WhatsApp)

2. Idempotency and IDs

1. Access Pattern and Store

1. Three Receipt States

1. Why Fan Out on the Server

1. Offline Path and Push

1. Presence = Heartbeat + TTL

2. Who Gets Presence and Typing

1. Media Flow (WhatsApp Chat App)

1. Scale Out and Multi-Region

1. Numbers (WhatsApp Chat App)

3. Latency and Guarantees

2. Routing a Message

3. Backbone and Media Path

2. Connect Flow

3. APIs

4. Failures and Metrics

3. Ordering

4. Failure and Metrics

2. Data Model

3. API, Failures, Metrics

2. Receipt Flow

3. Data Model

4. Groups, Failures, Metrics

2. Fan-out on Write

3. Group Data and Order

4. Failures and Metrics

2. Reconnect and Drain

3. Failures and Metrics

3. Flapping and Failure

2. Signal Protocol and Keys

3. Trade-offs and Metrics

2. Trade-offs

3. Interview Questions

2. Path and Keys

3. Stores

4. Pitfalls