Scalability and Load Balancing

7 pages · ~5 min✓ Reviewed

Start here. Next up: Caching.

System DesignPage 1

Scaling Mental Model

1. Vertical vs Horizontal

	Vertical (scale up)	Horizontal (scale out)
Change	Bigger machine: more CPU/RAM	More machines behind a LB
Capacity	Limited, hard ceiling	Elastic, near-linear
Upside	Simple, no code change	Resilient, fault tolerant
Downside	SPOF, downtime on resize	Complex: needs stateless design, data consistency, networking

2. Example: 4-vCPU API at 90% CPU

Vertical: resize means downtime, still one box
Horizontal: 4 x 4 = 16 vCPU total, no downtime

3. Common Mistake

Scaling the app tier while a single database stays the bottleneck

Scale the DB too

Add read replicas, caching or sharding. More app instances just queue on the same database.

System DesignPage 2

Stateless Services

1. What Stateless Means

Stateless: any instance can serve any request
No per-user state in instance memory or local disk
Sessions → Redis, files → S3, data → database
Or carry state in a signed token (JWT)

2. Stateless vs Stateful

	Stateless	Stateful
Instances	interchangeable	unique
Kill, replace, autoscale	freely	risky
Needs	nothing extra	sticky routing or replication

3. Example: Session in Redis

Request 1Server A saves session
Rediskey = session ID cookie
Request 2Server B reads it, still logged in

4. Mistakes

In-memory session behind an LB → random logouts
Local-disk uploads → missing files

Fix

Use a shared store. Sticky sessions are a crutch, not the fix.

Interview Point: why not keep sessions in memory? The next request may hit an instance without them.
Stateless vs sticky sessions? Sticky pins a user to one server; stateless needs no pinning.

System DesignPage 3

Load Balancers: L4 vs L7

1. What a Load Balancer Is

One entry point: a VIP or DNS name
Spreads traffic across healthy backends
Hides backend addresses

2. L4 vs L7

	L4 (transport)	L7 (application)
Reads	IP + TCP/UDP port	HTTP host, path, headers, cookies
Wins	speed, any protocol	path/host routing, TLS termination, rewrites, sticky cookies
Cost	no content awareness	more CPU per request
Example	AWS NLB	AWS ALB, Nginx, HAProxy

Client
LBVIP / DNS name
Backendshealthy only

3. Mistake

The LB is a SPOF

Run it redundantly: multi-AZ, active-passive, or DNS/anycast across several LBs.

4. Interview Point

Path routing needs which layer? L7, it reads the HTTP path
Why pick L4? Speed, low overhead, any TCP/UDP protocol
How avoid an LB SPOF? Several LBs: multi-AZ or DNS/anycast

System DesignPage 4

Balancing Algorithms and Health Checks

1. Picking a Server

Algorithm	Picks	Best when
Round robin	Next server in order	Equal servers, similar request cost
Weighted RR	Bigger servers get more requests	Mixed server sizes
Least connections	Fewest active connections	Uneven or long-lived requests
IP hash	Same client or key, same server	Affinity, cache locality

Consistent hashing limits remapping when servers join or leave
Plain hash by server count remaps most keys on a change

2. Health Checks

LB probes each backend every N seconds
Probe: TCP connect, or HTTP GET /health expecting 200

Probeevery N s
Failure thresholdreached
Removedno traffic
Success thresholdreached
Re-addedserves again

3. Common Mistakes

Shallow /health returns 200 while the DB is down
Deep check hits the DB: one DB blip removes every instance at once
No connection draining on removal drops in-flight requests

Health check depth

Check the instance itself and its must-have dependencies, but never let one shared dependency fail all instances together. Always drain before removal.

System DesignPage 5

Latency, Throughput and Estimation

1. Latency vs Throughput

	Latency	Throughput
Measures	Time for one request	Requests or bytes per second
Unit	ms: report p50/p95/p99	QPS / RPS, bytes/s

concurrency = throughput x latency (Little's law)
200 RPS x 50 ms = 10 requests in flight
More instances raise throughput, not per-request latency
p99 climbs near 100% load: keep utilization 60-70% (30-40% headroom)

2. Estimation Recipe

Average QPSDAU x actions/day / 86,400
Peak2-3x average
Storagewrites/day x size x retention
BandwidthQPS x payload size

3. Example

10M DAU x 10 requests / 86,400 = 1,150 QPS average
Peak is about 3,000 QPS
3,000 / 500 per instance = 6 instances
Add one spare: 7 for N+1

4. Common Mistakes

Sizing for average load, not peak
Ignoring p99 and watching only the average
Forgetting 1 day is about 100,000 s (quick rounding)

System DesignPage 6

Scaling Cheat Sheet

1. Vertical vs horizontal

	Vertical	Horizontal
Change	Bigger box	More boxes
Good	Simple	Elastic
Catch	Capped, SPOF	Needs stateless

2. Stateless and L4 vs L7

Sessions in Redis
Files in object storage
Config from environment
No local state on the instance

	L4	L7
Routes on	IP + port	HTTP host, path
Traits	Fast, any protocol	TLS termination

3. Algorithms and quick math

Round robin: uniform servers
Weighted: mixed sizes
Least connections: uneven requests
Hash: affinity

1 day ≈ 86,400 s
Peak ≈ 2-3x average
Concurrency = RPS x latency
Run at 60-70% utilization, plus N+1 spare

Remember

Scaling out only works when instances are stateless; the single database stays the bottleneck.

System DesignPage 7

Check yourself

An API serves 200 RPS at 50 ms latency. You add instances and it now serves 400 RPS at the same latency. How many requests are in flight?

400 × 0.05 = 20 (concurrency = throughput × latency)
Adding instances raised throughput, not per-request latency

Behind a round-robin LB, users are logged out at random. Sessions sit in each instance's memory. Why, and what is the fix?

The next request lands on an instance with no copy of the session
Move sessions to a shared store like Redis; sticky sessions only hide the bug

Every /health check queries the database, and the database goes down. What does the LB do?

Every instance fails its check and is removed, so no healthy backends remain: full outage
Keep the check shallow, and handle the DB failure separately