What System Design Is: From One Server to Millions

It is 9:40 pm, ten days before Diwali. A food creator with 20 lakh followers posts a reel about Meera's small online shop of handmade brass diyas from Moradabad. In five minutes, 40,000 people tap the link. The shop runs on one rented server that holds both the website and its database. The processor hits 100%, pages take 20 seconds to load, and then the browser shows only an error. By the time the reel stops trending, the shop has lost the best night of its year. Nothing in Meera's code was wrong. The design was sized for 50 orders a day.

System design is deciding which machines a product runs on, how data moves between them, and what happens when one of them is slow, full or gone. Code decides how one machine does a task. A design decides how many machines keep doing it for millions of people while disks fail and traffic spikes.

There is no single right design, only decisions and their costs. This chapter follows Meera's shop from one server to millions of users, one bottleneck at a time. A bottleneck is the part that runs out of capacity first. Each step shows what breaks, with a number, what fixes it, what the fix costs, and which chapter goes deeper. Every part of a large system is there because a number forced it in.

Meera's first design was right for a new shop: one machine running both the web app and the database is cheap, quick to set up and has nothing to coordinate. It can serve a few hundred requests a second, far more than 50 orders a day need.

Two numbers describe how any system copes with load. Latency is how long one request takes, end to end. Throughput is how many requests the system completes each second. On the night of the reel, about 1,300 requests a second arrived (40,000 visitors making about 10 requests each over five minutes) at a box built for a few hundred. Requests waited in line, latency climbed from a fraction of a second to 20 seconds, and the rest timed out.

What breaks. The web app and the database fight over one machine: the database wants memory to keep popular rows close, and the app wants processor time to build pages. The box is also a single point of failure, a part whose failure stops everything: one crash, full disk or bad update takes the whole shop offline.

The fix. Move the database to its own machine, the way a restaurant moves its storeroom out of the kitchen: the cooks get room to work, but every ingredient now takes a short walk. Each machine is sized for its job, memory and fast disks for the database, processor cores for the app. The cost: every query now crosses the network, about 0.5 ms per round trip inside one datacenter (a building full of servers), so a page that runs 20 queries one after another gains 10 ms. There are also two single points of failure now, and still not enough capacity for 1,300 requests a second. For that, the app needs more machines. Databases: Replication, Partitioning and Choosing a Store goes deeper.

A year later, Meera plans the next Diwali sale and expects about 4,000 requests a second at the 9 pm peak. One app server handles about 1,000 simple requests a second, and it should not run near that limit: each request waits behind the others, and at 90% busy the wait is about ten times what it is at idle. There are two ways to get more capacity.

Bigger: vertical scaling, or scaling up. Move to a machine with more cores and memory. Nothing in the code changes. But there is a ceiling (the largest machines have a few hundred cores), changing size usually means a restart, and it is still one machine that can fail.

More: horizontal scaling, or scaling out. Run several ordinary machines side by side. The app has no practical ceiling, and one machine can fail while the others carry on. The price is new work: something must spread requests across the machines, and no machine may keep a user's data in its own memory.

Think of one bigger truck against a fleet of tempos: the truck is simpler to run, but it has a size limit and one breakdown stops every delivery. Real systems use both. Scale up the database, which is hard to split; scale out the app servers, which are easy to copy. Availability, Reliability, Scalability and Latency compares the two in depth.

Meera's team scales out, and the first attempt fails in a useful way: with two app servers, shoppers are logged out and their carts empty at random. Each server kept sessions (who is logged in, what is in the cart) in its own memory, and the next request often landed on the other server.

Two changes fix it. A load balancer is one address in front of all the servers that hands each request to one of them, like the token counter at a bank that sends you to the next free window. It checks each server's health every few seconds and stops sending traffic to one that fails two or three checks in a row. And the servers become stateless: they keep nothing about a user between requests. Sessions and carts move to a shared store that every server can reach, or into a signed token the browser sends back each time. Any server can now answer any request, the way any bank window can serve you when your file is not locked in one clerk's drawer.

The numbers. 4,000 requests a second at about 1,000 per server is four servers running flat out. The team runs six, so each sits near 67%, and losing one leaves five at 80%. The cost: the load balancer is a new part that can fail, so it runs as a pair or as a managed service, and reading a session from the shared store adds under a millisecond to each request. Load Balancers goes deeper.

With six app servers, the bottleneck moves to the one database behind them. Each request makes about five database reads on average, so the sale's 4,000 requests a second become 20,000 reads a second, plus a few hundred writes for orders and carts. The database runs at 95% busy, where each query waits about twenty times longer than at idle: pages that took 60 ms now take over a second, and checkout writes wait behind people browsing.

The fix. Add read replicas: copies of the database on other machines. The original, now called the primary, takes every write and streams each change to the replicas, which serve reads. It is like printing extra copies of a timetable: everyone can read one, but every change must reach every copy. Two replicas take the browsing reads, about 10,000 a second each, and the primary keeps the writes and the reads that must be exact, such as a payment's status.

The cost. Replication is usually asynchronous: the primary confirms a write without waiting for the replicas, so they run a little behind, well under a second normally and seconds when busy. A shopper who places an order and opens 'My orders' a moment later may not see it if that read goes to a replica, so the app sends each user's reads to the primary for a short while after they write. Replicas do nothing for writes, because every replica must apply every write too. Databases: Replication, Partitioning and Choosing a Store goes deeper, and Consensus, Leader Election and Coordination covers promoting a replica when the primary dies.

Replicas add capacity, but each one copies the whole database to serve what is mostly the same few hundred sale products, read again and again. A cache keeps those hot answers in memory, like a shopkeeper who keeps the day's best sellers on the counter instead of fetching each one from the godown.

The mechanism. On each read, the app server asks the cache first. On a hit, the answer comes back in under a millisecond. On a miss, the server reads the database, about 50 ms for this shop's product query, and stores the result in the cache with an expiry time, such as 60 seconds. One cache node handles about 100,000 simple operations a second.

The numbers. Of the 20,000 reads a second, suppose 95% are hits. Then only 1,000 a second reach the database, a twentieth of the load, and most pages get faster too. Replicas multiply capacity; a cache removes demand. That is why many teams add a cache before any replica.

The cost. A cached answer can be stale: if Meera changes a price, shoppers may see the old one until the entry expires, unless the app deletes the entry when the price changes. The cache is one more system to run. And if it restarts empty, every read falls on the database at once, so the database must survive that burst or the cache must refill gradually. Distributed Caches goes deeper.

After six steps, the shop has stateless app servers behind a load balancer, a cache, and a database with replicas. Before going further, follow one sale-night request through that picture, including the moment a server dies.

The miss cost about 50 ms of server time and the hit about 5 ms, so the cache decides most page times; a phone's mobile network adds about 60 ms to either. When server 3 died, the shop lost capacity but no data, because servers keep nothing between requests. That is why the team runs six servers for a load four could carry: spare capacity turns a crash into a non-event.

The pages are now fast, but the photos are not. Of the sale's 4,000 requests a second, about 1,500 open a product page, and each page pulls about 1 MB of photos: 1.5 GB a second, about 12 Gbps, all sent from servers in Mumbai. A shopper in Guwahati, about 2,000 km away, pays at least 20 ms for every round trip, often twice that on real routes, and a page needs several.

The fix. A content delivery network (CDN) is a network of edge servers in many cities that keep copies of static files: photos, scripts, stylesheets and video. It works like stocking regional warehouses instead of shipping every parcel from one city. The first shopper in Guwahati to ask for a photo makes the nearby edge fetch it from the shop, a miss; everyone after that gets the edge's copy in a few milliseconds, a hit. The photos themselves move off the app servers into a blob store, a storage service built for large files, so the servers stay stateless.

The numbers. If the edges serve 95% of photo bytes, the shop's own servers send 0.6 Gbps instead of 12. The cost: a changed photo can stay stale at the edges until its copy expires, so files get versioned names such as diya-42-v3.jpg, and a change means a new name. You pay the CDN for every gigabyte it delivers, and it helps only with content that is the same for many people, not a shopper's cart. Content Delivery Networks and Blob Stores go deeper.

Checkout is the next thing to strain. Placing an order used to do everything before replying: save the order (about 10 ms), make the invoice PDF (300 ms), send the email (400 ms) and the SMS (300 ms). The shopper waited over a second. On sale night the SMS provider slowed to 5 seconds a message, and each checkout held a server thread while it waited. At 50 checkouts a second, that is 250 requests stuck at any moment, and browsing requests began to queue behind them.

The fix. Save the order, put a message such as 'order 9812 placed' on a queue, and reply in about 50 ms. Workers, separate processes, take messages off the queue and do the slow work at their own pace, retrying when a provider fails. It works like the order slips on a restaurant's rail: the waiter takes your order and moves on, and the kitchen works through the slips. When the SMS provider slows down, the queue grows; checkout does not.

The numbers. If orders arrive at 50 a second and workers send 30 SMS a second, the queue grows by 20 a second: about 12,000 messages after ten minutes, which the workers clear once the rush eases. The cost: the SMS arrives later, minutes late at the worst. Most queues deliver each message at least once, so one can arrive twice, and a worker must check 'SMS already sent for order 9812' before sending. Queue depth also becomes a number to watch. Message Queues and Publish-Subscribe go deeper.

Three years on, the shop is a marketplace with 10,000 sellers. At the festival peak, orders, cart updates and stock changes reach about 15,000 writes a second, and the orders table holds 8 TB. One primary handles thousands to tens of thousands of simple writes a second and a few to tens of terabytes, so the shop is near the limit. The next step splits the data into shards: slices that each live on their own primary, with their own replicas. The scene replays the database's story, then takes that step.

Copies fixed the reads, because any copy can answer a read. Only splitting fixes writes: a replica must apply every write, while a shard takes only its own share.

Sharding is like splitting one thick register into four by roll number, so each clerk handles a quarter. The shop assigns users to shards by user id, so all of one user's orders live together. It runs four shards, not the scene's two: each primary takes about 3,750 writes a second and 2 TB.

The cost. Anything that crosses shards gets harder, the way finding every student from one school now means checking all four registers. A seller's orders are spread over every shard, so the shop keeps a second copy arranged by seller. One transaction cannot easily span two shards, adding a shard means moving data while serving traffic, and one viral product's stock count still lands on a single shard. Shard last: it is the hardest step to undo, and many products never need it. Databases: Replication, Partitioning and Choosing a Store and Sharded Counters go deeper.

Everything so far runs in one region, a group of datacenters in one metropolitan area, here Mumbai. Inside it, the shop already spreads servers and database copies across two or three zones, separate datacenters with their own power and network, so a fire or power cut in one is survivable. A whole region can still fail, from a network fault, a disaster or a bad change pushed everywhere at once. If the shop promises 99.99% availability, it may be down about 52 minutes a year; one three-hour regional outage spends more than three years of that allowance.

The fix. Run a full copy of the shop in a second region, such as Delhi, and copy the data there continuously. DNS, the internet's address book, or a global load balancer sends shoppers to a healthy region. It is like a bank keeping a second branch, with copies of every ledger, in another city.

The cost. Distance makes copying slow. Mumbai to Delhi is about a 30 ms round trip, so waiting for Delhi to confirm every write would add at least 30 ms to each one; most systems copy asynchronously and accept that a regional disaster may lose the last few seconds of writes. Switching regions takes minutes and must be practised, or it fails on the day it is needed. Two full-size regions roughly double the bill, so many teams keep a smaller standby that grows when called on. And if both regions accept writes for the same data, two updates can collide. Availability, Reliability, Scalability and Latency, DNS: How a Name Becomes an Address, and Lessons from Real Outages go deeper.

Here are the ten steps together. As you step through them, recall the number that forced each one.

The order is not a law. A news site full of readers may add a cache and a CDN in its first month; a payments app may run a second region for safety long before its traffic needs one. What stays the same is the method: measure, find the part that runs out first, fix it the cheapest way that works, and expect the bottleneck to move. Add a step only when a number demands it, because every step is one more part that can fail.

The building-block chapters take each of these parts apart. The product chapters put them back together for sixteen real products.

Meera's shop grew one bottleneck at a time, reacting to numbers as they arrived. A new product starts before any traffic exists, with requirements and one number. Here is a small one to design from scratch.

The product: the registration site for a college fest. Registrations open at 10 am, and most students sign up in the first few minutes.

Requirements come in two kinds. Functional requirements say what users can do. Non-functional requirements say how well the system must do it: how fast, how available, how correct, each with a number where possible, because 'fast' cannot be tested and 'confirmed within a minute' can. Asking the fest team who uses the site, what they do and what must never happen gives this list:

One requirement matters most, and it shapes the design: never take a payment without a registration. For a link shortener the deciding requirement would be fast redirects; for a chat app, message order. The other key word is rush: the site is idle most of the year and gets nearly all its traffic in a few minutes. Before drawing anything, size that rush.

A rush is not a number yet. Four assumptions make it one: how many students register, how many requests each makes, how long the rush lasts, and how much hotter its first minute is than the average.

Now place that number on the shop's journey. At about 1,000 simple requests a second per server, two app servers carry the peak at half capacity and a third covers a crash: step 4. Writes are small, about 100 registrations a second at the peak, far inside one database. Confirmation emails go through a queue, as at step 8, so a burst of them cannot slow registration.

Everything else on the journey would be waste here: no replicas, cache, shards or second region. Each would add cost and new ways to fail, for a site that runs hot for ten minutes a year. The right design is the smallest one that meets the requirements and the numbers.

With requirements and a peak number agreed, the first sketch almost draws itself. Start with the three calls that the student's phone and the payment gateway make.

POST /registrations          { email, events[] }        -> 201 { id, paymentUrl }
POST /payments/callback      { id, status, paymentRef } -> 200   (from the payment gateway)
GET  /registrations/{id}                                -> 200 { status, qrCodeUrl }

Three calls cover the whole flow; everything else is detail.

Three choices make this design safe. First, the callback is idempotent: doing it twice has the same effect as doing it once. The handler runs UPDATE registrations SET status = 'paid' WHERE id = ? AND status = 'pending'; if no row changes, the callback is a repeat and nothing else happens, so a retry cannot send a second email. It also checks the gateway's signature, so nobody can fake a payment.

Second, slow work goes through a queue, so a burst of confirmations cannot slow registrations. Third, the callback is never the only path. A UPI payment can stay pending for minutes and a callback can be lost in the rush, so a reconciliation job asks the gateway about every registration still pending after 15 minutes and marks it paid or failed. That job keeps the promise 'never take a payment without a registration'. Design a Payment System goes deeper.

Design interviews ask for what you just did, on a whiteboard, in about 40 minutes. A 45-minute slot loses about 5 minutes to introductions and your own questions at the end, and the time that remains follows six steps.

The fest design maps onto it step for step: the requirements, the rush estimate, the three calls, the sketch, and a deep dive on the callback and reconciliation. Two habits from this chapter carry most of the weight. Start with the smallest design that meets the requirements and let numbers force each addition, as the shop grew. And for every box you draw, say what happens when it is slow or gone.

The last chapter, How to Practise: A 45-Minute Checklist, turns these steps into a checklist with practice alarms, phrases that help, a worked round, mock prompts and a scoring rubric.