Availability, Reliability, Scalability and Latency

It is 8:40 pm on the night of a cricket final, and the first innings has just ended. On a food delivery app in Bengaluru, orders jump from 1,500 a minute to 6,000 in five minutes. The order servers hit full CPU, the database runs out of connections, and checkout takes 8 seconds instead of a third of a second. Hungry users tap Place order again, and every tap adds load. By 8:50 the app is down at the busiest moment of its year, yet no machine was broken. It had no spare capacity, no limit on repeated requests and no way to drop less important work first.

Each gap is a non-functional requirement: how well the app does its job, rather than what the job is. The fix starts with writing each quality as a number the design must meet. The delivery app, followed through this whole chapter, writes four:

A wish such as "fast and always up" cannot be tested; a number can, and each number points to a mechanism, starting with what 99.95% allows.

What does the first promise, 99.95%, actually allow? Availability is the share of time, or of requests, for which a system does its job for users. It is usually quoted by its count of nines: "three nines" is 99.9%. Each nine you add divides the downtime you are allowed by ten. The table turns each level into time, for a 365-day year and a 30-day month.

The delivery app's 99.95% allows about 22 minutes of failed ordering a month. Five nines leaves about five minutes a year, less than it takes a person to notice an alert and log in, so very high availability has to come from automation, not from people reacting.

Planned downtime counts too, if users feel it. A 30-minute database upgrade each month would spend the whole 22-minute budget before anything had broken. That is why upgrades are done on a standby copy, which then takes over in seconds.

Machines fail every week at scale, so how does the app lose one and still stay inside 22 minutes a month? Any part whose failure takes the whole system down is a single point of failure (SPOF). The cure is redundancy: run more than one of everything that matters, so that losing a copy becomes a capacity problem instead of an outage.

At an ordinary dinner peak the order service handles about 1,800 requests a second, and one server handles about 1,000. Compare one big server with three smaller ones behind a load balancer, which checks the health of each server and sends traffic only to healthy ones.

Setup B kept serving only because the survivors had spare capacity: 1,800 requests a second is 60% of three servers but 90% of two, and at 90% requests queue and slow down, as the latency part of this chapter shows. So plan for N + 1: enough servers to carry the peak with one missing, each near 60 to 70% busy.

The load balancer is now a single point of failure itself; the load balancer chapter gives it a partner.

An order touches several parts, so how do their availabilities combine? Think of a chain and a rope. A chain breaks if any link breaks: when a request needs every part, the parts are in series, and their availabilities multiply. A rope holds while any strand holds: when any one copy can serve, the copies are in parallel, and it is their chances of being down that multiply.

The same arithmetic along the order path:

Doubling everything looks like six nines on paper, but the parallel formula assumes copies fail independently. Copies in one rack, on one power feed or receiving the same bad deploy fail together, so real targets stop at 99.95 to 99.99%. The database is the hardest part to double, because its copies must agree on the data.

Staying up is only the first promise. The second said a confirmed order is never lost, and that is a different quality. Reliability means the system keeps doing the right thing over time: correct answers and no lost work. Availability only asks whether it answers. The two usually move together, but not always.

Picture two versions of the delivery app. One is always up but silently drops one order in a thousand: available, not reliable. The other goes offline for an hour on a quiet Tuesday morning but never loses an order: reliable, less available. Users forgive the second far more easily, because a lost order is a hungry customer who has already paid.

Durability is the narrowest promise: once a write is confirmed, it is never lost. It comes from copying each write to several machines, often 3 in different zones, before the app shows "Order placed". A database that is down for an hour but loses nothing was unavailable yet durable. A server that answers every request while the only copy of the data sits on one disk is available, until that disk dies and the data is gone for good.

Extra copies are one form of fault tolerance: keeping a fault, one component going wrong, from becoming a failure, the system as a whole stopping service. Faults are routine at scale: disks fail at roughly 1 to 2% a year, so a fleet of 10,000 disks loses one every few days. Spare servers, extra copies, timeouts and graceful degradation (if recommendations fail, show a plain list of nearby restaurants) keep most faults away from users. The ones that get through need fast recovery.

Failures will happen, so how fast the app recovers matters as much as how often it fails. Two numbers capture this. MTBF (mean time between failures) is how long the system runs, on average, before something breaks. MTTR (mean time to recovery) is how long it takes to get back to normal. Availability is roughly MTBF ÷ (MTBF + MTTR).

Suppose something takes the order database down about once a month: a dead host, a full disk, a bad query. Compare three ways that can go:

The last two rows land in the same place: cutting MTTR from an hour to six minutes gains as much as making failures ten times rarer, and it is usually far easier. The hand-fixed monthly failure misses the app's target, because an hour a month is nearly three times its 22-minute budget.

Recovery time splits into four steps, and each can be shortened:

Automation shortens detect and repair the most: a health check notices in seconds, and a standby is promoted without waiting for a person. Checkpoints shorten repair for long jobs: the nightly payout job to restaurants saves its place every 10 minutes, so a crash costs at most 10 minutes of work. Then test it all, because a failover script or backup never run on purpose may not work when it matters.

The third promise was 5× the usual dinner peak. Scalability is the ability to handle more load by adding resources, with cost growing roughly in proportion. To scale up is to move to a bigger machine: no code changes, but a ceiling, and still one machine that can fail. To scale out is to add machines behind a load balancer: no practical ceiling, but the servers must be interchangeable. New servers take minutes to start, so the app adds them on a schedule before big matches.

Interchangeable means stateless: a server keeps nothing about a user between requests. The delivery app keeps each cart in a shared cache, not in the memory of the server that built it, so any server can take any request. A stateful server keeps sessions in its own memory, which ties users to one machine and loses their carts when it dies.

Statelessness does not remove state; it pushes it down to the data tier. That is why the database is the usual bottleneck: app servers can be multiplied in minutes, but the data everyone shares lives in one place. Relieve it in roughly this order, cheapest first:

The order also protects maintainability: how cheaply a system can be run and changed after launch, where most of its cost arrives. Every extra store is one more thing to understand and repair at 3 am, so one relational database, which handles thousands of writes a second, is a better start for orders than three specialised stores.

The fourth promise was about speed. Latency is how long one request takes, in milliseconds. Throughput is how many requests the system completes each second. Think of a toll plaza: latency is how long one car spends at the booth, throughput is cars per hour. Opening more booths raises throughput without making any single car faster.

The two are tied by Little's law: requests in flight = throughput × latency. If the home-screen API serves 2,000 requests a second at 50 ms each, about 100 are in progress at any moment. If latency doubles to 100 ms at the same traffic, 200 are in flight, and threads, memory and connections must stretch to hold them.

This is how the database ran out of connections in the opening story. Each app server keeps a connection pool, a fixed set of open database connections that requests borrow and return. The restaurant service sends the menu database about 2,000 queries a second at 5 ms each, so about 10 connections are busy. If a lock pushes queries to 100 ms, 200 are needed. When the pool runs dry, requests wait for a connection while the app servers look idle.

Adding app servers then makes it worse: 100 servers with 50 connections each would open 5,000, far more than most relational databases handle well. Size pools with Little's law, keep them small, and fail fast when none is free, with a connection pooler in front of the database sharing a few hundred connections among all the servers.

Little's law shows how slow requests pile up. But why does a request slow down at all, just because its server is busy? Think of a single ticket counter. When people arrive faster than the clerk can serve them, even for a moment, a queue forms, and the busier the counter, the longer each burst takes to clear.

A simple queueing model puts a number on it: the average time a request spends at a server is its service time ÷ (1 − utilisation), where utilisation is the share of time the server is busy. The table shows how fast that grows:

The curve is gentle until about 70% and then turns steep, so capacity plans target 60 to 70% utilisation at peak, leaving room for bursts to clear.

Now the redundancy scene adds up. Three order servers at 60% each take about 2.5× their service time; when one dies, the survivors run at 90% and take about 10×. That is slow, but standing. Run two servers at 90% instead, and losing one puts 180% of a server's work on the other: an outage. Real servers are messier than this model, but the shape of the curve holds.

A busy moment slows some requests far more than others, so how should latency be measured? Not by the average, which hides the slow requests users complain about. Use percentiles: the p50 (median) is the time half of all requests finish within, the p95 the time 19 in 20 finish within, and the p99 the time 99 in 100 finish within. A p50 of 40 ms with a p99 of 800 ms feels fast for most users and broken for one request in a hundred.

One in a hundred reaches more people than it sounds, because a page makes many calls. The delivery app's home screen asks about 10 services at once (nearby restaurants, offers, delivery times, ratings, the cart and more) and must wait for all of them. For n calls, the chance that at least one is slower than its own p99 is 1 − 0.99ⁿ:

So about one home-screen load in ten meets some service's slowest 1%. That tail latency, the slowest 1% or so of each part, becomes an everyday experience for the whole page.

Tails come from garbage-collection pauses, cache misses, busy neighbours on shared machines, queueing and retries. So set targets on a high percentile, make fewer calls, and give optional ones tight timeouts. For reads that are safe to repeat, send a hedged request: if no reply has come by the p95, ask another copy and use whichever answers first, for about 5% extra load.

A p99 target of 300 ms says where the app must end up, not how to get there. A latency budget splits the target across the hops a request takes, so each team knows its share. If the shares do not fit, you learn before building which call to cache, run in parallel or drop. Here is the home screen's budget for a phone on a mobile network:

The phone network and one database miss already take more than a third of the 300 ms, so the app caches the list of nearby restaurants and keeps the miss rare.

The headroom row is not waste: it absorbs a retry or a slow tail, and a budget without it fails on the first bad evening.

Every number so far assumed the app could keep up with its traffic. The last part asks what happens when it cannot.

At the innings break, the app met more traffic than it could serve. Every system has a limit, and a sale, a final or a retry storm will find it. Past that point some requests must fail; the only choice is which ones, and how fast. A server that accepts everything builds a queue, and at 99% busy each request takes about 100 times as long, so it ends up answering users who have already given up.

An overloaded system must say no early and cheaply, so that the work it accepts still succeeds. Five tools do this:

Shedding feels harsh, but a server that queues everything serves almost nothing on time, while one that quickly rejects a fifth of requests serves the other four fifths well. Rejected clients should wait and try again later, and how they retry decides what happens next.

Rejected and timed-out calls tend to come straight back. A slow database makes every caller retry, so it gets more work exactly when it can least cope; three layers of 3 attempts turn one tap into 27 calls, as the abstractions chapter showed. This retry storm can outlast the fault that started it.

The breaker took the load off, and the cache kept diners served. A circuit breaker wraps calls to one dependency: closed, calls flow and failures are counted; open, after a threshold such as half the calls in 10 seconds, calls fail at once for perhaps 30 seconds; half-open, a few trial calls decide whether it closes. Three habits help:

• Retry budget: retries may add at most about 10% to normal traffic; once it is spent, fail fast instead.
• Fallbacks: while a breaker is open, serve a cached or simpler answer, such as menus a few minutes old.
• Ramped recovery: let traffic back in steps, because cold caches and queued retries can knock a service over again.

Everything so far runs in one place. What if the place itself fails? Cloud providers group data centres into zones, each with its own power and network, and nearby zones into a region, such as Mumbai. The app spreads its servers and database copies over three Mumbai zones, which survives a zone failure at little latency cost. A second region, Delhi, guards against the rest: a regional disaster, or a bad change pushed to every zone.

Two numbers set the design. The recovery point objective (RPO) is how much recent data a disaster may lose, measured in time. The recovery time objective (RTO) is how long the service may stay down while it moves. RTO is an MTTR target for a disaster; RPO is a durability target. The app picks an RTO of 30 minutes, and an RPO of zero for payments and a few seconds for everything else.

The copying between regions decides the RPO. Synchronous replication waits for Delhi to confirm each write before answering the user, so losing Mumbai loses nothing, but every such write pays the Mumbai to Delhi round trip of about 30 ms. Asynchronous replication answers at once and copies a moment later, so the RPO is the replication lag, usually seconds: if Mumbai fails at 9:00:00 with a 2-second lag, orders confirmed after 8:59:58 never reached Delhi.

That is why synchronous copying is kept for data whose loss is unacceptable: payment records, not carts, ratings or logs. The split has a useful side effect. If a disaster loses an order from its last few seconds, its payment record survives in Delhi, so the app can find the charge and refund it or restore the order.

With the data copied to Delhi, who serves users from there? There are two shapes:

The delivery app starts active-passive: a 30-minute RTO does not need the cost and complexity of active-active. Three details decide whether its failover actually works:

• Steer traffic with DNS or anycast. DNS failover waits for cached answers to expire, often minutes, as the next chapter explains; anycast reroutes faster but needs a large provider's network.
• Fence the old region. Make sure the old primary has stopped taking writes before the standby starts, or both change the same orders; the coordination chapter shows how.
• Keep data where the rules require it. Indian rules require payment data to be stored in India, so both regions stay in the country; check current rules.

A failover that has never been run is a hope, not a plan. Rehearse it on a schedule, time it against the RTO, and practise failing back too.

These lines turn the chapter into targets and mechanisms, each carrying its number or its cost: