Lessons from Real Outages

It is 9:40 pm on the night of a cricket final, and a tea stall near Pune station is packed. A fan scans the stall's QR code to pay ₹20. The payments app spins, then fails, and across India millions of payments fail the same way. No server has died. Ten minutes ago, an engineer shipped a one-line config change that cut the timeout on database calls from 2 seconds to 200 milliseconds. Every server loaded it within seconds. At peak many queries take longer than that, so calls time out, every layer retries them, and the database sinks under many times its normal load.

This payments app is the chapter's running example, because its bad night holds most of the lessons. Disks and servers die every day at scale, and the redundancy from Availability, Reliability, Scalability and Latency absorbs them; the outages that make the news start with something ordinary, like that config change.

After a serious incident, well-run teams write a postmortem: the timeline, the impact, the trigger, why the damage spread, and the actions that stop a repeat. Some publish theirs. This chapter tells seven patterns from public postmortems as short stories: what happened, why it spread, and what stops it. Six follow one published write-up each, at the level it describes; the seventh, overload, appears in too many to need one.

Triggers are small and cannot be ruled out: people mistype, and tests miss cases. Spread and recovery are where design helps. Most lessons in this chapter make sure one mistake reaches only a small part of the system and can be undone quickly. The next page asks how small that part can be.

The tea-stall outage reached every user because every server took the change at once. Blast radius is how much of a system, and how many users, one failure or one change can affect. Think of the sealed compartments in a ship's hull: a leak floods one compartment, not the ship. Systems get compartments in two ways: split the system into parts that fail independently (in space), and change one part at a time (in time).

These are the usual units of space, from smallest to largest:

Servers, zones and regions mostly contain hardware faults; only cells also contain a bad change, because each cell can be changed on its own. Take the payments app's 20 million users. As one big deployment, a bad change reaches all 20 million at once. Split into 10 cells of 2 million and change one cell at a time, and a mistake reaches at most 2 million before anyone reacts. Start each rollout on 1% of the first cell, and about 20,000 people meet it first.

Cells are not free: ten copies of every component to run, watch and upgrade, plus a routing layer that maps each user to a cell. Below a few million users, zones and staged rollouts give most of the protection for far less work. Ask of every component and every tool: what is the most this one thing can break?

The first story shows how far one typed command can reach when the tool behind it sets no limit. In 2017, a large cloud provider was debugging a slow billing process in its storage service in one region. Following an established procedure, an authorised engineer ran a command to take a few servers out of service. One input was entered incorrectly, and the command removed a larger set of servers than intended, including servers that two core parts of the storage service depended on.

Why it spread. Both parts needed a full restart, which had not been done in the larger regions for many years. The service had grown a lot since, so the restart and its safety checks took longer than expected. Many websites and services that kept files there broke too. For a while even the provider's status page could not be updated, because its admin tool depended on the same storage service.

What stops it. Each weakness has a matching fix:

The provider's published follow-ups matched three of these rows: the tool now removes capacity more slowly and refuses to go below a subsystem's minimum, and the status page's admin tool runs across several regions. It also said it would speed up splitting the slowest part to recover into cells.

The command removed capacity. The next story added a file, and the file reached every machine at the same moment. In 2024, a cybersecurity company sent a routine content update to the security agent that runs on its customers' Windows computers. The update was a data file the agent reads, not new program code. The file triggered a bug in the agent, which runs deep inside the operating system, and the computers crashed.

Why it spread. The update went to customers' machines all at once, instead of to a small group first, so no small group could reveal the problem. The company soon pulled the bad file, but the agent loads during start-up, so many machines crashed again on every restart, before they could fetch the fix. Airlines, hospitals, banks and broadcasters were among those hit, and a large number of machines had to be repaired by hand, one at a time.

The pattern reaches far beyond security software. Many teams stage code carefully, yet treat config, feature flags, rules and data files as safe to change everywhere at once. A config change can break a system as badly as a code change, and config systems are built to be fast, so a bad value reaches every machine in seconds. The payments app's timeout was exactly this: one value in a config file, sent down a fast path that skipped the stages its code went through.

The third story is another fast path for data, with burnt CPU instead of crashes. In 2019, a company whose network sits in front of a great many websites, to speed them up and protect them, added a rule to its web application firewall, the layer that inspects requests for attacks. The rule contained a regular expression that made the matching engine do an enormous amount of work. CPUs on the machines serving web traffic went to nearly 100%, and visitors to the sites behind the network got errors.

Why it spread. The company's software releases went out in stages, but firewall rules were deployed to every server in one go. Service came back within about half an hour, when the team used a global switch to turn off the firewall's managed rules; the bad rule was then removed and the rules switched back on.

The engine was working as designed. Most regex engines in popular languages backtrack: when part of a pattern fails, they go back and try another way to match. With some patterns, such as a repeat inside a repeat, the number of ways to try doubles with each extra character. This is catastrophic backtracking.

pattern           ^(a+)+$
'a' × 40          matches at once
'a' × 20 + 'b'    fails after ~1 million tries: milliseconds
'a' × 30 + 'b'    fails after ~1 billion tries: seconds
'a' × 40 + 'b'    fails after ~1 trillion tries: minutes to hours

A textbook pattern, not the incident's rule. Times are rough and depend on the engine and machine.

What stops it. Input that matches is instant, so tests that only check what a rule catches will pass it. Use an engine built on the RE2 design, such as Go's standard regexp package, whose time grows only linearly with the input, at the cost of features such as backreferences. Otherwise cap the time one match may take. Keep a kill switch per feature, as the company did, and stage rules too: even a fast lane can send a rule to 1% of servers and watch CPU for a few minutes. The payments app's firewall rules need the same care.

The next story shows a safety check doing harm because every site ran it at once. In 2021, a large social network vanished from the internet for several hours. During routine maintenance, a command meant to check the capacity of its backbone, the private network joining its data centres, instead took down every backbone connection. An audit tool built to stop commands like this had a bug and let it through.

Why it spread. With the backbone gone, the data centres were cut off from each other and from the internet. The company's DNS servers were designed to withdraw their BGP (Border Gateway Protocol) announcements, the messages that tell other networks how to reach them, whenever they could not reach the data centres. That is a sensible check for one sick site, but every site tripped it together, so the rest of the internet could no longer find any of the company's names.

Recovery was slow. As DNS: How a Name Becomes an Address warned, everything that depends on names breaks along with DNS, and that included many internal tools; remote access, even the backup path, was down too. Engineers had to go to the data centres in person, where machines are deliberately hard to reach and change. Once the backbone returned, services came back in stages to avoid a dangerous surge, something the company had practised in drills.

The first rule applies to the payments app's load balancers: if most servers fail their health check at once, the balancer keeps them all in rotation, because a broken check is more likely than a dead fleet.

The stories so far were about spread; this one is about recovery when the last line of defence was never tested. In 2017, a company that hosts code for developers was fighting a database problem late at night: spam was loading the primary database, and the replica copying it had fallen behind. While rebuilding the replica, an engineer ran a command to delete its data directory, in a terminal connected to the primary instead. Within seconds, almost all of the production database was gone.

Why recovery failed. The regular dumps had been failing silently, because the backup tool's version did not match the database's, and the emails reporting the failures were being rejected. Other backup methods turned out to be missing or broken too. The restore came from a copy an engineer had happened to take some hours earlier for a test environment, so changes made since, such as new issues and comments, were lost; the code in the repositories was not affected.

What stops it. Four habits would have turned this into a short outage:

The RPO and RTO from Availability, Reliability, Scalability and Latency turn these habits into requirements. For the payments app's ledger, nightly dumps could lose up to a day of payments, which no bank would accept, so it also ships the database's log of changes to backup storage every few minutes: an RPO of minutes even for a mistake the replicas copied. Only a timed restore drill tells you your real RPO and RTO.

Every story so far began with a change someone made. The next kind of outage needs no change at all: it waits for a date. In 2018, a certificate built into software from a telecom equipment maker expired. The software ran inside the networks of several mobile operators in more than one country, and when the date passed it stopped working. Millions of people lost mobile data for most of a day, until fixed software was installed.

Why it spread. This is a time bomb: a failure that waits for a date or a count. Common ones are a TLS certificate or API credential that expires, a licence date, and a counter stored as a signed 32-bit integer, which tops out at about 2.1 billion. Every copy of the software shares the same calendar and the same counters, so a time bomb goes off everywhere at once. No canary can catch it, because nothing was deployed.

The payments app carries time bombs too: the certificates on its connections to each bank, and any transaction id stored in 32 bits.

The last pattern is the one behind the tea-stall outage, and it runs through so many public postmortems that it needs no single company. A dependency slows for a moment: a database fails over, a burst of traffic arrives, or a timeout is set too short. Calls time out, and every layer that sees a timeout tries again. The retries land when spare capacity is lowest, so more calls time out and more retries follow.

Why it spread. Each retry setting looks harmless alone. In the payments app, a payment goes from the phone app to an API gateway, then to the payments service, then to its database, and each of the three layers tries a failing call up to 3 times. The overload can outlast its trigger, because the retries alone keep the database past its limit.

A database that was already struggling at 10,000 calls a second now gets 270,000, and most of that work is wasted: by the time it answers, the caller has given up and tried again. A retry that rescues one lost packet becomes an attack on your own system when a whole dependency is slow.

What stops it. Availability, Reliability, Scalability and Latency teaches these defences in full. Here, only the gateway retries, within a budget of about 10% extra calls and with jittered backoff, so the database sees at most 1.1 times its load, not 27. A circuit breaker stops calls to the stalled database for a few seconds, low-value work is shed, and traffic returns in steps. Timeouts are set a little above the healthy p99, never guessed. And every retry of a payment carries the same idempotency key, as Design a Payment System explains, so a retry never charges anyone twice.

Line the stories up and one lesson stands out: in most of them, one change reached everything at once. A staged rollout sends a change to a small part of the system, watches it, then widens step by step. The first small group is the canary, named after the birds miners once carried underground as an early warning of gas.

Here is the payments app's timeout change, config v42, shipped two ways: to every region at once, then to a watched 1% canary.

The bug was identical both times. What changed was how many people met it before the system reacted: everyone, or about 1%. Staging does not prevent the mistake; it caps the damage and buys time to undo it.

A typical plan, with example waits, looks like this:

As Monitoring: Metrics, Logs, Traces and Alerts explains, the canary is compared with a baseline, a same-sized group still on the old version, and the rollout rolls back on its own when the canary is clearly worse. Some bugs, like the short timeout, show only at peak, so let a stage run through a busy hour, and freeze risky changes before known peaks such as a final or a festival sale.

The next question is how fast you can undo a change that got past the canary. When a change goes wrong, the first job is to stop the harm, not to understand it. This is mitigation; the root cause can wait until users are safe.

Two tools make mitigation fast. A kill switch turns one feature off at runtime, without a deploy, usually through a flag the code checks; one restored service in the firewall story. If the payments app's new cashback feature starts failing, a switch turns cashback off within a minute while payments keep flowing. Rollback returns to the last version that worked, and it is usually faster and safer than writing a fix under pressure. Keep every change reversible: keep the previous build ready, version every config file, and change database schemas in steps the old code can still run against.

Brakes on automation are the other half, the lesson of the command that removed too much. Automation makes bad changes as fast as good ones, so any tool that can touch thousands of servers needs limits. It is the idea of Rate Limiters, applied to your own operations:

def remove_servers(pool, servers):
    if len(servers) > max(1, pool.size // 20):
        raise Refused("more than 5% of the pool in one command")
    if pool.healthy - len(servers) < pool.minimum:
        raise Refused("would leave the pool below its minimum")
    for batch in chunks(servers, size=2):      # slow on purpose
        drain_and_remove(batch)
        if not healthy_for(pool, minutes=5):   # stop at the first sign of harm
            raise Refused("pool unhealthy, stopping here")

Three brakes: a cap per command, a floor the pool never drops below, and a slow pace that halts on bad signals.

Stages, switches and brakes make outages rarer and smaller, but something will still break. How the first hour goes depends on preparation done months earlier: who does what, and whether your repair tools work when production does not. Most teams name four roles:

Clear roles keep five people from trying five fixes at once. On the night of the final, the payments app's commander would order the rollback of v42 first and ask why later. The tools for that night must not depend on production:

Within a few days the team writes a blameless postmortem, which asks why the system let one slip become an outage, not who to punish. Each action item changes the system and has an owner and a date: "the removal tool refuses to go below minimum capacity", not "be more careful". Then practise: restore a backup every week, fail over a region every quarter, and switch off a dependency on purpose. These drills, often called game days, are how the social network had rehearsed coming back in safe steps.

Outage questions rarely come by name; they arrive as "how would you deploy this?" or "what if the database slows down?", and these lines answer them with numbers.