Monitoring: Metrics, Logs, Traces and Alerts

10:00 am, board results day. Two million students (20 lakh) open the results site at once. Pages load, the servers look busy but fine, and the team relaxes. Meanwhile one of thirty servers has a full disk, and every request it serves fails: about 3% of students get a blank screen. Nobody on the team knows. The first sign is an angry post at 10:40, shared ten thousand times. With monitoring, a graph turns red at 10:01, an alert pages the on-call engineer at 10:05, and the bad server is out of rotation before most students notice. This chapter builds that second morning.

Monitoring answers three questions, in this order:

• Is it working for users right now? Not "are the servers up", but "can students see their results, quickly".
• If not, where is it broken? Which service, which server, which dependency.
• What changed? A deploy, a traffic spike, a full disk, a slow database.

The first question is the hardest to answer well. Which numbers tell you users are fine?

If you can watch only four numbers for a service, watch these. Together they catch most of the problems users feel, and the table shows each one on the results site.

These are the ideas from Availability, Reliability, Scalability and Latency, turned into things you can graph. Saturation is the early warning, because systems usually slow down before they fail.

Two shorter checklists cover the same ground. RED (rate, errors, duration) is for anything that serves requests, such as an API or a queue consumer. USE (utilisation, saturation, errors) is for resources: CPU, disks, network links, connection pools. Apply RED to every service and USE to every machine and pool it depends on.

The golden signals are numbers, but numbers alone cannot say why one student’s lookup failed. Systems emit three kinds of data about themselves, together called telemetry, and each answers a different question.

A good setup uses all three, because each is cheap where the others are expensive. Here is one failed lookup seen three ways:

metric  http_requests_total{service="results", status="500"}  1423
log     {"ts":"2026-10-16T10:00:03Z", "level":"error", "request_id":"r-81f2", "msg":"marksheet timeout"}
trace   r-81f2: gateway 3,120 ms > results 3,090 ms > marksheet 3,010 ms > document store 3,000 ms (timeout)

A counter that went up, a log line with details, and a trace showing where the time went.

The trace line above packs a whole story into one row; this page unpacks it. A results page can involve several calls. A trace records each call as a span: which service, when it started, how long it took, and which span called it. All spans of one request share a trace id, which every service passes along in a request header; the standard header is called traceparent.

You rarely write this by hand. OpenTelemetry, an open standard with SDKs for most languages and a collector, creates the spans, passes the header and also emits metrics and logs, so you can change monitoring vendors without changing your code. The failed lookup’s spans look like this:

The trace answers in seconds what logs alone would take an hour to piece together: the document store timed out after 3 seconds, and every span above it was waiting. It even suggests a fix, because a 3-second timeout on a call the student is waiting for is too long.

Tracing every request is expensive, so systems sample: keep perhaps 1% of normal requests, but keep every trace that ended in an error or ran slowly. Deciding after a request finishes is called tail-based sampling; it keeps the interesting traces and drops the dull ones, at the cost of buffering every trace for a few seconds until it ends.

Traces and metrics are useless while they sit on thousands of servers; they must travel to one place. Usually a small agent runs on every server, gathering numbers from the machine (CPU, memory, disk) and from the services on it. There are two ways to move them.

Both work at scale, and many setups mix them: pull inside one network, push from short jobs and from places the server cannot reach. Whichever you choose, decide how a dead server shows up, because a silent server must never look like a healthy one.

Once the numbers arrive, they need a store built for their shape. A time series is a metric name plus a set of labels, holding a list of (timestamp, value) points. Points arrive in time order, are almost never updated, and are read as ranges. Neighbouring points look alike, so they compress very well, often to a couple of bytes a point.

The results site’s fleet shows the scale. 500 servers with 200 series each, sampled every 15 seconds, make 100,000 series and 576 million points a day. At about 2 bytes a point that is roughly 1.2 GB a day, so 15 days of raw points fit in under 20 GB. Part 5 scales the same sum to a whole company.

Raw data is cheap for days, not for years, so older data is kept at lower resolution, a process called downsampling:

Each step down cuts the points twentyfold or more, so two years of history costs less than the raw fortnight.

Stored numbers help only if someone looks at the right moment, and nobody watches graphs all night. An alert is a rule that watches a metric and notifies a person when it crosses a line. The key decision is what to alert on. A symptom is something users feel: errors, slowness, failed lookups. A cause is something that might lead to a symptom: high CPU, one dead server, a disk at 80%.

Each kind gets a different response:

Symptoms page, causes wait. The reason is arithmetic: a fleet has dozens of causes going slightly wrong every day, and paging on each one buries the page that matters. A symptom alert also catches causes nobody thought to watch, such as the full disk on results morning.

The cost is that a symptom alert fires only once users feel something; canaries and error budgets, later in this part and the next, catch trouble while it is small. First, watch one symptom alert end to end.

Here is a symptom alert end to end on results morning. Agents report error rates, a rule watches the overall rate, and only a sustained spike pages the on-call engineer.

The rule needed three bad checks in a row before it paged; real rules usually ask for a few minutes. That short wait filters out one-off blips, so the page that does arrive is worth waking up for. But one bad server in thirty would never have crossed the line at all, which is the problem the next page solves.

In the scene, a person rolled back the hotfix. Most teams automate that step with a canary: the new version goes to a small slice of servers first, and a deploy tool compares that slice with the rest of the fleet before going wider.

The comparison is the point. One bad server in thirty lifts the fleet’s error rate only from 0.2% to about 0.8%, under the 2% page line, but its own 18% stands out at once against its 29 neighbours. A canary turns a fleet-wide blip into an obvious outlier, while only about 3% of students ever see the bad version.

When a page does arrive, the engineer opens a dashboard. It is read by a tired person looking for a problem, often at 3 am, so it should answer "is this service healthy?" in five seconds. Four habits make that possible:

• Golden signals at the top: traffic, errors, latency percentiles and saturation, in that order.
• Dependencies next: the database, cache and services this one calls, with their own errors and latency.
• Deploys and config changes as markers on every graph. The most common answer to "what changed?" is "we did".
• Fewer panels. Eight graphs everyone understands beat forty that nobody does.

On-call means carrying a phone that alerts can ring, usually for a week at a time in a rotation across the team. It works only if pages are rare and meaningful. A team that gets dozens of alerts a day starts ignoring them, and the one real alert gets lost. This is alert fatigue, and every page should pass four tests:

Everything so far measures requests that reach your servers. Many failures happen before that: the page’s script crashes on one phone model, DNS fails in one city, a certificate expires, or a CDN edge misbehaves. Your servers see nothing, so their graphs look perfect while students suffer. Four tools together cover the whole path:

The last row is subtle. If students in one region cannot connect, your error rate does not rise; your traffic falls. A drop of 30% against the same minute last week is often the first sign of a problem outside your servers, such as the resolver outages from DNS: How a Name Becomes an Address or a misbehaving edge from Content Delivery Networks. On results day, compare with last year’s results day, not last week.

Seeing what users see raises the next question: how good is good enough? "As reliable as possible" is not a useful target, because every extra nine costs more and slows feature work. Teams agree on numbers instead, using four terms:

The error budget turns reliability into a shared decision. While budget remains, the team ships features and takes risks. When a bad month uses it up, risky launches pause and the team works on reliability until it recovers.

In time terms, 99.9% over 30 days allows about 43 minutes of complete outage: 0.1% of the month’s 43,200 minutes. But a budget counts requests, not minutes. If most of the month’s lookups arrive on results morning, ten minutes of failures that morning can spend more budget than a quiet week of trouble. That is why the budget, not a clock, is what alerts should watch, as the next page shows.

A fixed line like "errors above 2%" ignores how much budget is left. A burn rate alert asks a better question: how fast are we spending the budget? Burn rate is the current error rate divided by the rate the SLO allows. With a 0.1% budget, 0.1% errors is a burn rate of 1, which uses exactly the whole budget in 30 days, or 720 hours. An error rate of 1.44% is a burn rate of 14.4.

Multiply burn rate by time to get the share of the budget spent. The table shows the usual tiers:

Fast burns page because the budget will be gone in days; slow burns open a ticket because there is time to fix them in working hours.

One window alone misbehaves. A 1-hour window is slow to notice a sudden outage and keeps firing for up to an hour after it ends. So each tier also checks a short confirming window, about one twelfth as long: page only when the burn rate exceeds 14.4 over both the last hour and the last 5 minutes. A real outage trips both quickly, and once errors stop, the 5-minute window drops at once, so the page clears in minutes instead of an hour.

The results site is one of hundreds of services a large company runs, and all of them send metrics to one shared system. Designing that system tests everything in this chapter: the writes never stop, the volume is huge, and it must keep working while everything it watches is failing. It needs four things:

• Ingest metrics from every host and service at 10-second resolution, without ever blocking the senders.
• Answer dashboard queries over the last hour in about a second.
• Fire alerts within a minute and reach the right team once.
• Keep raw points for 15 days and summaries for a year or more.

The estimate that decides the shape assumes 100,000 hosts, each exposing 1,000 series, sampled every 10 seconds.

Ten million writes a second is the hard part, not the bytes. 1.7 TB a day compressed is modest, but 10 million inserts a second and 100 million active series are far more than one storage node can hold. So the design needs a sharded write path with a buffer in front, which the next page lays out.

The estimate says to shard the writes and protect the stores from bursts. Points flow through six stages:

Sizing follows from the estimate. If one storage shard comfortably takes about 500,000 points a second, 10 million needs 20 shards, or 40 to 60 nodes with two or three replicas each.

The log carries points before storage compression. Assume about 20 bytes a point on the wire: an 8-byte value, a timestamp sent as a small delta and a short series reference, with full labels sent once per batch, not per point. That is 200 MB a second, or 20 partitions at about 10 MB a second each, so plan 30 to 40 for headroom.

Why a log in front, instead of writing straight to the shards? A shard that pauses to compact or restart would otherwise push back on 100,000 agents at once. With the log, the shard falls a few seconds behind and catches up, and no agent ever blocks.

The boxes are simple; the hard parts are inside them, and the table names each one.

Which row matters most depends on the pressure. For cost, it is cardinality limits, downsampling and object storage at about $0.02 per GB-month (check current prices). For reliability, it is the log as a buffer, replicated shards and the separate watcher. For speed, it is recording rules and query fan-out.

Monitoring closes almost every design, and "design a metrics system" is also asked on its own; these lines carry the chapter into either.