Distributed Logging

It is 8:30 pm on a Friday, the IPL final is halfway through its first innings, and Bengaluru is ordering dinner. A food delivery app starts failing one order in twenty with "payment could not be completed". Each order passes through five of the app's forty services, each running as dozens of copies, and every copy writes its own log file on its own machine. The on-call engineer has three terminals open and is searching files one machine at a time, while autoscaling has already replaced two of the failing machines, taking their logs with them. Central logging turns that hunt into one search.

Logs are timestamped lines describing what a program did. Centralised logging ships every line, soon after it is written, to one place where it is kept for a set time and can be searched in seconds. It is one of the three views from the monitoring chapter: metrics tell you something is wrong, traces show where the time went, and logs record the details of what happened. This chapter builds that pipeline for the food delivery app.

Many people read logs, each for a different reason.

Each reader needs a different slice, from the last ten minutes to last year, and that spread is what makes retention a design question.

Pin down the requirements before drawing any boxes, then estimate the volume, because for logging the volume decides most of the design.

• Never block the app: logging must not slow requests, and a log system outage must not take services down.
• Lose little: keep every error line, even while the log store is down.
• Search fast: find all lines for one request or one user, across all services, within seconds.
• Stay fresh: a line written now should be searchable within roughly 10 to 30 seconds.
• Stay affordable: recent logs fast, old logs cheap, and useless logs deleted.

The volume follows from the fleet: forty services at about fifty copies each.

That can be more data than the app's own business database. Log volume grows with traffic and with every line a developer adds, so cost and retention are design questions from day one, not afterthoughts.

Volume is one problem; finding the right line is the other. A plain text line is easy to write and hard to search. A structured log is a line made of named fields, usually one JSON object per line, so the store can filter on level or duration_ms exactly instead of guessing with text patterns.

Plain text:
2026-10-16 20:31:07 ERROR payment failed for order 88231 after 3021ms

Structured, one JSON object per line:
{"ts":"2026-10-16T15:01:07.412Z", "level":"ERROR", "service":"payments",
 "version":"4.12.0", "host":"pay-7f9c", "request_id":"r-5f2a9c",
 "order_id":88231, "event":"charge_failed", "reason":"gateway_timeout",
 "duration_ms":3021}

The same event. The structured line stores time in UTC (20:31 IST is 15:01 UTC) and can be filtered on any field.

A few fields earn their place in every line:

Of those fields, one ties the services together. A request id (also called a correlation id) is a unique id created when a request first enters the system, usually at the gateway. Every service writes it into each log line and passes it on in a header, such as X-Request-Id or the standard traceparent header used by tracing tools.

Searching request_id = r-5f2a9c now returns the whole story as one list. The id must also travel through queues and background jobs: put it inside each message and have the consumer log it. Otherwise the trail stops at the first queue.

Not every line deserves to be stored, let alone stored for months. Log levels mark how important a line is, and each service decides which levels it writes.

Levels decide what is written; sampling decides how much of it is kept. Sampling keeps only a fraction of routine lines: keeping 5% of successful requests cuts their volume twentyfold while still showing what normal looks like. Decide per request, not per line, so a kept request keeps all its lines and you never read half a story.

• Head sampling decides when the request starts. It is simple and cheap, but it may drop the very request that later failed.
• Tail sampling holds a request's lines briefly and decides at the end: keep every failed or slow request, sample the rest. Better results, more memory.
• Collapse repeats: a bug that logs the same error 10,000 times a second should produce one line with a count.
• Change levels at runtime: switching DEBUG on for one service for 15 minutes beats a redeploy.
• Never sample audit events: records of who logged in or changed a payout account go to a separate stream, covered near the end of this chapter.

With the lines worth keeping defined, the next question is how they leave the machine. Services should not send logs over the network themselves. They write to standard output or a local file, which is fast and never waits on the network. A small collection agent on each machine, or on each node of a container cluster, does the rest.

When the network or the queue is down, the agent keeps reading into a buffer on local disk with a fixed size limit. If the buffer fills, it must drop something, and the sensible order is DEBUG first, then INFO, and errors last. Shipping is at least once: after a crash the agent resends from its last checkpoint, so a few lines may arrive twice, which is usually harmless for logs.

Agents could write straight into the log store, but large pipelines put a queue, a replicated log as in the message queue chapter, in the middle. Watch one failed order's lines travel from two machines to the on-call engineer's screen, then watch an incident push the store behind.

The request id is what turns files on separate machines into one story. The queue is what lets the store fall behind without losing lines: the queue turns a store outage into a delay instead of data loss. The next page sizes it.

The scene showed what the queue absorbs. It earns its place in four ways:

• It absorbs bursts: an incident can multiply log volume five times over, exactly when you need the logs most.
• It decouples: the store can be upgraded, slowed or restarted while agents keep shipping.
• It feeds many readers: the search index, a cheap archive in a blob store and security tools each read the same stream at their own pace.
• It allows replay: if a parser bug mangled an hour of lines, fix the parser and read that hour again.

Now the numbers. One partition of a log-based queue handles about 10 MB/s (the message queue chapter). At the app's 50 MB/s, that is at least 5 partitions on an ordinary evening, and a storm at 250 MB/s needs 25, so plan for about 32. Spread lines across partitions by host rather than by service, or the busiest service makes one partition hot. Three days of retention at 50 MB/s is 50 MB × 259,200 s ≈ 13 TB per copy, about 39 TB with three replicas. Sizing on raw bytes leaves headroom, since agents compress batches 5 to 10 times.

The advice flips for small systems. With a few dozen machines, agents with disk buffers can write straight to the store, and a queue would be one more system to run for little gain. It pays off once storms, several readers or replays matter.

Lines leave the queue for a store that must find them fast. Logs are written once, almost never changed, and nearly always searched by time first: "errors from payments in the last 15 minutes". Log stores exploit this with time-based partitions, one per hour or per day. A query opens only the partitions in its time range, and deleting old data means dropping whole partitions. Two indexing styles compete:

Indexing every word is the inverted index from the distributed search chapter, applied to log lines. Indexing only labels and scanning the rest sounds slow, but the haystack is small: one service's lines for 15 minutes come to about 1 GB before compression (50 copies × 50 lines a second × 500 B × 900 s), which a few machines can scan in seconds. Many teams mix both: full indexing for a few days of hot data, labels only for older data.

Most log searches look at the last few hours. Hardly anyone reads a line from eight months ago unless a security team or an auditor asks. So logs move through tiers, from fast and expensive to slow and cheap, and are deleted at the end.

The hot tier is where the money goes. Seven days at 4.3 TB a day is about 30 TB of raw lines. With every field indexed, assume the stored lines plus the index take about twice that, plus one replica so a dead node loses nothing: 30 × 2 × 2 ≈ 120 TB of SSD. At an assumed $0.10 per GB-month for network SSD (check current prices), that is about $12,000 a month for the disks alone, before the machines.

Logs compress well, often 5 to 10 times, because lines repeat the same field names and values. Indexing labels only and keeping lines compressed 10× shrinks that week to about 6 TB, about $600 a month. A year of cold archives at 10× is about 160 TB, roughly $3,200 a month at about $0.02 per GB-month for object storage.

Retention differs by kind: debug lines for days, request logs for weeks, audit trails for years. The law can set a floor: in India, CERT-In's 2022 directions ask many organisations to keep logs of their systems for a rolling 180 days, stored in India, so check what applies to yours.

Long retention has a dark side. Logs are copied to many machines, kept for months and read by many people, so anything sensitive written into a line leaks to all of them. Treat the log store as a place that holds no secrets and as little personal data as possible.

The safest data is data never written. Four habits make that stick:

• Allow-list, do not block-list: the logging library writes only fields known to be safe, instead of trying to spot every dangerous one.
• Redact in the pipeline too: a second check in the agents or indexers masks patterns, such as card numbers, that slipped through.
• Control who can search: restrict access, record every search of the logs, and keep production logs off laptops.
• Remember the law: privacy laws such as India's Digital Personal Data Protection Act, 2023 give people rights over their data, and erasing one person from months of logs is very hard. Not logging it is far easier.

Some lines are evidence, not debugging help: who logged in, who changed a restaurant's payout account, who exported customer data, which admin approved a refund. Security teams and auditors may need them years later, and they must trust that nobody edited them. These audit logs travel in a separate stream with stricter rules.

An action and its audit record succeed or fail together, so no payout change can happen without its record. Keep audit events in write-once storage, such as a blob-store bucket set to refuse deletes until a retention date. Link each batch to the one before it with a hash, so a missing or edited batch shows up, and keep the latest hash where the log writers cannot change it, such as a separate account or a published daily digest: anyone who can rewrite the whole chain can also recompute every hash. An audit line still records the user id and the action, never the card number.

Logging pipelines fail in predictable ways, and most failures peak during incidents, when the logs matter most.

Almost every fix either sheds low-value lines first or buys time with buffers. Three numbers catch most of these early: lines received per service, queue lag, and lines dropped by agents. A busy service that suddenly sends zero lines means the pipeline is broken, not the service.

Logging usually comes up inside a bigger design, and a few sentences show that you know the whole path from the line in the code to the search box.