Handbooks / System Design / Chapter 13
Observability and Operations
7 pages · ~5 min✓ Reviewed
Builds on IDs, Time and Ordering.
Logs, Metrics and Traces
1. The Three Signals
- Logs: timestamped discrete events
- Metrics: numeric time series, aggregated cheaply
- Traces: one request across services, as spans
2. How a Trace Is Built
- Edgemakes trace ID
- Propagate
traceparentheader (W3C) - Each spanparent ID, duration, status
3. Compare and Combine
| Metrics | Traces | Logs | |
|---|---|---|---|
SLIs, SLOs, SLAs and Error Budgets
1. SLI, SLO, SLA
| SLI | SLO | SLA | |
|---|---|---|---|
| Is | Measured ratio | Internal target | Contract with penalties |
| Answers | What you measure | What you aim for | What you promise and pay for |
| Looks like | good / valid events | 99.9% over 30 days | Looser than the SLO |
Alerting on Symptoms
1. Symptoms, Not Causes
- Page on symptoms: error rate, latency, availability
- Not on causes: CPU, memory, a single pod down
- Symptom = a user feels it; cause = maybe nobody does
2. Burn Rate
- Burn rate = current error-budget use ÷ allowed rate
- Burn rate 1 uses exactly the whole budget by window end
- Burn rate 14.4 over 1h eats about 2% of a 30-day budget
Safe Releases: Blue-Green, Canary and Feature Flags
1. Three Strategies
- Blue-green: two full environments, traffic switches at once
- Canary: a small slice goes to the new version first
- Feature flags: deploy is not release, code ships dark
- 1%canary
- 5%compare SLIs
- 25%compare SLIs
- 100%full rollout
2. Rollback and Trade-offs
Autoscaling, Disaster Recovery and Multi-Region
1. Autoscaling
- Horizontal: add instances on a metric target
- Targets: CPU 60%, requests per pod, queue depth
- Cooldown or stabilization window stops flapping
- Set min for HA, max for cost control
bash
kubectl autoscale deployment api --cpu-percent=60 --min=3 --max=20Kubernetes HPA
2. RPO, RTO and Regions
Reliability Cheat Sheet
1. Signals and Budget
- Metrics say what, traces say where, logs say why
- Keep metric labels low-cardinality: no
user_id,request_id - Error budget = 1 - SLO
| SLO | Budget per 30 days |
|---|---|
| 99.9% | about 43 min |
Check yourself
SLO is 99.9% over 30 days. Outages so far: 20 min and 30 min. What is the state?
- Budget is about 43 min; 50 min used, so it is overspent.
- Freeze feature releases and fix reliability.
A team adds user_id as a label on a request counter. What goes wrong?
- Each user makes a new time series, so cost and memory explode.
- Put user_id in logs and traces instead.
A canary looks healthy, but a migration dropped a column. Why does rollback fail?
- The old version still reads that column.