Handbooks / CI/CD & DevOps / Chapter 5
Monitoring and Rollback
7 pages · ~5 min✓ Reviewed
Builds on Deployment Strategies.
Logs, Metrics and Traces: The Mental Model
1. Observability vs Monitoring
- Observability: explain unknown failures from outside, using emitted telemetry
- Monitoring: checks only known failure conditions
2. The Three Signals
| Metrics | Traces | Logs | |
|---|---|---|---|
| Is | Numeric time series | Spans sharing one trace ID | Timestamped event records |
| Strength | Cheap, aggregatable | Follows one request across services | Detail on one event |
| Answers | THAT it is wrong | WHERE | WHY |
| Cost | Cheapest | – | Highest at volume |
- Metric alertTHAT
- TraceWHERE
- Log lineWHY
3. Cardinality Mistake
- User ID or request ID as a metric label
- Every value makes a new series: series count and cost explode
- Put them in logs and traces instead
Remember
Metrics detect, traces locate, logs explain. Keep high-cardinality values out of labels.
The Four Golden Signals
1. The Four Signals
| Signal | Measures | Example |
|---|---|---|
| Latency | time to serve a request | p99 of /checkout |
| Traffic | demand | requests per second |
| Errors | rate of failed requests | 5xx, wrong content, too slow |
| Saturation | how full the tightest resource is | CPU, memory, disk, pool, queue depth |
- Minimum dashboard for any user-facing service
- Track latency for successes and failures separately
- A fast error response hides real slowness
2. RED vs USE
| Method | Letters | Suits |
|---|---|---|
| RED | rate, errors, duration | request-driven services |
| USE | utilization, saturation, errors | resources: hosts, disks |
3. Average Latency Trap
- Alerting on the mean: averages hide the slow tail
- Alert on p95 and p99 instead
Mean 120ms, p99 3s
Still means 1 in 100 users waits 3s.
SLIs, SLOs and Error Budgets
1. SLI, SLO, SLA
| Term | Is | Example |
|---|---|---|
| SLI | measured good / valid events | successful requests ÷ total requests |
| SLO | target for the SLI over a window | 99.9% over 30 days |
| SLA | contract with consequences | penalty if missed |
2. Error Budget and Burn Rate
- Error budget = 100% − SLO
| SLO | Downtime allowed per 30 days |
|---|---|
| 99.9% | about 43 min |
| 99.99% | about 4.3 min |
- Burn rate = how fast the budget is spent
- Burn rate 1: budget ends exactly at the window end
- Burn rate 10: budget empty in a tenth of the window
3. Policy and Mistakes
- Budget healthy: ship features faster
- Budget spent: freeze risky releases, fix reliability
- Internal SLO tighter than the SLA
Never aim for 100%
Unachievable, and each extra nine costs far more. Pick SLIs users feel (errors, latency), not server CPU.
Alerting That Pages the Right Human
1. Page on Symptoms
- Alert on symptoms users feel
- Error rate, latency, SLO burn
- Not causes: CPU at 80%
- Page only if a human must act now
- Rest: ticket or dashboard
2. Burn Rate vs Threshold
| Threshold | Burn rate | |
|---|---|---|
| Tied to SLO | No | Yes |
| Brief spike | Noisy | Tolerated |
- Page: 1h and 5m both above ~14x (~2% budget in 1h)
- Ticket: ~1x slow burn over 3 days
- Short 5m window: resets fast after recovery
for:stops a single blip firing
yaml
alert: HighErrorRate
expr: sum(rate(http_requests_total{code=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.01
for: 5m
annotations:
runbook: https://wiki.example.com/runbooks/high-errors3. Alert Fatigue
Noisy pages
Non-actionable pages get ignored. Every alert needs an owner and a runbook. Delete or demote what nobody acts on.
Rollback vs Roll Forward and Incident Basics
1. Rollback or Roll Forward
| Rollback | Roll forward | |
|---|---|---|
| Does | Redeploy last good version | Ship a fix on top |
| Default when | Recent deploy caused it | Rollback is unsafe |
- Roll forward: irreversible migration, or data in a new format
- Roll forward: one-line fix faster than a redeploy
- Expand/contract migrations and flags keep rollback possible
2. Incident Response
| Role | Job |
|---|---|
| Commander | Coordinates, decides |
| Ops lead | Fixes |
| Comms lead | Updates stakeholders |
| Declare | SEV1 = full outage; one channel, one timeline |
Commands: undo rollout, check status, recent logs, revert commit.
bash
kubectl rollout undo deployment/api
kubectl rollout status deployment/api
kubectl logs deploy/api --since=15m
git revert <sha>3. Postmortem and Mistakes
- Blameless: timeline, impact, root cause, contributing factors, what went well
- Action items need owners and dates; judge the system, not the person
- Track MTTD detect, MTTA acknowledge, MTTR recover
Mitigate first
Stop impact by rollback, failover or flag before debugging live. An untested rollback, or one skipping the schema change, fails.
Cheat Sheet: Observability, SLOs and Incidents
1. Signals and SLO math
| Pillar | Tells you |
|---|---|
| Metrics | THAT it broke |
| Traces | WHERE |
| Logs | WHY |
- No user or request IDs in metric labels (cardinality)
- Golden signals: latency, traffic, errors, saturation
- Use p95/p99, not the mean
- Budget = 100% - SLO: 99.9% over 30d is about 43 min, 99.99% about 4.3 min
- Burn rate 1 spends the whole budget over the window
- SLA is looser than SLO
2. Alerts and rollback
- Page on symptoms and SLO burn only
- 1h and 5m windows both above 14x, plus
for: - Every page needs a runbook and an owner
bash
kubectl rollout undo deployment/api
rollback by default
- Roll forward only if rollback is unsafe
- Expand/contract migrations and feature flags keep rollback open
3. Incident flow
- Severitydeclare
- Commanderassign
- Mitigatefirst
- Communicate
- Postmortemblameless; MTTD, MTTA, MTTR
- Outage walk-through? Alert, severity, commander, mitigate, communicate, postmortem
- New service SLO? Pick a user-facing SLI, set the target from baseline, review
- Dashboard? Four golden signals plus SLO burn
Check yourself
A dashboard adds a user_id label to a request counter. You have 1 million users. What breaks?
- Each label value makes its own time series: about 1M series per metric.
- Storage and query cost explode. Put user_id in logs or traces.
The SLO is 99.9% over 30 days and you have already had 30 minutes of downtime. Budget left?
- Budget is about 43 min, so roughly 13 min remain.
- Another outage that long would exhaust it and trigger a release freeze.
A bad deploy shipped with a migration that dropped a column. Roll back or roll forward?
- Roll forward: the old version needs the dropped column, so rollback is unsafe.
- Mitigate first, for example by toggling a flag, then ship the fix.