Handbooks / Microservices / Chapter 8
Observability and Deployment
7 pages · ~5 min✓ Reviewed
Builds on Configuration and Feature Flags. Next up: Testing Microservices.
Observability Mental Model
1. Three Pillars
| Pillar | What it is | Answers |
|---|---|---|
| Logs | discrete events | What happened? |
| Metrics | aggregated numbers over time | How much, how often? |
| Traces | one request's path across services | Where did time go? |
- Monitoring: known questions, via dashboards and alerts
- Observability: ask new questions about unknown failures
- It needs rich telemetry: all three pillars together
2. Logs and Metrics
- Emit JSON with fixed keys, not free text
- Keys:
timestamp,level,service,message,trace_id - Result: logs are queryable and filterable
json
{"timestamp":"2026-03-01T10:15:02Z","level":"error","service":"orders","message":"payment timeout","trace_id":"4bf92f35"}| Type | Behaviour | Use |
|---|---|---|
| Counter | only goes up | query with rate() |
| Gauge | up and down | queue depth |
| Histogram | bucketed values | percentiles |
3. Choosing Signals
- RED for services: Rate, Errors, Duration
- USE for resources: Utilization, Saturation, Errors
Tracing, Correlation IDs and Health Checks
1. Correlation ID and Traces
- Correlation ID: unique ID made at the edge, or reused from the incoming header
- Passed on every downstream call and log line to stitch one request together
- A trace is a tree of spans
- Each span:
trace_id,span_id,parent_span_id, start time, duration
2. Propagation and Log Link
- Context travels in the W3C traceparent header
- Format:
version-trace_id-parent_id-flags - OpenTelemetry = standard SDK and export format
- Put
trace_idin every log line: jump from a log to its full trace
text
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
json
{"level":"error","service":"orders",
"trace_id":"4bf92f35","msg":"payment timeout"}3. Health Probes
| Probe | On failure |
|---|---|
| Liveness | container is restarted |
| Readiness |
Containers and Kubernetes Core
1. Containers and Objects
- Container = isolated process sharing the host kernel
- Namespaces = isolation; cgroups = resource limits
- Image = immutable layered filesystem; container = its running instance
| Object | Role |
|---|---|
| Pod | smallest unit, one or more containers |
| Deployment | manages ReplicaSets and rolling updates |
| Service | stable virtual IP and DNS for pods |
| Ingress | HTTP routing in |
| ConfigMap / Secret | config |
2. Control Loop and Resources
- Declaredesired state
- Reconcilecontrollers fix drift
- Schedulenodes fit requests
| Requests | Limits |
|---|
Release Strategies: Rolling, Blue-Green, Canary
1. Blue-Green and Canary
- Blue-green: two identical environments, deploy to the idle one
- Test it, then flip the switch: Service
selectoror load balancer - Rollback = flip back; cost = double capacity
- Canary: send 1-5% of traffic to the new version
- Watch error rate and latency
- Step up to 100%, or abort
2. Which One?
| Cost | Rollback | Exposure | |
|---|---|---|---|
| Rolling | cheapest | slow | mixed versions |
| Blue-green | 2x capacity | instant flip | all-or-nothing |
| Canary | small extra | abort the slice | smallest blast radius, needs metrics |
Service Mesh Basics and Everyday kubectl
1. Service Mesh
- Control planeIstio, Linkerd
- Envoy sidecarbeside each pod
- Service trafficproxy to proxy
- mTLS between services
- Retries and timeouts
- Traffic splitting: weighted routing for canaries
- Uniform metrics and traces, no app code changes
Trade-off
Extra latency, per-pod cost, complexity. Skip it for a handful of services.
2. Everyday kubectl
| Command | Use |
|---|---|
get pods -o wide | node and IP |
describe pod <p> | read Events |
logs <p> -f --previous | follow; last crash |
exec -it <p> -- sh | shell inside |
apply -f file.yaml | apply manifest |
rollout status/undo deployment/<d> | watch or revert |
Production Observability Cheat Sheet
1. Signals
| Signal | Answers | Method |
|---|---|---|
| Logs | what happened | - |
| Metrics | how much, how often | RED services, USE resources |
| Traces | where time went | - |
2. Correlation
- traceparentheader in
- trace_idevery log line
- Message headerscopy on async hops
3. Probes and Releases
- Liveness fails: container restarts
- Readiness fails: removed from traffic
- Startup: gates the other two until boot ends
| Release | Trade-off |
|---|---|
| Rolling |
Check yourself
A liveness probe calls /health, which pings the database. The database goes down for a minute. What happens to the pods?
- Every pod fails liveness at once, so Kubernetes restarts them all: a restart storm.
- Restarting fixes nothing. Check dependencies in readiness, and keep liveness to the process itself.
Service A calls B over HTTP with a traceparent header. B publishes to a queue. The consumer's logs show a new trace_id. Why?
- The context was dropped at the async hop.
- Copy the trace context into the message headers, and have the consumer read it back.
You deploy blue-green, and the migration renames column name to full_name. Green errors, so you flip back to blue. What happens?
- Blue still queries
name, so it fails too. The flip is not a safe rollback. - Use expand-and-contract: add the new column first, and drop the old one only after the cutover.