Observability in Production

7 pages · ~5 min✓ Reviewed

FastAPIPage 1

Production Observability Mental Model

1. Three signals, one ID

Signal	Answers	Shape
Logs	What happened?	discrete events
Metrics	How much, how often?	aggregated numbers over time
Traces	Where did the time go?	one request across services

Request ID is the glue
One ID on every log line and span
Follow one request across all signals

python

import logging, uuid
from fastapi import FastAPI
app = FastAPI(); log = logging.getLogger("app")
@app.middleware("http")
async def stamp(request, call_next):
    rid = request.headers.get("x-request-id") or str(uuid.uuid4())
    log.info("request_id=%s", rid)
    return await call_next(request)

2. Probes and lifecycle

Probe	Asks	On failure
`/healthz` liveness	Process alive?	Restart
`/readyz` readiness	Can it take traffic now?	Removed from load balancer

Startuplifespan start
Ready/readyz 200
Serving
TerminatingpreStop sleep
SIGTERM/readyz 503
Drainin-flight requests
Exitlifespan shutdown

3. Where it lives

Middleware wraps every request
Lifespan hooks run at startup and shutdown
Applies to every route, handlers untouched

Remember

Timeouts bound every step of the pipeline.

FastAPIPage 2

Uvicorn Logging and Request IDs

1. Loggers and Config

Logger	Carries
`uvicorn`	the server
`uvicorn.error`	lifecycle and errors (despite the name)
`uvicorn.access`	one line per request

--log-config logging.yaml or dictConfig
disable_existing_loggers: False, or app loggers go silent
--log-level debug | info | warning | error
--no-access-log cuts noise; access log is on by default
Format as JSON (python-json-logger, structlog) for aggregators

2. Request ID

Header or uuid4X-Request-ID
ContextVarset per request
Filteradds request_id
Response headerechoed back

FastAPIPage 3

Health, Readiness and Prometheus Metrics

1. Liveness vs Readiness

Probe	Path	Checks	On failure
Liveness	`GET /healthz`	none, always 200	restart pod
Readiness	`GET /readyz`	DB, cache	503, leaves load balancer

DB check in liveness: a DB outage restarts every healthy pod in a loop

2. Readiness at Shutdown

Pod Terminatingendpoint removal starts
preStop sleep
SIGTERM
/readyz 503Uvicorn stops accepting
Drainin-flight requests
Lifespan shutdown

3. Prometheus Metrics

FastAPIPage 4

OpenTelemetry Tracing, Timeouts and Graceful Shutdown

1. Traces and Logs

Trace = tree of spans sharing one trace_id
W3C traceparent header carries it between services
Auto spans: add httpx and SQLAlchemy instrumentors
Log trace_id next to request_id: log line → trace
Custom span: tracer.start_as_current_span('name')

python

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
provider = TracerProvider(); provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)
FastAPIInstrumentor.instrument_app(app)

provider + batch processor + OTLP exporter, then instrument

python

import logging
from opentelemetry import trace
tracer = trace.get_tracer(__name__)
class TraceFilter(logging.Filter):
    def filter(self, record):
        ctx = trace.get_current_span().get_span_context()
        record.trace_id = format(ctx.trace_id, '032x')
        return True

add it to your handler like the request_id filter

2. Timeouts

FastAPIPage 5

Common Mistakes in Production Observability

1. Logging and Request IDs

print() or basicConfig after Uvicorn starts: handlers duplicated or ignored
Use dictConfig, set disable_existing_loggers False, set propagate on purpose
Request ID in a plain global: IDs leak between concurrent requests

python

import contextvars
request_id = contextvars.ContextVar("request_id", default="-")
request_id.set(rid)  # in middleware

a ContextVar is isolated per request

2. Probes and Shutdown

Liveness checks the DB: outage causes a restart storm
Readiness never fails at shutdown: dropped requests

Terminatingendpoint removal starts
preStop sleep
SIGTERMstop accepting, /readyz 503
Drainin-flight requests
Lifespan shutdown

3. Cardinality and Timeouts

FastAPIPage 6

Production Observability Cheat Sheet

1. Run and probes

bash

uvicorn app:app --log-config logging.yaml \
  --timeout-keep-alive 5 \
  --timeout-graceful-shutdown 25

log config, idle cap, drain cap

Probe	Means	Fails when
`/healthz`	alive, no deps	process dead
`/readyz`	can take traffic	503 on dependency failure or shutdown

2. Metrics and tracing

Type	Query
Counter	`rate()`
Histogram	`histogram_quantile(0.95, ...)`

Metric labels = route template only (/users/{id})

FastAPIPage 7

Check yourself

Your /healthz liveness probe runs SELECT 1 on the DB. The DB goes down for 2 minutes. What happens to the pods?

Probes fail on every pod, so Kubernetes restarts all of them.
Restarts don't fix the DB. Use /readyz for dependency checks.

A middleware stores the request ID in a module-level variable. Requests A and B overlap. What can A's later log lines show?

B's ID. B overwrote the shared global while A was awaiting.
Fix: use a ContextVar, which is isolated per request.

A Prometheus counter is labeled with the raw path /users/8841. What goes wrong, and what is the fix?

Each user ID creates a new time series, so memory and cost blow up.
Label by the route template /users/{id}.

Type	Behaviour	Query
Counter	only goes up	`rate()`
Gauge	up and down	read directly
Histogram	buckets	`histogram_quantile` for p95/p99

Idle connection	`--timeout-keep-alive` (default 5s)
Outbound call	`httpx.Timeout(5.0)`
Slow await	`asyncio.timeout(2)` or `wait_for`

Mistake	Fix
Labels: user ID, raw URL	Route template only
Unbounded span attributes	Bounded, short values
No outbound timeout	`httpx.Timeout`, `asyncio.timeout()`

Observability in Production

Production Observability Mental Model

1. Three signals, one ID

2. Probes and lifecycle

3. Where it lives

Uvicorn Logging and Request IDs

1. Loggers and Config

2. Request ID

Health, Readiness and Prometheus Metrics

1. Liveness vs Readiness

2. Readiness at Shutdown

3. Prometheus Metrics

OpenTelemetry Tracing, Timeouts and Graceful Shutdown

1. Traces and Logs

2. Timeouts

Common Mistakes in Production Observability

1. Logging and Request IDs

2. Probes and Shutdown

3. Cardinality and Timeouts

Production Observability Cheat Sheet

1. Run and probes

2. Metrics and tracing

Check yourself

3. Graceful Shutdown

3. Kubernetes shutdown order