Resilience Patterns

• Resilience pattern = rule limiting how far one bad dependency hurts its callers
• Slow is worse than down: callers hold threads, sockets, memory while waiting

• In flight = arrival rate x latency
• 100 req/s: 50 ms is 5 in flight, 5 s is 500
• 100x more threads needed, so the pool is exhausted

• Every network call needs an explicit timeout
• Connect timeout: short, ~1 s
• Read timeout: dependency p99 x 1.5
• Default of infinite or 30-60 s = classic outage cause
• Pass one deadline down the chain: gRPC deadline, X-Request-Deadline
• Downstream calls never outlive a caller who gave up

delay = min(10, 0.1 * 2 ** attempt)
time.sleep(random.uniform(0, delay))

full jitter

• Backoff gives the dependency room to recover
• Jitter breaks synchronized retries

• Failure-rate and slow-call thresholds
• Minimum calls before judging the window
• Sliding window: count or time based
• Open wait; permitted trial calls
• One breaker per dependency, not global

• Own bounded pool per dependency: thread pool or semaphore
• e.g. 10 concurrent calls to payments API
• Slow dependency exhausts only its own compartment (ship hull)
• Pool full: reject immediately, never queue unboundedly

• Caps requests per client per window
• Over limit: 429 Too Many Requests + Retry-After
• Rate limit protects the server; breaker protects the client

• Shared state: Redis INCR + EXPIRE, or atomic Lua script
• Key by user, API key or IP: choose deliberately
• Per tenant keys stop noisy neighbours

• Fallback = degraded but useful answer when call fails or breaker is open
• Forms: cache, default, static content, simpler algorithm, queue for later

• Cheap, local, tested
• Never call another fragile dependency
• Emit metric and alert whenever it fires

• Timeout on every call
• Capped retries, backoff, jitter
• Breaker + bulkhead per dependency
• Rate limit and load shedding at edge
• Fallback for non-critical features

• Connect timeout ~1 s; read timeout ~p99 x 1.5
• Downstream timeout shorter than upstream
• Propagate one deadline down the chain

import random

def delay(n, base=0.1, cap=10):
    return random.uniform(0, min(cap, base * 2**n))

full jitter: random(0, min(cap, base x 2^n))

• Max 3 attempts, idempotent calls only
• Retry at one layer; budget ~10% of requests