Scaling and Health

7 pages · ~5 min✓ Reviewed

Builds on Storage.

KubernetesPage 1

Probes and Resources: The Mental Model

1. Who Decides What

Probes: kubelet health checks on a container
Requests and limits: scheduler and kernel contract
HPA: changes the replica count
Together: pod is placed, kept alive, sent traffic

2. Probes

Probe	On failure
Liveness	kubelet restarts the container
Readiness	removed from Service endpoints, not restarted
Startup	holds off liveness and readiness until booted

httpGet: 2xx/3xx passes
tcpSocket: port opens
exec: exit code 0
grpc: gRPC health check
Each probe runs on its own periodSeconds timer

3. Requests vs Limits

	Effect
Requests	scheduler reserves them when placing the pod
Limits	enforced at runtime: CPU throttled, memory OOMKilled

4. Interview Point

Liveness vs readiness? Liveness restarts the container; readiness only stops traffic
Slow-starting app? A startup probe
Readiness fails? Pod leaves Service endpoints: no traffic, no restart

KubernetesPage 2

Configuring Probes Correctly

1. Key Fields

Field	Default	Meaning
`initialDelaySeconds`	0	wait before first probe
`periodSeconds`	10	gap between probes
`timeoutSeconds`	1	max wait per probe
`successThreshold`	1	passes to recover
`failureThreshold`	3	fails before action

Startup budget = failureThreshold x periodSeconds, e.g. 30 x 10 = 300s to boot

2. Which Probe Asks What

Probe	Asks	Gates endpoints?
Liveness	Should I restart?	no
Readiness	Should I get traffic?	yes
Startup	Has it finished booting?	no

yaml

readinessProbe:
  httpGet:
    path: /healthz
    port: 8080
  periodSeconds: 5
  failureThreshold: 3

3. Mistakes

Liveness checks a database or dependency: cascading restarts
Short timeoutSeconds on a slow app: restart loop
Big initialDelaySeconds: use a startup probe instead

4. Interview Point

Liveness and dependencies? Never: one outage restarts every pod
Size a startup probe? failureThreshold x periodSeconds above worst boot time
Default failureThreshold? 3

KubernetesPage 3

Requests, Limits and QoS Classes

1. CPU vs Memory

	CPU	Memory
Unit	cores / millicores (`500m` = 0.5)	`Mi` / `Gi`
Type	compressible	incompressible
Over limit	throttled	OOMKilled, exit 137

yaml

resources:
  requests:
    cpu: 250m
    memory: 256Mi
  limits:
    cpu: 500m
    memory: 512Mi

requests reserve, limits enforce

2. QoS and Scheduling

Class	Rule	Evicted
Guaranteed	requests = limits, every container, CPU + memory	last
Burstable	at least one request or limit set	second
BestEffort	none set	first

Scheduler counts requests only
Pending: no node with enough unreserved allocatable

3. Common Mistakes

No requests: HPA cannot compute CPU utilization
CPU limit too low: silent latency from throttling
Memory limit below working set: repeated OOMKills

KubernetesPage 4

Horizontal Pod Autoscaler

1. The Formula

desired = ceil(current × currentMetric / target)
Checks about every 15s; needs metrics-server
CPU % is measured against requests, not limits
No CPU request → metric shows <unknown>, no scaling

2. Create and Tune

bash

kubectl autoscale deployment web --cpu-percent=70 --min=2 --max=10

or an autoscaling/v2 HorizontalPodAutoscaler

v2 YAML: minReplicas, maxReplicas, metrics list
Scale-down waits a 300s stabilization window (default)
Scale-up reacts faster; tune both in spec.behavior

3. Who Scales What

Tool	Changes	Trigger
HPA	pod count	metric vs target
VPA	pod requests and limits	observed usage
Cluster Autoscaler	node count	Pending pods

KubernetesPage 5

Troubleshooting CrashLoopBackOff, Pending and ImagePullBackOff

1. Reading the Status

Status	What happens	First check
CrashLoopBackOff	Starts, exits, restarts: 10s, 20s, 40s, max 5m	`logs --previous`
Pending	Not scheduled	`describe pod` Events
ImagePullBackOff	Pull fails, retried with backoff	Image reference

Exit 1 app error, 137 OOMKilled/SIGKILL, 143 SIGTERM
Confirm OOM: describe pod Last State shows Reason: OOMKilled
Also check whether the liveness probe kills it
Pending: Insufficient cpu, untolerated taints, selector mismatch, unbound PVC
Pull: wrong name or tag, no imagePullSecrets, rate limits

2. Triage Order

get podread status
describe podEvents at bottom
logs--previous if crashed
get events--sort-by=.lastTimestamp

3. Mistakes

Reading current logs of a restarting pod, not --previous
Guessing resources instead of kubectl top pod
Readiness failure shows 0/1 Ready with no restarts

4. Interview Point

Pending first? describe pod Events
OOMKilled or app crash? Last State Reason OOMKilled; exit 1 is the app
Why back off? Growing restart delay stops a tight crash loop wasting node CPU; pull retries spare the registry

KubernetesPage 6

Probes, Resources, HPA and Troubleshooting Cheat Sheet

1. Probes and Resources

Liveness restarts the container
Readiness gates Service traffic
Startup protects slow boots: budget = failureThreshold x periodSeconds
Scheduler uses requests; kernel enforces limits
CPU over limit: throttled. Memory over limit: OOMKilled (137)

QoS	Rule
Guaranteed	requests = limits
Burstable	some request or limit set
BestEffort	none set, evicted first

2. HPA

ceil(current x currentMetric / target)
Needs metrics-server and CPU requests
Scale-down window: 300s by default

bash

kubectl autoscale deployment web --cpu-percent=70 --min=2 --max=10

3. Debug and Status Causes

bash

kubectl describe pod X
kubectl logs X --previous
kubectl get events --sort-by=.lastTimestamp
kubectl top pod

Status	Cause
CrashLoopBackOff	app or liveness probe kills container
Pending	no schedulable node or unbound PVC
ImagePullBackOff	bad image name or missing pull secret

Interview Point

Not serving traffic? describe, then logs --previous, then events. Pending = scheduling; CrashLoopBackOff, ImagePullBackOff = container.

KubernetesPage 7

Check yourself

A pod shows 0/1 Ready, status Running, RESTARTS 0. Which probe is failing, and where is its traffic going?

The readiness probe is failing.
The pod leaves the Service endpoints. It is not restarted, so traffic goes to the other pods.

A startup probe has failureThreshold 30 and periodSeconds 10. The app needs 400s to boot. What happens?

The budget is 30 x 10 = 300s, so the probe fails first.
kubelet restarts the container, and it loops. Raise failureThreshold to 40 or more.

An HPA targets 60% CPU. 4 replicas run at 90% of their requests. How many replicas does it want?

ceil(4 x 90 / 60) = ceil(6) = 6 replicas.