Handbooks / Generative AI & LLMs / Chapter 9

Model Serving and Inference Optimization

7 pages · ~5 min✓ Reviewed

Builds on Guardrails and Red-teaming. Next up: Multimodal Models.

Generative AI & LLMsPage 1

Prefill vs Decode: The Mental Model

1. Two Phases

	Prefill	Decode
Tokens per pass	whole prompt	one token
Bound by	compute	memory bandwidth
Cache	fills KV cache	reads KV cache + weights
Sets	TTFT	TPOT

Prefill is one parallel forward pass
Decode reads all weights every step
Each token needs the previous: no parallelism in one request

2. The Loop

Queuewait
Prefillprompt
1st token
Decoderepeat

3. Latency Maths

TTFT ≈ queue wait + prefill time
TPOT = one decode step
Long prompt hurts TTFT; long output hurts total

Total latency

TTFT + (N − 1) × TPOT, N = output tokens

Why is decode memory-bound? One token per pass, little compute per weight read
What lowers TTFT? Shorter prompts, less queue wait

Generative AI & LLMsPage 2

The KV Cache

1. Why Cache

Attention needs K and V of every previous token
Cached, so no prefix recompute per decode step

2. Cache Size

Per token = 2 × layers × KV heads × head dim × bytes
Grows linearly with sequence length and batch size

Llama-2-7B, FP16	Value
Layers × KV heads × head dim	32 × 32 × 128
Bytes per value	2
Per token	≈ 0.5 MB
4k-token sequence	≈ 2 GB

3. Shrink It

GQA / MQA: query heads share KV heads, cache several times smaller
Paged attention (vLLM): fixed-size blocks, no fragmentation, prefix sharing

Weights-only sizing

The KV cache, not the weights, often caps batch size and context length.

Generative AI & LLMsPage 3

Batching and Continuous Batching

1. Why Batch

Decode is memory-bound: each step reads all weights
Batching shares one weight read across many requests
Throughput rises almost free

2. Static vs Continuous

	Static	Continuous
Schedules	whole batch	every token step
Finished request	idles behind longest	leaves, slot freed
New request	waits for next batch	joins immediately
GPU slots	wasted	kept full

3. Chunked Prefill and Limits

Chunked prefill: long prompt split into pieces, mixed with decode steps
One big prompt no longer stalls other users' tokens
Bigger batch: more throughput, but higher per-request latency and KV cache use

Generative AI & LLMsPage 4

Quantization: INT8 and INT4

1. What It Buys

Store weights (optionally activations or KV cache) in fewer bits
FP16 → INT8 is 2× smaller; INT4 is 4× smaller
Decode is memory-bound, so fewer bytes per weight gives faster tokens
Smaller weights fit a bigger model on one GPU

2. Which Scheme

Scheme	Quantizes	Speeds up
INT4 weight-only (GPTQ, AWQ)	weights	decode; common serving choice
INT8 W+A (SmoothQuant)	weights + activations	decode and compute-bound prefill

3. Quality and Mistakes

Lossy: INT8 is usually near-lossless
INT4 shows small quality drops
Outliers and small models suffer most

QuantizeINT8 / INT4
Task accuracyvs FP16
Perplexityvs FP16
Ship

Generative AI & LLMsPage 5

Speculative Decoding and Latency vs Throughput

1. Speculative Decoding

1Draft modelsmall, proposes k tokens
2Target modelverifies all k in one pass
3Keep prefixaccepted tokens stay
4Resamplefirst rejected, from target

Lossless: the output distribution equals the target model's
Rejected tokens are resampled from the target, so quality is unchanged
Speedup depends on the acceptance rate

2. When It Pays Off

Situation	Gain
Good draft, low temperature	Large
Poor draft model	Little or negative
High-temperature sampling	Little or negative
Low batch, latency-sensitive	Best fit
Large batch, GPU already busy	Shrinks

3. Latency vs Throughput

Bigger batch: more tokens/s overall

Generative AI & LLMsPage 6

LLM Inference Cheat Sheet

1. Prefill vs Decode

	Prefill	Decode
Work	whole prompt, parallel	one token per pass
Bound by	compute	memory bandwidth
Sets	TTFT	TPOT

Latency = TTFT + (N − 1) × TPOT
KV bytes per token = 2 × layers × KV heads × head dim × bytes

2. Serving Levers

Lever	Rule
Batching	continuous > static; add chunked prefill to protect TPOT; cap batch by KV memory
Precision	FP16 → INT8 is 2× smaller, INT4 is 4×; always re-evaluate quality
Speculative	lossless; best at low batch with high acceptance

Generative AI & LLMsPage 7

Check yourself

TTFT is 200 ms and TPOT is 30 ms. A reply has 101 output tokens. What is the total latency?

200 + (101 − 1) × 30 = 3200 ms
The first token comes out of prefill, so only 100 decode steps remain

A 7B FP16 model (14 GB weights, 0.5 MB KV per token) runs on a 24 GB GPU. The plan is batch 32 at 4k context. What breaks?

Each sequence needs about 2 GB of KV cache, so 32 need about 64 GB
Only about 10 GB is free, so roughly 5 sequences fit
The bug is sizing for weights only

Speculative decoding gave a big speedup at batch 1 but almost none at batch 256. Why?

At large batch the GPU is already busy, so the spare capacity it uses is gone
Output quality stays lossless either way