Handbooks / Generative AI & LLMs / Chapter 5

Fine-Tuning vs Prompting

7 pages · ~5 min✓ Reviewed

Builds on Evaluation and Hallucinations.

Generative AI & LLMsPage 1

The Customization Ladder

1. Three Levers

	Prompting	RAG	Fine-tuning
Changes	Instructions at inference	Docs injected at query time	Weights
Fixes	Task clarity	Missing or changing knowledge, citations	Behavior, style, format, domain skill
Limit	Context window, model's skills	No weight change	Poor at fresh facts
Start	Zero cost, instant	Index + retriever	Data + training

2. Which Gap?

Always try prompting first
Fine-tuning shapes how it answers, not today's facts

Mistake

Fine-tuning to add knowledge: facts go stale and get hallucinated. Use RAG for facts.

3. Interview Point

Fine-tune on company docs? No: weak at facts, stale, no citations. Use RAG
Combine RAG and fine-tuning? Domain behavior plus fresh facts
Failing eval? Missing facts: RAG. Wrong format: fine-tune. Else fix the prompt

Generative AI & LLMsPage 2

Choosing Between Prompt, RAG and Fine-Tune

1. Decision Order

Prompt+ few-shot baseline
Add RAGfacts missing or stale
Fine-tunebehavior still fails, or a smaller model is needed

2. Compare

	Prompt	RAG	Fine-tune
Data freshness	fixed	best	poor
Style/format	ok	ok	best
Setup effort	lowest	medium	highest
Per-query cost	long prompts	adds context	can shrink prompts

3. When Each Wins

Fine-tune wins	RAG wins
Strict output format	Data changes often
Tone	Source citations needed
High-volume classification or extraction	Per-user access control
Replaces long few-shot, cuts tokens	Corpus too big for context

Mistake

Fine-tuning to add fresh facts gives stale, hallucinated answers. Use RAG.

4. Interview Point

Weekly policy changes? RAG: update the docs, no retraining; fine-tune only for tone
Fine-tune beat prompting? Same held-out eval, prompt-only baseline vs tuned; compare metric, cost, latency
Long context replaces RAG? Only for small corpora; cost, latency, no per-user access control or citations at scale

Generative AI & LLMsPage 3

Full Fine-Tuning vs LoRA and PEFT

1. Full FT vs PEFT

Full FT updates all weights
Needs weights + gradients + Adam states: ~12-16 bytes/param
PEFT freezes the base, trains few extra params

2. LoRA and QLoRA

Method	Trains	7B memory
Full FT	all weights	~100 GB+
LoRA	BA, r 8-64, under 1%	far less
QLoRA	LoRA on 4-bit base	one 16-24 GB GPU

LoRA: frozen W + trainable low-rank BA
Adapters are MBs, swappable per task or customer
Merge into W: no extra latency
Full FT: full model copy per task

3. Key knobs

Knob	Setting
rank r	update capacity
alpha	scaling, often 2r
targets	attention q,v or all linear
LR	LoRA 1e-4 to 2e-4, full FT ~1e-5

LR trap

Full-FT LR under-trains the adapter.

4. Interview Point

Why low rank? Updates have low intrinsic rank, so BA fits them
LoRA vs QLoRA? QLoRA: less memory, small quality loss, slower
Full FT still worth it? Big domain shift, new skills, top quality

Generative AI & LLMsPage 4

Data Needs, Costs and Distillation

1. Data

Quality beats quantity
Style/format with LoRA: 100s–1000s clean, diverse examples
New skills or domains: 10k+ examples
Instruction/response pairs or chat JSONL, same as the inference template
Hold out 10–20% as eval set

2. Cost

Item	Note
Training	tokens × epochs × model size
LoRA	mainly cuts memory; compute drops only modestly
Hidden	labeling, eval, re-training per base upgrade
Hosted fine-tune	often pricier per token than base
Merged LoRA	self-hosted, no overhead
Multi-adapter	one shared base

3. Distillation

Teacherlarge model
Studentsmall model
Resultnear-teacher quality, cheaper

Teacher outputs: labels or soft logits
Check teacher license terms
Best on a narrow task

Remember

Eval set stays real, never teacher-generated.

Generative AI & LLMsPage 5

Choosing Model Size and Common Mistakes

1. Size and Picking

Per 1B params	fp16	4-bit
Weights	~2 GB	~0.5 GB

Strongest modelprove feasibility
Step downone size
Eval checkmetric vs target
Keep last passsmallest sufficient

Bigger = better reasoning, fewer failures; costs more latency, cost, memory
Memory = weights + KV cache
Small fine-tuned model often beats a large prompted one on a narrow task

2. Common Mistakes

Fine-tuning to add knowledge: hallucinated, stale facts; use RAG
No eval set or baseline: cannot tell if it helped
Noisy, duplicated or test-leaking data

Over-training

Causes catastrophic forgetting and overfitting. Fix: fewer epochs, lower LR, lower rank, mix in general data.

3. Interview Point

Smallest sufficient model? Step down from the strongest until eval drops below target.
Catastrophic forgetting? Tuning erases general skills; mix in general data.
Detect overfitting? Train loss keeps falling while held-out eval worsens.

Generative AI & LLMsPage 6

Customization Cheat Sheet

1. Pick the Fix

Gap	Fix
Missing or changing facts	RAG
Wrong style, format or behavior	Fine-tune
Unclear task	Better prompt + few-shot
Domain behavior + fresh facts	RAG + fine-tune

Order of effort

1Prompt
2RAG
3LoRA / QLoRA
4Full fine-tune

2. LoRA Defaults

r 8-16, alpha 2r, all linear layers
LR 1e-4 to 2e-4, 1-3 epochs
QLoRA fits a 7B in ~16 GB

python

LoraConfig(r=16, lora_alpha=32,
           target_modules="all-linear")
# lr 2e-4, num_train_epochs=2

3. Data and Cost

100s-1000s clean examples for style
10k+ for new skills
Hold out an eval set
Keep a prompt-only baseline

Cut cost with

Distillation to a small student, merged adapters, and the smallest model that still meets the eval bar.

Generative AI & LLMsPage 7

Check yourself

A bot is fine-tuned on policy docs. Policies change weekly and it now states old rules. Fix?

Use RAG: retrieve current docs at query time.
Fine-tuning injects facts poorly and goes stale.

LoRA rank r=8 on a 4096x4096 weight: how many params train?

B (4096x8) plus A (8x4096) = 65,536.
Full weight is 16.7M, so about 0.4%.

Train loss keeps falling, eval loss rises, general answers worsen. Diagnosis and fix?

Overfitting and catastrophic forgetting.
Fewer epochs, lower LR or rank, mix in general data.