Handbooks / RAG & AI Agents / Chapter 9
Evaluating RAG and Agents
7 pages · ~5 min✓ Reviewed
Builds on Guardrails and Evaluation. Next up: Cost and Latency in Production.
Evaluation Mental Model and Dataset Building
1. Evals as Unit Tests
- Evals = unit tests for non-deterministic systems
- Fixed inputs, scored outputs, tracked over time
- Changes are measured, not guessed
- An overall score alone cannot locate the failing stage
- Retriever
- Generator
- Agent steps
- End to end
2. Golden Record
json
{"input": "How do I reset SSO?",
"reference": "Admin console > SSO > Reset",
"gold_doc_ids": ["kb-112"],
"tags": {"category": "auth",
"difficulty": "hard"}}input + reference or expected facts + gold doc IDs + tags
3. Sources and Coverage
| Source | Trust |
|---|---|
| Real production logs | best |
| Expert-written cases | good |
| Synthetic LLM cases | always human-spot-check |
- Edge cases on purpose: unanswerable, ambiguous, adversarial, rare categories
- 50-200 curated cases beat thousands of noisy ones
- Keep a held-out split of cases
Overfitting
Tuning prompts against the same cases every time fits the set, not the task. Keep the held-out split unseen.
Retrieval Metrics: Recall@k, MRR and Friends
1. Recall and Precision
- Recall@k = relevant docs in top k / total relevant docs
- Tells you if the answer evidence reached the generator
- Precision@k = relevant docs in top k / k
- Low precision = noisy context that distracts the model
2. MRR and nDCG
- MRR = mean over queries of 1/rank of the first relevant hit
- Only the first hit counts; later hits are ignored
- nDCG handles graded relevance and rewards several good docs ranked high
| First hit at | 1/rank |
|---|---|
| Rank 1 | 1.0 |
| Rank 3 | 0.33 |
| Rank 4 | 0.25 |
| Miss | 0 |
3. Worked Example
- Query Agold at rank 3 → 1/3
- Query Bgold at rank 1 → 1
- MRR(1/3 + 1) / 2 ≈ 0.67
Answer Faithfulness and Relevance
1. Faithfulness
- Faithfulness (groundedness): every claim is supported by the retrieved context
- Low faithfulness = hallucination
- Example: 4 of 5 claims supported gives score 0.8
- Answerfull text
- Atomic claimssplit
- Check eachvs context
- Scoresupported / total
2. Three Relevance Checks
| Metric | Asks | Tells you |
|---|---|---|
| Faithfulness | Is it in the context? | Hallucination |
| Answer relevance | Does it address the question? | On topic, true or not |
| Context relevance | How much context was useful? | Retrieval vs generation failure |
- Faithful but irrelevant: true to the context, ignores the question
- Relevant but unfaithful: answers the question with invented facts
3. Where to Fix
LLM-as-Judge: Rubrics and Biases
1. Rubric Design
- Open-ended output, exact match fails: use a judge model
- The judge prompt is a spec: test it like code
- One criterion per judge call
- Anchors: what a 1, 3 and 5 look like
- Reasoning first, then score;
temperature=0
json
{"criterion": "faithfulness",
"reasoning": "2 of 3 claims cited",
"score": 3}structured JSON, reasoning before score
2. Pairwise and Biases
- Pairwise is more reliable than absolute scores
- Run both orders; a disagreement counts as a tie
| Bias | Judge tends to |
|---|---|
| Position | prefer the first answer |
| Verbosity | prefer the longer answer |
| Self-preference | favor its own model family |
| Hard math or facts | be weak: verify separately |
3. Calibrate
Agent Trajectory Evaluation and CI Regression Suites
1. Trajectory and Metrics
- Trajectory = ordered steps: thoughts, tool calls, arguments, observations
- Score the path, not just the final answer
- Metrics: tool selection, argument validity, step efficiency (steps vs optimal)
- Also: loops or redundant calls, and task success rate
2. Matching a Reference
| Mode | Passes when |
|---|---|
| Exact | same steps, same order |
| In-order | reference steps appear in order, extras allowed |
| Any-order | all reference steps present, any order |
- Prefer outcome checks: final state, e.g. the DB row created
- Many valid routes exist, so exact paths fail good runs
3. Regression Suite in CI
- Every PRsmall fast subset
- Nightlyfull set
- Gatethreshold per metric
Evals Cheat Sheet
1. Dataset and Retrieval
- Record: input + reference + gold doc IDs + tags
- Mine production logs; add unanswerable cases
- Keep a held-out split
- MRR example: ranks 3 and 1 give (1/3 + 1)/2 ≈ 0.67
| Metric | Asks | Score |
|---|---|---|
recall@k | found? | relevant in top k / total relevant |
precision@k | noise? | relevant in top k / k |
| MRR | first hit | mean(1/rank of first hit) |
| nDCG | graded relevance | good docs ranked high |
2. Generation and Judge
- Faithfulness = supported claims / total claims
- Answer relevance: addresses the question
- Context relevance: how much context was useful
- Judge: one criterion per call, anchored rubric
- Reasoning first, temp 0, swap order
Check yourself
A judge picks the answer shown first in 80% of pairwise runs, whatever the answer is. What is this, and what is the fix?
- Position bias.
- Run both orders and count a disagreement as a tie.
The gold doc ranks 2nd for query A, 4th for query B, and is not retrieved for query C. What is MRR?
- (0.5 + 0.25 + 0) / 3 = 0.25
- The miss counts as 0. It is not skipped.
Recall@5 is 0.95 but faithfulness is 0.60. Fix retrieval or generation?
- Generation. Recall of 0.95 means the gold docs reached the generator.
- Tune the prompt or model.