Handbooks / Generative AI & LLMs / Chapter 4

Evaluation and Hallucinations

7 pages · ~4 min✓ Reviewed

Builds on Embeddings and Vector Search. Next up: Fine-Tuning vs Prompting.

Generative AI & LLMsPage 1

Why Models Hallucinate

1. What It Is

  • Hallucination = fluent output that is false or unsupported by the source
  • Model predicts the likely next token, not a verified fact
  • Cause: gaps or errors in training data
  • Cause: rare, long-tail facts
  • Cause: knowledge cutoff (no data after a date)

2. Two Kinds

TypeVersus the contextExample
IntrinsicContradicts itSource says 2019, answer says 2021
ExtrinsicAdds claims it cannot verifyInvents a quote not in the source

3. Why It Persists

  • Training rewards confident guessing over "I don't know"
  • Models abstain too rarely
  • Allow a 'not found' answer and instruct it: abstaining becomes far likelier

4. Common Mistake

Low temperature is not truth

Temperature 0 makes decoding greedy: it picks the likeliest token, not a verified fact.

Generative AI & LLMsPage 2

Grounding and Citations

1. What Grounding Is

  • Grounding = answers rest on supplied sources, not model memory
  • Sources: RAG chunks, tool/API results, database lookups

2. Workflow and Citations

  1. Retrievetop-k chunks
  2. Promptchunks as context
  3. Instructanswer only from the context
  4. Fallbackelse say 'not found'
  • Citation = tag each claim with a source ID or chunk quote
  • A reader or a program can then verify the claim

3. Verify Citations

  • Check the cited ID exists in the retrieved set
  • Check the quoted span appears in that source
  • Reject any claim that fails either check
python
def verify(cite, sources):
    text = sources.get(cite["id"])
    return text is not None and cite["quote"] in text

missing ID or missing quote means reject

A citation proves nothing alone

Models can cite real sources that do not support the claim. Bad retrieval gives confident wrong answers.

Generative AI & LLMsPage 3

Evaluation Methods

1. Three Ways to Score

MethodGood forCatch
Benchmarkcheap, reproducible, comparablecontamination; may not match your task
LLM-as-judgescales cheaply, rubric scoringposition, verbosity, self-preference bias
Human evalgold standard for nuance, safetyslow, costly
  • Classic MMLU, HumanEval, GSM8K are saturated at ~90%+
  • MMLU-Pro, GPQA Diamond are nearing saturation too
  • Separate current models with Humanity's Last Exam, SWE-bench Pro

2. Golden Set and Agreement

  1. Golden set50-200 real examples
  2. Expected answersper example
  3. Runevery prompt or model change
  • Measure inter-annotator agreement: Cohen's kappa

3. Mistakes

  • Relying on public benchmark rank alone
  • Trusting an unvalidated judge
Calibrate the judge

Compare judge scores with human labels, and swap answer order to cancel position bias.

Generative AI & LLMsPage 4

Latency and Cost Metrics

1. Latency and Throughput

  1. Sendprompt
  2. TTFTfirst token
  3. Streamper-token time
  4. Donetotal latency
  • TTFT = time to first token; drives perceived speed
  • Inter-token latency = gap between streamed tokens
  • Total latency = TTFT + tokens × per-token time
  • Throughput = tokens/sec or requests/sec
  • Track p50 and p95/p99, not just the average

2. Cost

  • Cost = input tokens × input price + output tokens × output price
  • Output tokens usually cost several times more than input

3. Levers and Mistakes

LeverUse it for
Smaller modeleasy tasks
Prompt cachingrepeated prefixes
Shorter promptsfewer input tokens
max_tokens capbounds output cost
Batchingoffline jobs
StreamingUX, faster first feedback
Cost per successful task

Average cost per call hides waste. Retries, long contexts and agent loops multiply spend.

Generative AI & LLMsPage 5

Safety Basics

1. Prompt Injection

  • Prompt injection: untrusted text carries instructions that hijack the model
  • Carriers: a web page, an email, a document
  • Indirect injection is the common real-world form

2. Layered Defenses

  1. Separatetrusted vs untrusted
  2. Least privilegeminimal tools
  3. Confirmdestructive actions
  4. Sandboxexecution

3. Guardrails

StageChecks
Inputmoderation or classifier filter, PII redaction
Outputschema validation, refusal handling
Before shell, SQL, HTMLtreat output as untrusted (injection risk)

4. Mistakes

  • Secrets in prompts: never put keys or tokens there
  • Model output passed raw to a shell, SQL or HTML
Not a defense alone

A system-prompt line 'ignore malicious instructions' is not enough. Add red-teaming and log flagged cases.

Generative AI & LLMsPage 6

LLM Reliability Cheat Sheet

1. Cut hallucinations

  1. Groundretrieve top-k
  2. Citesource ID + quote
  3. Allow 'not found'no forced guess
  4. VerifyID exists, quote matches

2. Eval stack

LayerUse it for
Golden set (50-200)regressions, every change
Benchmark (HLE, SWE-bench Pro)model choice
Calibrated LLM-judgescale
Human reviewhigh stakes

3. Latency and cost

  • TTFT = how fast it feels
  • p95 = reliability, not the average
  • Cost = input tokens x price + output tokens x price
  • Smaller model
  • Prompt caching
  • max_tokens cap
  • Batching

4. Safety

  • Assume injection
  • Least privilege tools
  • Validate outputs
  • Redact PII
  • Red-team
Temperature 0

It makes decoding greedy: the likeliest token, not a verified fact.

Generative AI & LLMsPage 7

Check yourself

You set temperature to 0, yet the model still states a wrong fact. Why?
  • Temperature 0 only makes decoding (nearly) greedy, not true.
  • It still picks the likeliest token, which can be wrong.
  • Fix: ground the answer with retrieval.
Your LLM judge always prefers the longer answer. What is this, and how do you fix it?
  • Verbosity bias.
  • Use a rubric that penalizes padding, or compare length-matched answers. Calibrate against human labels.
  • Swapping answer order fixes position bias only.
An answer cites [doc3], and doc3 exists. Is the claim verified?
  • No. Check that the quoted span appears in doc3 and supports the claim.
  • Bad retrieval can still give a confident wrong answer.