Handbooks / Generative AI & LLMs / Chapter 4
Retrieval-Augmented Generation
7 pages · ~5 min✓ Reviewed
Builds on Embeddings and Vector Search. Next up: Evaluation and Hallucinations.
What RAG Is and the Mental Model
1. What RAG Is
- RAG = retrieve relevant text at query time, put it in the prompt
- The LLM answers from that text
- Model weights stay unchanged
- Fresh or private facts come in through the prompt
Two phases
- 1Offline ingestionload, clean, chunk, embed, index
- 2Online queryembed query, retrieve, rerank, assemble, generate
2. RAG vs the Alternatives
| RAG | Fine-tuning | Long context | |
|---|---|---|---|
| Gives | fresh or private facts | style, format, skill | everything, every time |
| Cites sources | yes | no | only if prompted |
| Facts reliable | yes | no | degrades mid-prompt |
| Cost per query | small prompt | small prompt | high, slower |
- Fine-tuning changes behavior, not reliable facts
- Stuffing everything costs more, slows responses, loses details in the middle
- Retrieval keeps the prompt small
3. Mental Model
Remember
Wrong answer? Check the retrieved chunks first, before blaming the model or the prompt.
Chunking Strategies
1. What a Chunk Is
- A chunk is the unit that gets embedded and retrieved
- Too big: embedding is diluted, many topics blur into one vector
- Too small: context is lost, the fact has no surroundings
2. Splitting Methods
| Method | How it splits | Catch |
|---|---|---|
| Fixed-size | every 500 tokens | cuts sentences and ideas mid-way |
| Recursive | paragraph, then sentence, then word | usual default |
| Structure-aware | headings, code blocks, table rows | needs structured source |
Recursive split
- 1Paragraphtry first
- 2Sentenceif still too big
- 3Wordlast resort
3. Overlap and Metadata
- Overlap 10-20%: 500 tokens with 50-100 overlap
- Facts on a boundary are not lost
- Cost: a larger index
Embeddings and Indexing
1. Embeddings
- An embedding maps text to a dense vector
- Similar meaning gives nearby vectors
- Similarity: cosine or dot product
- Embed docs and queries with the same model
- New model means re-embed the whole corpus
2. Flat vs ANN Search
| Index | How it works | Trade-off | Tune |
|---|---|---|---|
| Flat (exact) | Compares every vector | Perfect recall, slow at scale | none |
| HNSW | Graph-based ANN | High recall, fast, heavy memory | ef_search |
| IVF | Clusters vectors, searches nearest clusters only | Less memory, may miss edge clusters | nprobe |
- ANN trades a little recall for big speed gains
- Higher
ef_search/nprobemeans more recall, more latency
Hybrid Search and Reranking
1. BM25 vs Vectors
| BM25 | Vectors | |
|---|---|---|
| Scores by | term frequency + rarity | meaning similarity |
| Wins on | IDs, error codes, names | paraphrase, synonyms |
| Misses | reworded text | exact rare tokens |
2. Hybrid Search and RRF
- Query
- BM25 + vectorin parallel
- Two ranked lists
- RRF mergeone list
- RRF score = sum of
1/(k+rank)over every list; k is about 60 - Uses ranks only, so no score normalization needed
- Docs high in both lists rise to the top
3. Retrieve Wide, Then Rerank
- 1Retrievetop 50-100, cheap
- 2Cross-encoder rerankscore each query-doc pair
- 3Keep top 3-8to the prompt
Context Assembly, Citations and Failure Modes
1. Assembling the Context
- Dedupe near-identical chunks
- Order by relevance
- Stay inside the token budget
- Label each chunk with ID
[1]and its source - Prompt rule: answer only from context, cite IDs, else "I don't know"
text
Answer ONLY from the context. Cite chunk IDs like [1]. If missing, say "I don't know". [1] (docs.acme.com/refunds) Refunds take 5 days. [2] (docs.acme.com/limits) Limit is $500.
prompt skeleton
2. Citations
- Model emits IDs[1][2]
- Checkwas ID in context?
- MapURL or page
3. Failure Modes
| Layer | Failure |
|---|---|
| Retrieval | Bad chunking |
| Retrieval | Vocabulary mismatch |
| Retrieval | Missing metadata filter |
| Retrieval | Stale index |
| Retrieval | Right chunk below cutoff |
| Generation |
RAG Cheat Sheet
1. Pipeline
- Loadoffline
- Clean
- Chunk
- Embed
- Index
- Queryonline
- Retrieve
- Rerank
- Assemble
- Generate
- Cite
2. Defaults
| Knob | Default |
|---|---|
| Chunking | recursive split, 300-800 tokens |
| Overlap | 10-20% |
| Metadata | on every chunk |
| Search | hybrid: BM25 + vectors |
| Fusion | RRF, k=60 |
| Candidates | top 50 |
| After rerank | top 5 |
3. Evaluate and Fix
| Layer | Metrics |
|---|---|
| Retrieval | recall@k, MRR |
Check yourself
With k=60, doc A is rank 1 in BM25 and rank 3 in vector search. Doc B is rank 2 in both. Which wins under RRF?
- A: 1/61 + 1/63 ≈ 0.03227
- B: 2/62 ≈ 0.03226
- A wins, barely. Rank 1 in one list outweighs steady rank 2.
You upgrade the embedding model but only embed new documents with it. Queries use the new model. What breaks?
- Old and new vectors live in different spaces, so similarity scores are meaningless.
- Old documents rank randomly or never appear.
- Fix: re-embed the whole corpus with one model.
Searching for error code E4021 returns unrelated pages, though one page contains it. Why, and what fixes it?
- Embeddings match meaning, and an exact ID token carries little of it.
- Add BM25 keyword search and fuse the two lists with RRF.