Handbooks / RAG & AI Agents / Chapter 2

Chunking and Retrieval

7 pages · ~5 min✓ Reviewed

Builds on The RAG Pipeline. Next up: Tool Calling.

RAG & AI AgentsPage 1

Retrieval Mental Model

1. What retrieval is

  • Retrieval picks the top-k chunks from an index for the prompt
  • Right chunk missing → answer cannot be grounded, even with a great LLM

2. Pipeline and two stages

  1. Rewritequery
  2. Filtermetadata
  3. Retrievedense + keyword
  4. Re-rankcandidates
  5. Top fewchunks
  6. Generateanswer
Stage 1Stage 2
JobRetrieveRe-rank
Costcheap, wideslow, precise
Keepstop 50-100top 3-5
Goalhigh recallprecision

3. Dense vs keyword

Dense vectorsBM25
Matchesmeaningexact tokens
MissesIDs, error codes, namesparaphrases
RAG & AI AgentsPage 2

Chunk Size and Overlap

1. Size and overlap

  • Chunk size = tokens per embedding
  • Overlap = share of previous chunk repeated at the next chunk's start
  • Start: 256-512 tokens, 10-20% overlap
  • Tune on recall@k with your own queries
SizeProCon
Small 100-200precise matchloses context
Large 800-1500keeps contextdilutes embedding, wastes prompt

2. Overlap

  • Stops a fact being split across a boundary
  • Size 500, overlap 50: chunk 2 starts at token 450
  • Cost: bigger index, near-duplicate hits

3. Split order

  1. Headingsfirst
  2. Paragraphs, code blocksnext
  3. Fixed token windowfallback

4. Mistakes

  • Zero overlap
  • 50% overlap duplicates the index
  • One chunk size for every doc type
RAG & AI AgentsPage 3

Metadata Filtering and Hybrid Search

1. Filtering: pre vs post

  • Metadata filter limits candidates by source, date, tenant_id, doc type, language
  • Fields must be stored at ingest
Pre-filterPost-filter
OrderInside the ANN searchTop-k first, then filter
ResultsExact, k keptCan return fewer than k or zero
VerdictPreferAvoid if selective

2. Hybrid search and RRF

  1. Query
  2. BM25 + densein parallel
  3. Merge ranksRRF
  4. Top-k
  • Use hybrid when queries hold IDs, acronyms, error codes
  • RRF score = sum of 1/(k+rank) over lists, k ≈ 60
  • RRF needs no score normalisation
  • Weighted sum needs normalised scores

3. Mistakes

  • Filtering on fields never stored at ingest
  • Post-filtering with a selective filter
  • Summing raw BM25 and cosine scores
Multi-tenant data

Always filter by tenant_id.

RAG & AI AgentsPage 4

Re-ranking and Query Rewriting

1. Re-ranking: Two Stages

  1. Retrieveabout 50 candidates
  2. Re-rankcross-encoder rescores
  3. Keeptop 3-5
Bi-encoderCross-encoder
Readsquery, doc apartquery + doc together
Indexableyesno
Speed / accuracyfast / goodslow / higher
  • A re-ranker rescores the candidates and reorders them
  • It only reorders stage 1 results: a missing answer stays missing
  • Fix recall first, then precision

2. Query Rewriting

TechniqueWhat it does
Expandspell out abbreviations
Standalonefollow-up + chat history becomes a full question
HyDEembed a generated hypothetical answer
Multi-queryseveral paraphrases, results merged

3. Mistakes

  • Re-ranking too few candidates
  • Rewrite that drifts from the user's intent
  • Added latency with no measured gain

4. Interview Point

  • Bi- vs cross-encoder? Separate, indexable vectors vs joint reading that is more accurate
  • Cross-encoder for first stage? It scores every pair with no index: too slow over the corpus
  • Rewriting in chat? It turns a vague follow-up into a standalone query
RAG & AI AgentsPage 5

Measuring Retrieval: recall@k and MRR

1. Labelled set

  • (query, relevant IDs) pairs, ideally 50-200 real queries
  • No set = tuning is guesswork

2. The two metrics

recall@kMRR
Scorerelevant found in top k, averaged; one relevant chunk = hit ratemean of 1/rank of first hit
Example8 of 10 queries hit in top 5 = 0.8ranks 1, 2, 4: (1+0.5+0.25)/3 = 0.583
Measurescandidate coverageordering, rewards right chunk first
Use forstage 1re-ranker (or nDCG)

3. Tuning loop

  1. Label set50-200 queries
  2. Baselinerecall@k, MRR
  3. Change ONE thingchunk size, hybrid
  4. Re-measurekeep if better

4. Mistakes

  • Synthetic queries only
  • Tuning on the test queries
  • Average only: inspect failing queries
  • k larger than the prompt can hold
RAG & AI AgentsPage 6

Retrieval Tuning Cheat Sheet

1. Chunk and Filter

  • 256-512 tokens, 10-20% overlap
  • Split on structure first, then token windows
  • Pick the size by recall@k, not habit
  • Pre-filter on metadata, always tenant_id

2. Hybrid, Re-rank, Rewrite

  1. BM25 + densemerge with RRF
  2. Top ~50bi-encoder
  3. Re-scorecross-encoder
  4. Keep 3-5
python
score = sum(1 / (60 + rank) for rank in ranks)

RRF: ranks of one chunk across lists

  • Re-ranker cannot recover a chunk that was never retrieved
  • Rewrites: standalone follow-up, expansion, HyDE, multi-query
  • Keep a rewrite only if the metrics improve

3. Measure

MetricFormulaTunes
recall@krelevant in top k / all relevant, mean over querieschunking, hybrid, filters
MRRmean of 1/rank of first relevantre-ranker
Debug order

recall@k first, then MRR. recall@k equals hit rate with one relevant chunk per query.

RAG & AI AgentsPage 7

Check yourself

A search takes the top 5 chunks, then filters on tenant_id = acme. Only 1 of the 5 belongs to acme. What comes back, and what is the fix?
  • Just 1 chunk, fewer than k, because filtering ran after the top-k cut.
  • Pre-filter inside the ANN search so all 5 come from acme.
RRF with k=60. Doc A is rank 1 in BM25 and rank 3 in dense. Doc B is rank 2 in both. Which wins?
  • A: 1/61 + 1/63 ≈ 0.03227.
  • B: 1/62 + 1/62 ≈ 0.03226.
  • A wins, narrowly.
The right chunk sits at rank 8 in stage 1. You re-rank only the top 5. Why does the answer stay wrong?
  • A re-ranker only reorders what stage 1 returned.
  • The chunk never reached it. Retrieve about 50 candidates, then re-rank.