The RAG Pipeline

• RAG = retrieve text at query time, add it to the prompt
• Model stays frozen; knowledge lives outside it
• Gap 1: stale knowledge (training cutoff)
• Gap 2: private data it never saw
• Gap 3: hallucination with no source to check

• Why not fine-tune facts? They go stale, give no citations, and cost a retrain to update
• Two pipelines? Offline indexing; online query
• Skip RAG when? The corpus is small and static, so it fits in context

• Metadata on every document: source, title, page, date, ACL

• Too big: diluted embedding
• Too small: lost context
• Start at 200-500 tokens with 10-20% overlap

• Prepend the section title so an orphan paragraph keeps context

for sec in split_on_headings(doc):
    for part in cap_tokens(sec.text, 400):
        chunks.append({
            "text": sec.title + "\n" + part,
            **sec.meta})

pseudocode: 400-token cap per chunk

• Embedding model: text → fixed-length vector, similar meaning lands close
• Compare by cosine or dot product; equal when normalised
• Same model for chunks and queries
• Model change → re-embed corpus
• Check max input tokens: no silent truncation

• ANN trades a little recall for speed

• Same model? Different models, different spaces: distances mean nothing
• HNSW? Graph hops skip most vectors: faster, may miss a true neighbour
• Flat enough? Small corpus: an O(N) scan is fast

• Embed the query, fetch top-k chunks (k often 5-20)
• Similarity threshold drops weak matches
• Dense = meaning; BM25 = IDs, names, error codes
• Merge both lists with RRF: score = Σ 1/(60 + rank)

• Metadata filters at query time: tenant, date, doc type, permissions
• Query rewriting and HyDE help vague or multi-turn questions

• Why BM25 plus vectors? Vectors miss exact IDs; BM25 misses meaning.
• Cross-encoder edge? Reads query and chunk jointly, so scores are sharper.
• Measure quality? recall@k (right chunk in top-k), MRR (mean of 1/rank of the first right chunk).

• Rule: answer only from the context
• Answer missing → say "I don't know"
• Top-k must fit the window, room left for the answer
• Dedupe near-identical chunks; order by relevance
• Best chunks first and last; middle recalled worst

• Chunks too large or too small
• No reranking
• Stuffing too much context
• No refusal rule
• Index not versioned
• Never evaluating

• Chunk 200-500 tokens, 10-20% overlap
• Retrieve top 20-50, rerank to 5
• Same embedding model for index and query; cosine similarity
• HNSW index for scale

• Hybrid search + reranker
• Metadata and ACL filters
• Refusal instruction, verified citations
• Index versioning
• Eval set: recall@k, faithfulness

• 60 seconds: index the docs offline; online embed, retrieve, rerank, prompt, cite
• Top 3 levers: hybrid search, reranking, chunking
• Fresh index: re-ingest and re-embed changed docs by doc ID, delete removed ones
• Model change: bump the index version, re-embed everything