Handbooks / RAG & AI Agents / Chapter 3
Embeddings and Vector Similarity
7 pages · ~4 min✓ Reviewed
Builds on The RAG Pipeline. Next up: Chunking and Retrieval.
What an Embedding Is
1. Embedding = Meaning as Geometry
- Embedding = fixed-length float vector made by a model
- Similar inputs land close together in space
- Similarity signal = distance or angle between vectors
- No single dimension means anything on its own
2. Same Model, Same Pipeline
- Compare vectors only from the same model and version
- Mixed models give meaningless scores
- One pipeline embeds docs at index time and queries at search time
- If the model requires it, add prefixes:
query:vspassage:
3. Core Workflow
- Chunksplit text
- Embedeach chunk
- Storevector + metadata
- Embed querysame model
- Top-knearest vectors
One model end to end
Docs and queries must share the same model, version and prefix rules, or scores mean nothing.
Distance Metrics and Normalisation
1. The Three Metrics
| Metric | Formula | Better = | Magnitude |
|---|---|---|---|
| Cosine | dot(a,b)/(|a||b|) | higher (-1 to 1) | ignored |
| Dot product | sum(a_i*b_i) | higher | rewarded |
| Euclidean (L2) | sqrt(sum((a_i-b_i)^2)) | lower | sensitive |
- Cosine measures angle only
- Dot product rewards angle and length: longer vectors score higher
- L2 is straight-line distance
2. Normalise to Unit Length
- v / |v|unit length
- cosine == dot|a|=|b|=1
- L2^2 = 2 - 2*cosmonotone
- Same rankingall three
python
import numpy as np v = v / np.linalg.norm(v)
normalise docs and queries the same way
3. Which Metric?
Choosing an Embedding Model and Dimensions
1. Selection Criteria
- Recall on your own domain data, not just MTEB rank
- Max input tokens and language coverage
- Latency, cost, licence
2. Dimensions
- More dims (1536, 3072): more capacity, slower search
- Memory = N * dims * 4 bytes (float32)
- Matryoshka models: truncate 3072 to 256, small loss
- Re-normalise after truncating
python
v = v[:256]
v = v / np.linalg.norm(v)truncate, then re-normalise
3. Quantisation and Migration
| Type | Size vs float32 |
|---|---|
| float16 | 1/2 |
| int8 | 1/4 |
| binary | 1/32 |
- Usually rescore top hits on full vectors
- Version indexv2
- Dual-write
- Re-embed corpus
- Switch reads
Vector Indexes: Flat, HNSW and IVF
1. Flat: Brute Force
- Compares the query to every vector: O(N) per query
- Exact, 100% recall; fine up to ~100k vectors
2. HNSW: Layered Graph
- Top layersparse, enter here
- Greedy hopsmove closer
- Descendlayer by layer
- Bottom layerdense, final
- Fast and high recall, but high memory
| Knob | Meaning | Trade-off |
|---|---|---|
M | links per node | more recall, more RAM |
efConstruction | build quality | slower build |
efSearch | candidates at query | more recall, more latency |
3. IVF and IVF-PQ
- k-means splits vectors into
nlistcells; needs a training step
ANN Recall Trade-offs and Metadata Filtering
1. Recall@k
- ANN trades exactness for speed
- recall@k = share of the true top-k (from flat search) the index returns
- Measure: run exact search on sampled queries, then compare
- Raise
efSearchornprobeuntil recall hits ~95% at acceptable p95
2. Metadata Filtering
| Post-filter | Pre-filter / filtered HNSW | |
|---|---|---|
| When | After top-k search | During traversal |
| Count | Fewer than k, or zero | Correct |
| Risk | Selective filter empties results | Slow search, fragmented graph |
| Fix | Overfetch a larger k | Brute force on the small filtered set |
3. Common Mistakes
- No recall measurement
- Filtering after a small top-k
- Mixing metrics between index and query
Embeddings and Vector Search Cheat Sheet
1. Similarity and Memory
| Metric | Measures | Better |
|---|---|---|
| Cosine | angle | higher |
| Dot | angle × magnitude | higher |
| L2 | distance | lower |
- Unit vectors: cosine dot==, all three rank identically
- Memory = N × dims × 4 bytes (float32)
- 1M × 1536 ≈ 6 GB before index overhead
2. Index Choice and Dials
| Index | Pick when | Dial |
|---|---|---|
| Flat | under ~100k, exact | none |
| HNSW | low latency, high recall, enough RAM | efSearch |
| IVF(-PQ) | huge or memory-bound corpus | nprobe |
Check yourself
You compare [3,4] with [6,8]. What do cosine and L2 say?
- Cosine = 1.0: same direction
- L2 = 5, not 0: magnitude differs
- After normalising both are [0.6,0.8], so L2 = 0
Top-10 search, then filter tenant=acme (1% of docs). You get 0 results. Why?
- Post-filtering dropped all ten neighbours
- Fix: pre-filter, overfetch, or brute-force the small filtered set
HNSW recall@10 is 82% and p95 has spare room. Which knob?
- Raise
efSearch: more candidates, higher recall, more latency - Check recall against flat search