Handbooks / Generative AI & LLMs / Chapter 3

Embeddings and Vector Search

7 pages · ~5 min✓ Reviewed

Builds on Prompting Techniques. Next up: Evaluation and Hallucinations.

Generative AI & LLMsPage 1

Embeddings and the Vector Mental Model

1. What an Embedding Is

Embedding = fixed-length array of floats made by a model
Typical sizes: 384, 768, 1536 dims
Similar meaning lands at nearby points

2. Meaning Is Position

Pair	Position
car / automobile	close
car / banana	far apart

Dimensions have no human-readable meaning
Only distances and angles between vectors mean anything
Different models = different spaces and dims: never mix them in one index

3. Query and Chunking Rules

Chunksplit long text
Embedone model
Indexvectors
Querysame model

Mixing models

Vectors from two models are not comparable, even when the dims match. A new model means re-embed everything.

Generative AI & LLMsPage 2

Similarity Metrics: Cosine, Dot Product, Euclidean

1. The Three Metrics

Metric	Formula	Reading
Cosine	(a·b) / (\|a\|\|b\|)	-1 to 1; angle only, ignores length
Dot product	sum of a_i*b_i	fastest; higher = closer
Euclidean (L2)	sqrt(sum (a_i-b_i)^2)	smaller = closer; sensitive to magnitude

Dot product equals cosine when vectors are L2-normalised to length 1

2. Worked Example

python

import math
def cos(x, y):
    d = sum(i * j for i, j in zip(x, y))
    return d / (math.hypot(*x) * math.hypot(*y))
a, b, c = [1, 0], [1, 1], [0, 1]
print(round(cos(a, b), 2), round(cos(a, c), 2))

a and b: 1 / (1 * 1.414)

output

0.71 0.0

Cosine 0 means unrelated (right angle)
Cosine near 1 means same direction

3. Mistakes

Using a metric different from the one the model was trained for
Reading cosine distance as if it were similarity

Distance flips the direction

Cosine distance = 1 - cosine. Smaller is closer, the opposite of similarity.

Generative AI & LLMsPage 3

Vector Databases and Core Workflow

1. What a Vector DB Is

Stores vectors + metadata, answers top-k nearest-neighbour queries
DBs: pgvector, Pinecone, Qdrant, Weaviate, Milvus
FAISS = a library, not a DB

2. Ingest and Query

Chunk200-500 tokens, overlap
Embedeach chunk
Upsertvector + id + metadata

Embed querysame model
Top-kchosen metric
Filtermetadata, optional
Returnchunks

3. pgvector and Filters

sql

CREATE EXTENSION vector;
CREATE TABLE docs(id serial, body text,
  emb vector(768));
SELECT body FROM docs
ORDER BY emb <=> $1 LIMIT 5;

$1 = query embedding

Operator	Meaning
`<=>`	cosine distance
`<->`	L2 distance
`<#>`	negative inner product

Filter after search

Tenant, date, language filters must run with the search. Top-k first, then filter, can return fewer than k results.

Generative AI & LLMsPage 4

Approximate Nearest Neighbour Indexes

1. Flat vs ANN

Flat search: query vs every vector, O(N·d)
Perfect recall, fine up to ~100k vectors
ANN trades a little recall for big speed
Most true neighbours, not guaranteed all
Measured as recall@k: share of true top-k found

2. HNSW

Top layersparse entry
Greedy hopnearest node
Drop a layerrepeat
Layer 0top-k

Layered proximity graph: fast, high recall, memory-heavy
M: links per node
efConstruction: build quality
efSearch: query breadth; higher = better recall, slower

3. IVF and PQ

IVF: k-means splits vectors into nlist cells
Search only the nprobe nearest cells; tune for recall
PQ compresses vectors: less memory, some accuracy lost

	HNSW	IVF
Memory	high	lower
Knob	`efSearch`	`nprobe`
Training	none	k-means first

Mistakes

Never skip recall vs a flat baseline. Never build IVF before the table holds enough representative data to train clusters.

Generative AI & LLMsPage 5

Semantic, Keyword and Hybrid Search

1. Keyword vs Semantic

	Keyword (BM25)	Semantic
Matches	exact terms, weighted by frequency and rarity	meaning, via embeddings
Wins on	IDs, error codes, names, rare jargon	'budget airfare' finds 'cheap flights'
Misses	synonyms, paraphrase	exact strings: `ERR_4012`, a SKU

2. Hybrid and the Pipeline

Hybrid = run both, merge the ranked lists
RRF score = sum over lists of 1/(k + rank), k≈60
Uses ranks only: no need to normalise incompatible scores

Hybrid retrievetop 50-100
Cross-encoderrerank
LLMtop 5

Reranker is accurate, but too slow for the whole corpus

3. Mistakes

New embedding model, old index not rebuilt
Ignoring chunk size
Judging by eyeballing results

Measure it

Keep a labelled query set and track recall@k and MRR.

Generative AI & LLMsPage 6

Embeddings and Vector Search Cheat Sheet

1. Vectors and Metrics

Embedding = fixed-length float vector
Same model for queries and documents
Never mix models in one index
Match the metric the model was trained on

Metric	Length	Note
Cosine	ignored	angle only
Dot product	used	= cosine if normalised
L2	sensitive	smaller = closer

2. Pipeline

Ingest: chunk → embed → upsert with metadata

Embedquery
Top-kANN search
Filtermetadata
Rerankbest first

3. Indexes and Search

Index	Idea	Tune
Flat	exact, slow at scale	none
HNSW	fast, high recall, more RAM	`efSearch`
IVF	cluster pruning	`nprobe`
PQ	compression	less accuracy

BM25: exact tokens (IDs, error codes)
Semantic: meaning, paraphrase
Hybrid: merge with RRF, k≈60
Reranker on the top 50-100

Measure it

Check recall@k against a flat baseline, using a labelled test query set.

Generative AI & LLMsPage 7

Check yourself

Query q=[1,0]. Docs x=[10,0] and y=[1,1]. Which wins under cosine, and which under L2 distance?

Cosine: x = 1.0, y ≈ 0.71, so x wins
L2: x is 9 away, y is 1 away, so y wins
Cosine ignores length, L2 does not

You ask for k=10 with a tenant filter applied after the search, but get only 2 rows. Why?

Top 10 were found across all tenants first
Only 2 of them matched the tenant
Apply the filter with the vector search

RRF with k=60. Doc A is rank 1 in BM25 and rank 3 in vectors. Doc B is rank 2 in both. Who wins?

A: 1/61 + 1/63 ≈ 0.03227
B: 2/62 ≈ 0.03226
A wins, by a hair

Pure vector search returns unrelated results for the query ERR_4012. Why, and what fixes it?

Embeddings capture meaning, not exact strings
BM25 matches the exact token
Go hybrid: merge both lists with RRF