Handbooks / Generative AI & LLMs / Chapter 1

How LLMs Work

7 pages · ~5 min✓ Reviewed

Start here. Next up: Prompting Techniques.

Generative AI & LLMsPage 1

Transformer Mental Model

1. The Block Stack

Stack of identical blocks: self-attention then feed-forward network
Each sublayer wrapped in residual connection + layer norm
Embeddings turn token IDs into vectors
Positional information is added so word order is not lost

Embed + positionIDs to vectors
Block × Nattention + FFN
Projectfinal vector
Logitsscore per vocab token

Each block refines the vector of every token
Last layer's vector → logits over the vocabulary

2. Decoder vs Encoder

	Decoder-only	Encoder-only
Examples	GPT, Claude, Llama	BERT
Reads	left to right	both directions
Used for	generating text	understanding tasks

3. Transformer vs RNN

RNN: tokens processed sequentially
Transformer: all tokens in parallel during training

Remember

Parallel training on all tokens is why transformers beat RNNs and could scale.

Generative AI & LLMsPage 2

Tokens and Next-Token Prediction

1. What a token is

Token = sub-word chunk; BPE merges frequent byte pairs
About 4 characters or 0.75 English words per token
Tokenizing is a deterministic text → ID map, done before the model
Model sees IDs only, never raw characters

2. The generation loop

1Logitsscore per vocab token
2Softmaxprobabilities
3Samplepick one token
4Appendto the input
5Repeatuntil stop token

Knob	Effect
Temperature	rescales logits: low = near-greedy, high = diverse
top-k	keep the k likeliest tokens
top-p	keep smallest set with mass ≥ p

3. Where it breaks

Letter counting: the word is split into odd chunks
Arithmetic: numbers split unevenly
Non-English text: many tokens per word

Remember

top-k and top-p cut candidates first, then one token is sampled. Generation is autoregressive.

Generative AI & LLMsPage 3

Attention Mechanism

1. Scores, Scaling and Masking

Each token makes a Query, a Key and a Value vector
Scores = softmax(QKᵀ/√d)
Output = weighted sum of Values

QKᵀraw scores
÷ √dscale down
Causal maskhide later tokens
Softmaxweights
× Vweighted sum

÷√d keeps dot products small, so softmax does not saturate and kill gradients
Causal mask: a token cannot attend to later tokens, so next-token training is valid

2. Heads, Cost and Cache

	What it does	Why it matters
Multi-head	Several attentions in parallel, then concatenated	Heads learn different relations
Cost	Grows as n² with sequence length	Root of long-context expense
KV cache	Stores past Keys and Values	One step per new token, no recompute

Generation

Only the new token needs a fresh Query; its Keys and Values join the cache.

Generative AI & LLMsPage 4

Context Windows in Practice

1. The Window Budget

Context window = max tokens of input + output the model attends to at once
Over the limit: input is truncated or the call errors
Input and output share the window, so a long prompt shrinks the answer room

Inputprompt + history
Outputroom left
Windowinput + output

2. Resent Every Turn

No memory between calls
The whole conversation is resent each turn, and you pay for it each time
Lost in the middle: long prompts are recalled worse mid-prompt

Position in prompt	Recall
Start	strong
Middle	weak
End	strong

3. Window vs Retrieval

Bigger window is not a substitute for retrieval
Long inputs cost more, run slower, dilute attention
Prompt caching reuses a repeated prefix: lower latency and cost

Order the prompt

1System rulesstable
2Reference docsstable
3New questionchanges

Stable first

Put stable content first, changing content last, so the cached prefix matches.

Generative AI & LLMsPage 5

Pretraining, Instruction Tuning and RLHF

1. The Three Stages

1Pretrainingnext-token prediction on huge raw text → base model: completes text, ignores instructions
2SFTfine-tune on prompt-response pairs → follows instructions, desired format
3RLHF / DPOpreference tuning → helpful, safe

Stage	Gives
Pretraining	knowledge
SFT	format and behavior
RLHF / DPO	helpfulness and safety, no new facts

2. RLHF vs DPO

Humans rank outputs
Reward modellearns preferences
PPO-style RLoptimizes the model

DPO skips the separate reward model
Trains directly on preference pairs
Simpler and more stable than PPO

3. Common Mistakes

Expecting a base model to chat: it just continues text
Assuming fine-tuning reliably teaches new knowledge
Reward hacking: the model games the reward model

Remember

Facts come from pretraining. SFT and RLHF only shape behavior.

Generative AI & LLMsPage 6

Transformers Cheat Sheet

1. The Pipeline

Text
Tokens
Embeddings
N × (attention + FFN)
Logits
Softmax
Sample
Append, repeatuntil stop token

2. Attention and Sampling

Attention = softmax(QKᵀ/√d)·V
Causal mask blocks later tokens during generation
Cost is O(n²) in sequence length

Knob	Controls
temperature	sharpness of the distribution
top-k	fixed count of candidates
top-p	probability mass of candidates

3. Context and Training

Window = input + output tokens
Whole conversation is resent every turn
Keep the stable prefix first so caching hits

1Pretrainknowledge
2SFTinstruction following
3RLHF / DPOpreference alignment

Remember

A base model does not chat, and RLHF adds no new facts.

Generative AI & LLMsPage 7

Check yourself

Logits for three tokens are 2, 1, 0. You drop temperature from 1.0 to 0.2. What happens to the top token's probability?

It rises toward 1.
Dividing logits by a small T widens the gaps before softmax.
Sampling becomes near-greedy.

An 8k window, a 7.5k-token prompt, and you ask for a 2k-token answer. What goes wrong?

Input and output share the window.
Only about 500 tokens fit, so the answer is cut off or the call errors.

A chatbot sends only the newest user message each call, yet expects replies to use earlier turns. Why does it fail?

The model has no memory between calls.
Resend the whole conversation every turn.
You pay for those resent tokens each time.

You prompt a base model with 'What is the capital of France?' and it replies with more quiz questions. Why?

Pretraining only teaches continuing text.
It never learned to follow instructions.
Instruction tuning adds that; RLHF aligns preferences.