Handbooks / RAG & AI Agents / Chapter 1

LLM and Prompt Basics for RAG

7 pages · ~5 min✓ Reviewed

Start here. Next up: The RAG Pipeline.

RAG & AI AgentsPage 1

Tokens and Context Windows: The Mental Model

1. What a Token Is

A token is a sub-word chunk: about 4 English characters or 0.75 words
The model reads and predicts tokens, never raw characters or words

2. Window and Billing

Input tokensprompt + context
Output budgetmax_tokens
Context windowone shared limit

Window = input + output; max_tokens is carved out of the same window
Going over the window truncates or errors
Billing, latency and limits are all counted in tokens

	Input	Output
Counts toward window	yes	yes
Price per token	lower	usually higher

3. Counting and Attention

Tokenizers differ per model: the same text gives different counts
Count with the provider's tokenizer, never word count, for limits
Details in the middle of a long prompt are recalled worse than the start or end

Long context is not free attention

A bigger window fits more, but does not read it all equally well. Put key facts at the start or end.

RAG & AI AgentsPage 2

Sampling Controls: Temperature and Top-p

1. What Sampling Does

1Modelscores every token
2Distributionprobability per token
3Samplingtemperature, top-p
4One tokenappended, repeat

2. Temperature vs Top-p

Temperature rescales the distribution: changes how flat it is
Top-p (nucleus) keeps the smallest set of tokens whose cumulative probability reaches p
With top_p=0.9, sample only inside that set
Tune one, leave the other at default

Temperature	Behaviour
`0`	near-deterministic (greedy)
~0.7 to 1	varied
above 1	erratic

3. Which Setting to Use

Task	Temperature
Extraction, classification	`0`

RAG & AI AgentsPage 3

System vs User Prompts

1. Who Carries What

	System prompt	User message
Holds	role, rules, tone, format	per-turn task and data
Lifetime	every turn	this turn only
Cacheable	yes, it is stable	rarely, it changes
Untrusted data	never	fenced here

Stable instructions go in system so they apply to every turn and can be cached
Untrusted input and retrieved docs go in the user turn, fenced: <context>...</context>

2. Example: Support Bot

python

system = "You are a support bot. Answer only from the provided context."
user = (
    "Question: How do I reset my password?\n"
    "<context>\n[doc1] Reset via Settings > Security.\n</context>"
)

rules in system; question plus fenced context in user

3. Limits and Mistakes

RAG & AI AgentsPage 4

Structured Output with JSON Schemas

1. Three Levels of Strength

Structured output = model returns parseable JSON that matches a schema
Code consumes it directly, no regex

Level	Guarantee
Prompt asks for JSON	Weakest: any text may come back
JSON mode	Valid JSON, any shape
Schema or tool call	Enforces the schema

2. The Schema

type: object, string, number
properties: the fields allowed
required: fields that must appear
enum: only these values
additionalProperties:false: no extra fields

json

{"type":"object",
"properties":{
"sentiment":{"enum":["pos","neg","neu"]},
"score":{"type":"number"}},
"required":["sentiment","score"]}

sentiment is limited to three labels

3. Validate and Mistakes

Call model
Parse JSON
ValidatePydantic / Zod
Retryon failure

RAG & AI AgentsPage 5

Grounding and RAG Prompt Templates

1. Why Models Hallucinate

The model predicts plausible text from training patterns, not verified facts
It has no built-in "I don't know"
Grounding puts retrieved passages in the prompt: answer from evidence, not memory

2. RAG Flow

Embedthe query
Retrievetop-k chunks
Build promptwith chunks
Generatethe answer

3. Template and Citations

text

SYSTEM: Answer only from the context.
Cite sources like [doc2].
Otherwise say "I don't know".
<context>
[doc1] chunk text
[doc2] chunk text
</context>
QUESTION: {question}

rules, then context, then question

Tag every chunk with a source ID
Require citations like [doc2] so answers can be checked

4. Mistakes

Irrelevant or too many chunks: noise and lost-in-the-middle
No refusal instruction
No check that the answer is supported by the context

Remember

Rules, then <context> with source IDs, then the question. Say: answer only from context, else don't know.

RAG & AI AgentsPage 6

Prompting Cheat Sheet

1. Tokens and Sampling

~4 chars per token (about 0.75 words)
Window = input + output tokens, max_tokens included
Count with the model's own tokenizer, not word count
Tune temperature or top-p, never both

Task	temperature
Extraction, classification, code, RAG	0
Brainstorming, creative writing	0.7 to 1

2. Prompts and Structured Output

	System	User
Holds	role, rules, format	task + fenced untrusted data
Fence	none needed	`<context>...</context>`

Schemarequired + enum
ParseJSON
ValidatePydantic / Zod
Retryon failure

RAG & AI AgentsPage 7

Check yourself

The window is 8,000 tokens. Your prompt is 7,500 tokens and you set max_tokens=1000. What happens?

Input plus output must fit, and 7,500 + 1,000 is over 8,000
The call errors or the output is cut short
Shrink the prompt or set max_tokens to 500 or less

The schema's enum is pos, neg or neu. JSON mode returned {"sentiment":"positive"}. Why was that allowed?

JSON mode only guarantees valid JSON, not your shape
Use schema-constrained output, then validate and retry

A RAG bot gets chunks that do not contain the answer, yet it answers confidently. What is missing?

A refusal rule: answer only from the context, otherwise say you don't know
Also require citations like [doc2] so answers can be checked