Agentic RAG

7 pages · ~5 min✓ Reviewed

Builds on Agent Loops and Memory. Next up: Guardrails and Evaluation.

RAG & AI AgentsPage 1

Agentic RAG: The Mental Model

1. Fixed vs Agentic

Agentic RAG: the LLM decides whether, what, how often to retrieve
It calls a search tool inside its loop

	Fixed RAG	Agentic RAG
Flow	retrieve top-k → stuff → generate	think → call tool → observe → repeat
Retrievals	exactly one	zero to many

2. The Retrieval Tool

json

{"name": "search",
"description": "Searches HR policy docs. Use for leave, pay, benefits.",
"parameters": {"query": "string",
"filters": "object?", "top_k": "int"}}

returns passages: id, text, source

Just a function schema: name, description, JSON-schema params
Returns passages with id, text, source

Description = routing prompt

Say what the corpus holds and when to use it, or the model misuses the tool.

3. Interview Point

Agentic vs fixed? The model loops and decides retrieval; fixed always retrieves once
Schema needs? Clear name, corpus-describing description, typed params, id/text/source results
Fixed still better? Simple, single-hop, latency- or cost-sensitive Q&A

RAG & AI AgentsPage 2

The Agent Retrieval Loop

1. One Turn of the Loop

Modelemits tool_call(search, query)
Appruns retriever
Tool messageresult returned
Modelcontinues or answers

2. Cap the Loop

python

for step in range(MAX_STEPS):
    msg = llm(messages, tools=[search])
    if not msg.tool_calls:
        break
    for call in msg.tool_calls:
        messages.append(run_tool(call))
# cap hit: force a final answer

no tool_call in the reply = done

Cap with max_steps (e.g. 4-6)
Add a total token budget too
Search can never run forever

3. Results and Parallel Calls

Return	Rule
Chunks	top 3-5
Text	trimmed
Ids	stable, like `[doc3#p2]`

Parallel tool calls: several sub-queries in one turn
Cuts latency for decomposed questions

4. Interview Point

RAG & AI AgentsPage 3

Query Rewriting, Decomposition and Multi-Hop

1. Three Query Moves

Rewrite resolves it and that one from history, adds keywords

Move	Example	Runs
Rewrite	'that one?' → 'Pro plan price'	1 standalone query
Decompose	'Compare A and B pricing' → 'A pricing', 'B pricing'	parallel
Multi-hop	'CEO of company that acquired X'	sequential

2. Why Fixed Top-K Misses Hop 2

Hop 1who acquired X?
BridgeAcme
Hop 2CEO of Acme

Fixed pipeline embeds the raw question once
Bridge entity never appears in the question
So the hop 2 chunk is never in top-k

3. Fan-Out or Hops

RAG & AI AgentsPage 4

Deciding When to Retrieve and Self-Check Loops

1. Retrieve or Skip

	Skip	Retrieve
Facts	chit-chat, math, general knowledge	private, recent or specific facts
Context	answer already in context	fact not in context

Routing comes from the tool description plus a system rule
Rule: 'search before answering any question about company docs'

2. Grade, Retry, Stop

Search
Graderelevant / partial / irrelevant
Retryweak grade
Answeror 'not found'

Weak grade: rewrite the query, broaden or narrow the filter, or switch source
Stop after N retries and say 'not found'

3. Groundedness Check

Each claim must be supported by a retrieved passage
Unsupported claim: drop it or flag it

RAG & AI AgentsPage 5

Citing Sources and Failure Modes vs Fixed RAG

1. Citing Sources

Require inline markers like [doc3#p2] that map to ids the tool returned
Render them as links or footnotes
Validate cited ids in code against this run's tool results
Unknown id → reject and regenerate
The regex matches the tool's id format

python

import re
seen = {'doc3#p2'}
answer = 'Pro is $9 [doc3#p2] [doc9#p1]'
print(set(re.findall(r'\[(doc\d+#p\d+)\]', answer)) - seen)

output

{'doc9#p1'}

2. Failure Modes

Failure	Fixed	Agentic
Recall miss, no retry	yes	can retry
Irrelevant top-k stuffed in	yes	grades it out
Multi-hop	cannot	can
Loops on one query	—	yes
Over-retrieves (latency, cost)	—	yes

RAG & AI AgentsPage 6

Agentic RAG Cheat Sheet

1. The Loop and the Tool

tool_call(search)query, filters
Observeid + text + source
Answerenough evidence
Retryuntil max_steps

Schema: name, corpus-describing description, query/filters/top_k
Returns id + text + source, 3-5 chunks
Cap with max_steps (4-6) and a token budget

2. Query Shape and Routing

Move	Use when	Runs
Rewrite	chatty turn → standalone query	once
Decompose	independent parts	parallel
Hop	query N needs result N-1	sequential

Retrieve: private or recent facts
Skip: chit-chat, math, in-context answers

3. Self-Check, Cite, Fail

RAG & AI AgentsPage 7

Check yourself

Fixed RAG gets 'Who is the CEO of the company that acquired Orbit?' and returns the wrong answer. Why?

The bridge entity (the acquirer) is not in the question, so one top-k search never finds it.
Agentic RAG runs hop 1 (find the acquirer), then hop 2 (find its CEO).

An agent calls search('refund policy') six times with the same results, hits max_steps and answers nothing. What fixes it?

Dedupe repeated queries, and force a rewritten query on each retry.
Stop after N retries and answer 'not found'.
max_steps only stopped the endless loop. It did not fix the cause.

The answer cites [doc9#p1], but this run's tool results held only doc3#p2 and doc4#p1. What should happen?

Reject the answer and regenerate it.
Every cited id must exist in this run's tool results, or the citation is fabricated.

	Fixed RAG	Agentic RAG
Fails by	silent miss	looping or skipping
Fix	better top-k	cap, dedupe, trace