Handbooks / RAG & AI Agents / Chapter 10

Cost and Latency in Production

7 pages · ~5 min✓ Reviewed

Builds on Evaluating RAG and Agents.

RAG & AI AgentsPage 1

Cost and Latency Mental Model

1. Cost Formula

Cost = input tokens × input price + output tokens × output price
Output tokens cost 3-5× input tokens, so cut output length first
Example at 4×: 500 output tokens cost as much as 2,000 input tokens

2. Agent Loops Grow Quadratically

Each turn resends the whole history
Assume history starts at 0: step n sends n × 1k tokens
Total sent = n(n+1)/2 × 1k, so input grows quadratically

Steps (n)	Input tokens sent in total
1	1k
2	3k
10	55k
20	210k

3. Latency

TTFT = time to first token: the wait before the first output token appears
Latency = TTFT + output tokens × per-token decode time
TTFT is driven by input size

4. Four Levers

Lever	Technique
Send fewer tokens	trim history
Reuse tokens	prompt caching
Cheaper model	routing
Do work concurrently	streaming, parallel calls

Interview trap

Measure cost per successful task, not per request: retries and failed runs hide in the total.

RAG & AI AgentsPage 2

Token Accounting and Prompt Caching

1. Count Every Token

Log usage from every response, never estimate
Tag each record with run_id, step and model

python

log.info(run_id=run_id, step=step, model=r.model,
  input_tokens=u.input_tokens,
  output_tokens=u.output_tokens,
  cache_read_tokens=u.cache_read_tokens,
  cache_write_tokens=u.cache_write_tokens)

one record per response

2. Prompt Caching

Input	Price vs base	When
Normal	1×	no cache
Cache write	≈ 1.25×	first request
Cache read	≈ 0.1×	later hits

Reuses a prefix of the prompt; pays off from the second hit
TTL is about 5 min by default; each hit refreshes it
Low-traffic prompts expire between calls and never get hits

3. Prefix Order and Breakers

RAG & AI AgentsPage 3

Model Routing and Streaming

1. What Routing Does

Step	Model
classify, extract, summarize, tool args	Small
planning, multi-step reasoning	Large

2. Router Types

Router	Decides by
Rule-based	task type, input length
Classifier	small model scores difficulty
Cascade	try small, escalate on low confidence or failed validation

Small modelfirst try
Checkconfidence, validation
Escalatelarge model

Cascade cost check

escalation rate × large cost + small cost < always-large cost, or routing loses money

3. Streaming

RAG & AI AgentsPage 4

Parallel Tools, Retrieval Caching, Timeouts and Fallbacks

1. Parallel Tools

Independent tool_use blocks in one turn run together
Use asyncio.gather; return all results in one message
Latency = slowest call, not the sum
Never parallelize dependent or write-then-read calls
Cap concurrency to avoid rate limits

python

async def run_all(calls, sem):
    async def one(c):
        async with sem:
            return await tool(c)
    return await asyncio.gather(*map(one, calls))

semaphore = concurrency cap

2. Retrieval Cache

	Exact-match	Semantic
Hit rate	low	high
Risk	none	similar but wrong result

Key = normalized query + filters + index version
Invalidate on index or document update; set a TTL
Never cache user-private results under a shared key

3. Timeouts and Fallbacks

RAG & AI AgentsPage 5

Tracing Agent Runs

1. Trace and Spans

Traceone agent run
LLM spanparent_id = trace
Tool spanparent_id = LLM
Retrieval spanparent_id = LLM

Span = one step; parent_id nests spans, so a loop reads as a tree
Per span: input, output, model, tokens, cost, latency, status, error
Pass one trace_id through every tool and service

2. Viewing and Slicing

python

with tracer.start_as_current_span("llm_call") as span:
    span.set_attribute("user_id", uid)
    span.set_attribute("prompt_version", "v7")
    span.set_attribute("cache_hit", False)

OpenTelemetry span with slicing metadata

View the timeline in OpenTelemetry, Langfuse or LangSmith
Find the slowest and costliest step
Slice by: user_id, prompt version, route chosen, cache hit or miss, fallback used

3. Common Mistakes

Logging raw PII or secrets in prompts
Sampling away failed runs
Tracing only the LLM call: tool latency stays invisible

Remember

Keep every failed run, redact inputs, and give each tool call its own span.

RAG & AI AgentsPage 6

Cost and Latency Cheat Sheet

1. Cost and Cache

Cost = in_tokens × in_price + out_tokens × out_price
Cache read ≈ 0.1× input price
Cache write ≈ 1.25× input price

Cache-friendly prompt order

1Tools
2System
3Stable context
4History
5New turn

Cache check

No timestamps up front. Verify cache_read_tokens > 0.

2. Speed and Routing

Lever	Rule
Routing	Rules first, then small-model cascade with escalation
Track	Escalation rate and quality per route
Streaming	Lowers TTFT only
Parallel tools	Independent calls cost the slowest call
Retrieval cache	Key = query + filters + index version

3. Resilience and Tracing

RAG & AI AgentsPage 7

Check yourself

An agent runs 10 steps, each adding about 1k tokens to the history. Roughly how many input tokens are billed in total?

About 55k, not 10k: step n resends n×1k tokens.
Input grows quadratically with steps, so cut history and cache the prefix.

Caching is on, but cache_read_tokens is always 0. The system prompt starts with Now: {timestamp}. What is wrong?

Caching needs an exact prefix, and the timestamp changes it on every request.
Move volatile text to the end, after the stable prefix.

Small model costs 1, large costs 10, and 95% of steps escalate. Does the cascade save money?

No: 1 + 0.95×10 = 10.5, above always-large at 10.
Small always runs, so a high escalation rate loses money.