Serving Agents and Reasoning Models
Agents make dozens of calls per task, and reasoning models write thousands of hidden tokens. What does that do to serving?
The situation
Your coding agent's per-request latency looks fine on the dashboard, but tasks take minutes and cost far more than planned. Requests have also got longer since you switched to a reasoning model. The interviewer asks:
Agents make dozens of calls per task, and reasoning models write thousands of hidden tokens. What does that do to serving?
Why it matters
Agentic and reasoning workloads are growing fast, and they break assumptions built for chat. A request may be cheap while the task is expensive. This lesson connects prefix caching, KV offloading and serving metrics to these workloads. For the agent loop itself, see What Is an Agent?
The idea
Think of a researcher who, before every step, rereads all their notes so far, then makes a phone call to check something, then writes a little more. If the notes stay open on the desk, rereading is quick. If someone tidies them away during every call, each step starts with a long trip to the archive.
Play
Try to break it. Let the cache expire during tool calls. Then raise the steps, then the hidden reasoning.
Price an agent task
Per task, not per requestIllustrative: a 5,000-token system prompt and tools, 200 visible output tokens per step, $3 per million input tokens, $0.30 per million cache reads, $3.75 per million cache writes, $15 per million output tokens, and 50 output tokens per second. Hidden reasoning is billed as output, and assumed to be dropped from the history, as many APIs do. Not any provider's real price.
A 20-step task reads about 423,000 input tokens, as the history is resent every step. With a warm cache, 91% of them are cheap cache reads, and the task costs about $0.47. If the cache expires at every tool call, the same task costs about $1.80, about 3.9 times as much. Now raise the steps to 40: input grows to 1,526,000 tokens, and the gap widens to $1.12 against $6.14. Then raise hidden reasoning to 4,000 tokens per step: cost rises to $1.52, and generation alone takes about 28 minutes, which is why reasoning models are planned around decode.
Now say why: why does an expired cache cost so much more at step 20 than at step 2?
How it works
In one breath
A chat question is one request. An agent task is a chain of many, each resending the growing history, with pauses while tools run. A reasoning model writes long hidden reasoning before answering, so it's dominated by slow, memory-bound decode. Both make the KV cache and its reuse central. Cache the shared history so each turn only processes new tokens, keep caches alive across tool calls, route each task back to the machine that holds its cache, and judge the system on cost and time per task, not per request.What changes
Reasoning models are decode-heavy
Every hidden reasoning token is generated one at a time, just like the answer. So a model that thinks for 5,000 tokens spends most of its time in memory-bound decode, and those tokens are usually billed as output. Ways to control this include thinking budgets, routing easy questions to a non-thinking mode, and speculative decoding for faster decode.
Agents resend their history
Each step of an agent sends the system prompt, the tools and the whole history so far, plus the latest tool result. Without caching, the work grows with the square of the steps. With caching, each step only processes its new tokens, and cached input is billed at a fraction of the price.
Keeping caches warm
| Problem | Fix |
|---|---|
| A slow tool call outlasts the cache's lifetime | Use a longer cache lifetime, or offload the cache rather than evicting it |
| Replicas each hold different caches | Route a task back to the same replica, or use a shared cache tier |
| Eviction drops agents' long histories | Keep shared prefixes and recent turns, and evict the middle |
| Editing or trimming early history breaks the match | Append rather than edit, or accept a re-prefill when you summarise |
Provider prices show why it matters. As of November 2026, Anthropic bills cache reads at 0.1× the input price, and less for its newest models, with writes at 1.25× for a 5-minute cache or 2× for an hour. OpenAI's newest models bill writes at 1.25× for a 30-minute default cache, with discounts of up to 95%. Check current pricing, as it changes often.
What research reports
- Smarter eviction: keeping shared prefixes and recent turns cut average time to first token by up to 60% against SGLang, in one 2026 study of agent workloads.
- Load-aware reuse: requests in one queue can differ by up to 100× in uncached work. A scheduler aware of this reported p99 time to first token up to 8.6× lower than earlier prefix-aware schedulers.
- Across models: agents and cascades move requests between models. A 2026 system reuses caches across models, rather than within one only.
The maths
Our own arithmetic for an agent that appends to its history, if every step hits the cache.
In practice
In practice
- Measure per task: track cost, steps, cache hit rate and wall-clock time per completed task, as well as per-request latency.
- Keep history append-only: stable tools first, then history, then new results. Avoid edits early in the prompt.
- Trim tool output: a 5,000-token tool result is resent at every later step. Summarise or truncate it.
- Budget reasoning: cap thinking where tasks allow, and send simple steps to a cheaper or non-thinking model.
- Separate batch from interactive: long-running background agents can use cheaper capacity and looser latency targets.
The answer that sounds right
"Each request is fast, so the agent is fast"
It sounds right, since the requests are what you monitor. But a task may make dozens of them, each resending the history, and broken caching or extra steps add up. Measure cost, steps and time per completed task as well.
Say it
Agents make dozens of calls per task, and reasoning models write thousands of hidden tokens. What does that do to serving?
Say your answer out loud
Aim for 30 seconds. It feels different from reading it, and that's the point.
Tick what you covered
The interviewer pushes further
Follow-up 1 of 3
An agent's cost jumped after a tool started taking 10 minutes. Why?
Answer it out loud, then check.
The cache probably expired during the wait, so the next step had to re-read the whole history as fresh input, and pay to write the cache again. Use a longer cache lifetime, or offload rather than evict, and check the cache hit rate per step.
Follow-up 2 of 3
Why measure per task rather than per request?
Answer it out loud, then check.
An agent's requests are cheap individually, but a task may make dozens of them. A change that makes each request faster can still make the task slower, or more expensive, if it adds steps or breaks caching.
Follow-up 3 of 3
How do reasoning models change capacity planning?
Answer it out loud, then check.
They generate far more output per request, so decode dominates time and GPU use. Plan on output tokens per second, and on concurrency held for longer, and consider separate pools or budgets for long thinking.
Agents and reasoning models change what dominates. A reasoning model writes thousands of hidden tokens, so it's mostly memory-bound decode, and I'd control that with thinking budgets, routing simple steps to cheaper modes, and speculative decoding. An agent resends its growing history at every step, so without caching, input grows with the square of the steps. With prefix caching, each step only processes new tokens, and cache reads are billed at a fraction of the price. To keep caches warm, I'd use longer cache lifetimes or offloading across slow tool calls, route each task back to the replica holding its cache, keep history append-only, and trim tool output. And I'd measure cost and time per task, not per request.
Key terms and sources
Why reasoning models are decode-heavy, how agents resend growing histories, caching across turns and tool calls, cache lifetimes, routing back to warm caches, and cost per task.
Sources
- Rethinking KV Cache Eviction for Agentic LLM Serving
- Fairway: Load-Aware KV Cache Reuse for LLM Prefill Scheduling
- CacheBridge: Cross-Model KV Transfer for Multi-Model LLM Serving
- Anthropic, Prompt caching
- OpenAI, Prompt caching
- Liu et al., LMCache
Explanations are ours, built from these sources. Lab numbers are illustrative unless a lab says otherwise. Many recent papers are preprints, and vendor results are their own.