Together AI and Ollama published a joint guide on November 16, 2026 for builders running open models on long-horizon coding agents. The official post argues that prefix caching only works when harnesses keep prompt prefixes stable and inference engines route each turn back to GPUs that still hold the KV cache.
If you have wondered how does prefix caching work for agents that resend entire repos every step, this is the cross-team answer from the client side (Ollama) and the engine side (Together).
TL;DR: what changes for builders
| Layer | What to fix | Why it matters |
|---|---|---|
| Harness | Stable system prompt; volatile facts in user messages | First changed token nukes cache for everything after it |
| Tools | Late-loaded tools at conversation end when templates allow | Tool list edits at the top replay the whole session |
| Tool output | Truncate with line ranges and totals | Huge file dumps bloat every future turn |
| Reasoning effort | Do not toggle mid-run on GLM-style templates | Effort lives in token one of the system block |
| Engine | Sticky routing plus idle cache offload | Wrong node equals full prefill replay |
Why agent traffic looks unlike chat
Together and Ollama describe a traffic shape most chat UIs hide: large input sequence length (ISL) and small output sequence length (OSL). Each agent step ships the system prompt, tool definitions, user goal, every file read, and every command output back to the model. The model returns a short tool call, then the loop repeats.
On the GPU that means prefill dominates. Decode still matters for reasoning models that emit long thinking traces, but by turn fifty the bottleneck is chewing through history, not streaming tokens.
KV cache slots holding prior token states so agent turns reuse earlier prefill work
Our KV cache explainer covers the same mechanics for self-hosted stacks. The Together post adds cross-request reuse: turn seven equals turn six plus a delta, so engines should not recompute turns one through six unless the harness broke the prefix.
The arithmetic in the post is deliberately idealized: ten equal turns without caching cost 55 units of prompt work; with caching, 10. At 100 turns the gap is 5,050 versus 100. Real prompts vary, but the scaling lesson is clear: uncached agent sessions grow roughly with the square of length.
Harness rules: protect the prefix
Keep the system prompt frozen
The easiest cache break is a system prompt that changes every turn: live timestamps, constantly refreshed git status, or a working directory string that updates after each cd. Move volatile state into the user turn once, not into the system template.
Session-unique metadata (user name, session start date) does not hurt within a session, but it prevents cross-session reuse of a warm setup cache at the engine. That is acceptable when you prioritize per-session speed.
Tool search and late definitions
Modern harnesses support tool search so agents pull definitions on demand instead of stuffing every tool into turn one. The cache question is where the definition lands.
Inserting into the top tool list changes the prefix before conversation history. On long jobs that can mean hundreds of thousands of tokens reprocessed. When the chat template allows, append new tool definitions at the conversation end.
Not every open model tolerates mid-conversation tool blocks. Template mismatches surface as API errors, not slow caches. Ollama maintainers note users should not need to know this; harness authors must.
Reasoning effort and GLM-5.3-Flash
Ollama reports a painful edge case: on GLM-5.3-Flash, reasoning effort is not a side channel. The template writes it into the first system block, before tools.
Raising effort after forty failed turns changes token one. The post recommends picking effort at start, compacting before changes, or starting a fresh session at the higher setting instead of paying a full replay.
Truncate tool output honestly
Oversized tool output hurts twice: it fills context, and it stays in every later turn even when cached. Harnesses should trim, but the post highlights a silent failure mode: returning lines 1–100 of 640 without saying the file continues.
The fix is explicit: tell the model it saw lines 1–100 of 640 and can search or request another range. That pattern pairs with Together agent eval lessons about agents over-trusting static examples.
Subagents: inherit versus brief
Open models such as Kimi K3, GLM-5.3, and GLM-5.3-Flash can spawn subagents. Inheriting the parent transcript maximizes prefix reuse for side threads. Sending a brief with only needed files minimizes memory on the engine.
Together warns inherit is not free: cached prefixes still consume GPU RAM for every generated token. A reviewer that needs two files should not drag three hundred parent turns.
Engine rules: routing and memory
Sticky routing and cache keys
A session cache lives on one node in one datacenter. Routing turn seven elsewhere forces a cold prefill and duplicates prefix storage.
Together describes two patterns:
- prompt_cache_key from the client so all turns hash to the same worker when possible.
- Longest-prefix hashing so a new session might warm-start from another user's identical system prompt and tool header.
That second pattern is why identical system prompts across users matter for SaaS hosts. Unique IDs in the system block guarantee cold starts.
Load-aware schedulers may skip the warm node if it is saturated. Perfect harness hygiene still sees cache misses when the fleet is hot.
Idle sessions and PagedAttention
Agents pause while tests or builds run. Engines can offload KV to CPU during idle windows instead of evicting and replaying entire transcripts on resume.
Together ties this to PagedAttention and continuous batching: memory blocks grow as needed; batches reshuffle as subagents complete. For operators, headroom is often the limiter, not FLOPs.
Agent-heavy fleets sometimes split prefill and decode clusters, shipping KV over RDMA. Long agent prefills then do not starve decode for short chat users.
GPU cluster layout suggesting how inference engines schedule agent prefill workloads
How this connects to your stack
If you self-host, compare vLLM versus SGLang for RadixAttention and prefix caching switches. If you buy hosted inference, ask vendors how sticky routing works for OpenAI-compatible agent clients.
Ollama's ollama launch path targets builders who want Claude Code, Codex, or Pi on open weights without abandoning familiar harnesses. Together's Link announcement and Compiler posts sit on the same November timeline: speed per token and agent-ready APIs are both production problems.
Measuring prefix-cache health in production
Treat prefix caching like any other SLO: you need traces, not vibes.
Latency shape. On a healthy cache, turn latency should grow sublinearly as transcript length grows. If p95 latency doubles every few turns, you are likely replaying full prefills. Log time-to-first-token separate from decode to see which phase moved.
Token accounting. Hosted APIs sometimes bill cached input cheaper than fresh input. Compare billed prompt tokens against logical transcript length. A cliff where billed tokens equal full history every turn means cache misses.
Routing metadata. If your provider exposes cache hit headers or server id, chart them per session id. Agents that fork subagents across keys should still reuse parent prefix where possible.
A/B harness changes. When you ship a new tool schema or system prompt, expect a one-time cold start. Roll out behind a flag and watch cost per resolved task, not cost per request.
OpenAI and Anthropic document prompt caching for API users; vLLM documents enable_prefix_caching. Those features help only when your harness keeps bytes stable. Our Confucius4 stable-prefix streaming ASR post shows the same prefix stability idea in voice agents, and Sarvam Token Factory markets prefix cache as a latency lever for Indian builders.
When to restart the session instead of tuning
Together and Ollama imply a practical rule: if you must change reasoning effort, tool headers, or system instructions mid-run, restarting may be cheaper than rebuilding a ninety-turn cache.
Compaction helps when your harness supports it: drop stale tool output while preserving decisions and open tasks. Compaction is its own engineering problem because sloppy summaries break caches too if they rewrite earlier assistant turns.
For CI agents that run hours, schedule checkpoint commits plus fresh sessions per milestone rather than one immortal thread.
Checklist before you blame the model
- Log the first differing token between turns; if it is in the system block, fix the harness.
- Confirm your provider honors prompt_cache_key or equivalent sticky routing.
- Cap tool output with range metadata, not blind truncation.
- Benchmark turn 30, 60, and 90 latency; quadratic replay shows up as a cliff, not gradual drift.
- When changing reasoning effort or tool headers, assume cache cold start and budget tokens accordingly.
Related reading
- Together AI agent evals: 75% to 97% completion
- What is KV cache in LLM inference?
- vLLM vs SGLang for prefix caching and rollouts
- Prefix Sliding: Stanford method for faster reasoning
- Together Compiler and agent-optimized inference
- HybridAgent latent reasoning with Together and Stanford
- Qwen Code open-source terminal coding agent
Sources: Together AI and Ollama post, Ollama documentation.
