explainx.ai350K+ learn from us
TrendingAI News TodayPathwaysSkills
Pricing
explainx.ai

Upskill in AI — 16 free pathways, live workshops & bootcamps, and 50+ courses from practitioners. Plus the skills, tools, and MCP servers to practice on.

follow us

follow on google

Add explainx.ai as a preferred source

corporate training

support@explainx.ai

get started

Find your pathTake Free Evaluation

community

Join the community

learn

mind: share how you thinkpathways — start freeworkshopsbootcampscoursescompare Explainxcertificationsmock testsexplainx universitycorporate traininglearn skills & mcp

discover

skillsmcp serversexplainx mcpestacktoolsmdx readeragentsllmsdesignsdictionarypeopleagi trackerfelony benchranks

company

aboutvisionmissionteaminstructorsteach on explainxpartnershipscommunityhackathonscareers

content

daily AI newsstate of AI — live resultsblogreleasespromptsgeneratorsresource libraryfor LLMsexplainx.ai kids

solutions

all solutionsdeveloper upskillingmarketing upskillingproduct manager upskillingleadership upskilling

newsletter · weekly

Get AI news, tools, and insights in your inbox.

supportcontactprivacytermsdata rightshow we create contentsubmission guidelines

© 2026 AISOLO Technologies Pvt Ltd

explainx.ai

On this page

  • TL;DR: what changes for builders
  • Why agent traffic looks unlike chat
  • Harness rules: protect the prefix
  • Engine rules: routing and memory
  • How this connects to your stack
  • Measuring prefix-cache health in production
  • When to restart the session instead of tuning
  • Checklist before you blame the model
  • Related reading
← Back to blog

explainx / blog

Together AI and Ollama: Prefix Cache Rules for Long Agent Sessions

Together AI, Ollama, Agent Infrastructure, Inference, Coding Agents

Part of Local AI

Together AI and Ollama explain November 16, 2026 how harnesses and inference engines must cooperate on prefix caching so coding agents stay fast past turn 90.

Nov 16, 2026·8 min read·Yash Thakker
add explainx.ai
go deep
Together AI and Ollama: Prefix Cache Rules for Long Agent Sessions

Together AI and Ollama published a joint guide on November 16, 2026 for builders running open models on long-horizon coding agents. The official post argues that prefix caching only works when harnesses keep prompt prefixes stable and inference engines route each turn back to GPUs that still hold the KV cache.

If you have wondered how does prefix caching work for agents that resend entire repos every step, this is the cross-team answer from the client side (Ollama) and the engine side (Together).

TL;DR: what changes for builders

table · 3 cols
LayerWhat to fixWhy it matters
HarnessStable system prompt; volatile facts in user messagesFirst changed token nukes cache for everything after it
ToolsLate-loaded tools at conversation end when templates allowTool list edits at the top replay the whole session
Tool outputTruncate with line ranges and totalsHuge file dumps bloat every future turn
Reasoning effortDo not toggle mid-run on GLM-style templatesEffort lives in token one of the system block
EngineSticky routing plus idle cache offloadWrong node equals full prefill replay
Daily briefFrom Yash · 350K+ students taught

Catch up on AI

One AI story explained in plain English, plus one thing to try. Every day, free. Unsubscribe anytime.

Why agent traffic looks unlike chat

Together and Ollama describe a traffic shape most chat UIs hide: large input sequence length (ISL) and small output sequence length (OSL). Each agent step ships the system prompt, tool definitions, user goal, every file read, and every command output back to the model. The model returns a short tool call, then the loop repeats.

On the GPU that means prefill dominates. Decode still matters for reasoning models that emit long thinking traces, but by turn fifty the bottleneck is chewing through history, not streaming tokens.

KV cache slots holding prior token states so agent turns reuse earlier prefill workKV cache slots holding prior token states so agent turns reuse earlier prefill work

Our KV cache explainer covers the same mechanics for self-hosted stacks. The Together post adds cross-request reuse: turn seven equals turn six plus a delta, so engines should not recompute turns one through six unless the harness broke the prefix.

The arithmetic in the post is deliberately idealized: ten equal turns without caching cost 55 units of prompt work; with caching, 10. At 100 turns the gap is 5,050 versus 100. Real prompts vary, but the scaling lesson is clear: uncached agent sessions grow roughly with the square of length.

Harness rules: protect the prefix

Keep the system prompt frozen

The easiest cache break is a system prompt that changes every turn: live timestamps, constantly refreshed git status, or a working directory string that updates after each cd. Move volatile state into the user turn once, not into the system template.

Session-unique metadata (user name, session start date) does not hurt within a session, but it prevents cross-session reuse of a warm setup cache at the engine. That is acceptable when you prioritize per-session speed.

Tool search and late definitions

Modern harnesses support tool search so agents pull definitions on demand instead of stuffing every tool into turn one. The cache question is where the definition lands.

Inserting into the top tool list changes the prefix before conversation history. On long jobs that can mean hundreds of thousands of tokens reprocessed. When the chat template allows, append new tool definitions at the conversation end.

Not every open model tolerates mid-conversation tool blocks. Template mismatches surface as API errors, not slow caches. Ollama maintainers note users should not need to know this; harness authors must.

Reasoning effort and GLM-5.3-Flash

Ollama reports a painful edge case: on GLM-5.3-Flash, reasoning effort is not a side channel. The template writes it into the first system block, before tools.

Raising effort after forty failed turns changes token one. The post recommends picking effort at start, compacting before changes, or starting a fresh session at the higher setting instead of paying a full replay.

Truncate tool output honestly

Oversized tool output hurts twice: it fills context, and it stays in every later turn even when cached. Harnesses should trim, but the post highlights a silent failure mode: returning lines 1–100 of 640 without saying the file continues.

The fix is explicit: tell the model it saw lines 1–100 of 640 and can search or request another range. That pattern pairs with Together agent eval lessons about agents over-trusting static examples.

Subagents: inherit versus brief

Open models such as Kimi K3, GLM-5.3, and GLM-5.3-Flash can spawn subagents. Inheriting the parent transcript maximizes prefix reuse for side threads. Sending a brief with only needed files minimizes memory on the engine.

Together warns inherit is not free: cached prefixes still consume GPU RAM for every generated token. A reviewer that needs two files should not drag three hundred parent turns.

Engine rules: routing and memory

Sticky routing and cache keys

A session cache lives on one node in one datacenter. Routing turn seven elsewhere forces a cold prefill and duplicates prefix storage.

Together describes two patterns:

  1. prompt_cache_key from the client so all turns hash to the same worker when possible.
  2. Longest-prefix hashing so a new session might warm-start from another user's identical system prompt and tool header.

That second pattern is why identical system prompts across users matter for SaaS hosts. Unique IDs in the system block guarantee cold starts.

Load-aware schedulers may skip the warm node if it is saturated. Perfect harness hygiene still sees cache misses when the fleet is hot.

Idle sessions and PagedAttention

Agents pause while tests or builds run. Engines can offload KV to CPU during idle windows instead of evicting and replaying entire transcripts on resume.

Together ties this to PagedAttention and continuous batching: memory blocks grow as needed; batches reshuffle as subagents complete. For operators, headroom is often the limiter, not FLOPs.

Agent-heavy fleets sometimes split prefill and decode clusters, shipping KV over RDMA. Long agent prefills then do not starve decode for short chat users.

GPU cluster layout suggesting how inference engines schedule agent prefill workloadsGPU cluster layout suggesting how inference engines schedule agent prefill workloads

How this connects to your stack

If you self-host, compare vLLM versus SGLang for RadixAttention and prefix caching switches. If you buy hosted inference, ask vendors how sticky routing works for OpenAI-compatible agent clients.

Ollama's ollama launch path targets builders who want Claude Code, Codex, or Pi on open weights without abandoning familiar harnesses. Together's Link announcement and Compiler posts sit on the same November timeline: speed per token and agent-ready APIs are both production problems.

Measuring prefix-cache health in production

Treat prefix caching like any other SLO: you need traces, not vibes.

Latency shape. On a healthy cache, turn latency should grow sublinearly as transcript length grows. If p95 latency doubles every few turns, you are likely replaying full prefills. Log time-to-first-token separate from decode to see which phase moved.

Token accounting. Hosted APIs sometimes bill cached input cheaper than fresh input. Compare billed prompt tokens against logical transcript length. A cliff where billed tokens equal full history every turn means cache misses.

Routing metadata. If your provider exposes cache hit headers or server id, chart them per session id. Agents that fork subagents across keys should still reuse parent prefix where possible.

A/B harness changes. When you ship a new tool schema or system prompt, expect a one-time cold start. Roll out behind a flag and watch cost per resolved task, not cost per request.

OpenAI and Anthropic document prompt caching for API users; vLLM documents enable_prefix_caching. Those features help only when your harness keeps bytes stable. Our Confucius4 stable-prefix streaming ASR post shows the same prefix stability idea in voice agents, and Sarvam Token Factory markets prefix cache as a latency lever for Indian builders.

When to restart the session instead of tuning

Together and Ollama imply a practical rule: if you must change reasoning effort, tool headers, or system instructions mid-run, restarting may be cheaper than rebuilding a ninety-turn cache.

Compaction helps when your harness supports it: drop stale tool output while preserving decisions and open tasks. Compaction is its own engineering problem because sloppy summaries break caches too if they rewrite earlier assistant turns.

For CI agents that run hours, schedule checkpoint commits plus fresh sessions per milestone rather than one immortal thread.

Checklist before you blame the model

  1. Log the first differing token between turns; if it is in the system block, fix the harness.
  2. Confirm your provider honors prompt_cache_key or equivalent sticky routing.
  3. Cap tool output with range metadata, not blind truncation.
  4. Benchmark turn 30, 60, and 90 latency; quadratic replay shows up as a cliff, not gradual drift.
  5. When changing reasoning effort or tool headers, assume cache cold start and budget tokens accordingly.

Related reading

  • Together AI agent evals: 75% to 97% completion
  • What is KV cache in LLM inference?
  • vLLM vs SGLang for prefix caching and rollouts
  • Prefix Sliding: Stanford method for faster reasoning
  • Together Compiler and agent-optimized inference
  • HybridAgent latent reasoning with Together and Stanford
  • Qwen Code open-source terminal coding agent

Sources: Together AI and Ollama post, Ollama documentation.

Spotted something out of date? Let us know.
Daily briefFrom Yash · 350K+ students taught

Catch up on AI

One AI story explained in plain English, plus one thing to try. Every day, free. Unsubscribe anytime.

  • ✓ Free
  • ✓ One email a day, no spam
  • ✓ Unsubscribe anytime
Yash Thakker

Written by

Yash Thakker

Yash is an AI expert with over 300K learners. Join his workshops →

View Yash Thakker in People in AI →

Related posts

Nov 16, 2026

Together AI Agent Evals: How Docs and Skills Lifted Success From 75% to 97%

On November 16, 2026 Together AI described how it evaluates coding agents against its own platform. Running real prompts in sandboxes with MCP and skills configs exposed documentation traps and lifted task completion from 75% to 97%.

Nov 10, 2026

Naive-N0.5-Flash: 2,100 tok/s Open Model and Latent Space Coding Test

NaiveAI markets Naive-N0.5-Flash as an open 309B MoE coding model with NaiveRT inference up to roughly 2,000 tokens per second. On November 10, 2026 Latent Space published an unbiased hands-on versus DeepSeek V4.1 Flash 1017 and GLM 5.3 Flash. We separate NaiveAI's peak-speed claims from end-to-end latency and app quality.

Oct 27, 2026

Offline Coding Agent on a Laptop: OpenCode, Ollama and Qwen3.5 9B With the Wi-Fi Off

Rittika Jindal, a principal engineer living where power and network outages are routine, tested whether a coding agent can work with the internet switched off. Her answer on a Mac laptop with OpenCode, Ollama and Qwen3.5 9B: yes for a small bug fix, with caveats. Here is the setup and what it does and does not prove.