The KV cache is the hidden memory bill on every autoregressive LLM call. Weights load once; the cache grows with every token you keep in context and with every parallel chat. If a 7B quantized model fits in 6 GB but a 32K thread crashes, the cache is usually why.
Quick answers
| Question | Direct answer |
|---|---|
| What is stored? | Per-layer key and value activations for each prior token |
| When is it built? | During prefill on the prompt, then extended each decode step |
| What drives size? | Layers, hidden width, heads, dtype, sequence length, batch concurrency |
| Who documents tuning? | vLLM CacheConfig, Hugging Face inference guide |
| Local levers | mlx --max-kv-size, llama.cpp context, Ollama num_ctx |
| Server levers | vLLM gpu_memory_utilization, prefix caching, KV offload |
What the KV cache actually is
KV cache explained as attention key-value pairs kept in a side pocket during decode
Transformer decoders generate one token at a time. At step t, each attention layer must let the new token attend to all positions 1 … t−1. Without a cache, the stack would re-run the full prompt through every layer on every step — correct, but wasteful.
During prefill, the model processes the whole prompt in one or more forward passes and writes keys and values for each position into the cache. During decode, only the latest token gets fresh projections; earlier keys and values are read from GPU memory. Hugging Face documents this pattern as the motivation for a static KV cache when compiling models with torch.compile.
The cache is not part of the downloadable checkpoint. It is runtime state, like an expanding scratch pad. That is why context-window pricing and prompt caching economics treat long inputs differently from short ones: prefill work and cache residency both scale with prompt length.
How this differs from model weights
| Model weights | KV cache | |
|---|---|---|
| Loaded when | Server or app start | Each session as tokens arrive |
| Shrinks with Q4/Q8 | Yes | No (unless server quantizes KV) |
| Grows during chat | Fixed | Grows with assistant output too |
| Shared across users | Yes | Only with deliberate prefix reuse |
Our quantization guide notes that Q4 may free enough VRAM for weights while leaving little headroom for a fat cache — a common laptop failure mode.
How memory scales (back-of-envelope)
LLM KV cache memory growth sketched as a long context ribbon passing through a ring
Exact bytes depend on architecture (GQA, MLA, sliding window), but planners still use a proportional model:
KV bytes ≈ 2 × layers × sequence_length × kv_heads × head_dim × bytes_per_element × concurrent_sequences
The leading factor 2 is keys plus values. Sequence length includes system prompt, tools, retrieved chunks, and every assistant token already emitted — not just the user message you typed last.
Grouped-query attention (GQA) cuts kv_heads versus full multi-head attention, which is why many 2026 open models advertise long context without the KV footprint of older 70B stacks. Sliding-window layers cap effective history but add complexity: some positions never enter cache, yet runtimes must still track masks correctly.
When you compare vLLM versus SGLang, both engines fight the same physics; they differ in schedulers, prefix-hash policies, and optional disaggregated prefill/decode when one GPU cannot hold model plus cache for your SLA.
Prefill, decode, and where time goes
| Phase | Computes | KV cache |
|---|---|---|
| Prefill | All prompt tokens in parallel (often compute-bound) | Filled for every prompt position |
| Decode | One new token per step (often memory-bandwidth-bound) | Append one slot per layer per step |
Long prompts dominate prefill latency; long answers dominate decode time because each step touches the full cache. Speculative decoding speeds decode by drafting multiple tokens, but verified steps still update KV states — a topic we cover in our companion speculative decoding guide.
For benchmarking long-context faithfulness, synthetic suites like RULER stress retrieval across hundreds of thousands of tokens — the same regime where KV sizing decides whether the benchmark runs at all.
Prefix caching and provider prompt cache
Automatic prefix caching on servers reuses KV blocks when a new request shares an exact token prefix with an earlier one. vLLM's design doc hashes blocks from parent hash, block tokens, and optional salts for multi-tenant isolation. Enable it with enable_prefix_caching=True on the engine. APC helps multi-turn chats and repeated system prompts, not one-off unique prompts.
Cloud APIs expose a similar idea as prompt caching: Anthropic and OpenAI bill cached input segments at a discount because the provider avoids recomputing identical prefill. That is a billing and latency feature for clients; locally you implement the same idea with vLLM, SGLang, or llama.cpp server flags. Read prompt caching tradeoffs before assuming cache hits are free on the security side — shared prefix stores can be a side channel in hostile multi-tenant layouts, which is why vLLM documents optional cache_salt.
Tuning on local runtimes
mlx-lm on Apple Silicon
Our MLX guide walks install and server mode. When wired memory warnings appear, mlx-lm suggests lowering quant width, raising iogpu.wired_limit_mb carefully, or passing --max-kv-size to cap cache growth. That trades maximum context for stability on 16–24 GB unified machines.
llama.cpp and Ollama
llama.cpp exposes context via -c / n_ctx. Ollama sets context per model in the Modelfile or environment. If generation slows sharply after a long thread, check whether you are paging or hitting GPU offload limits — often KV plus partial weight offload exceed VRAM.
vLLM and production servers
High-bandwidth memory stack feeding a chip, standing in for KV cache on GPU HBM
vLLM CacheConfig lists the levers operators actually use:
gpu_memory_utilization— fraction of device memory reserved for weights plus KV poolblock_size— tokens per paged KV blockenable_prefix_caching— reuse hashed prefix blockskv_offloading_size— spill blocks to host RAM via the offloading connector (usage guide)
When requests preempt, vLLM logs that KV space ran out and may recompute sequences later (optimization docs). Fix preemption by more KV memory, fewer concurrent seqs, tensor parallel sharding (smaller per-GPU weights, more room for cache), or pipeline parallel layouts — same trade space discussed in our vLLM vs SGLang comparison.
Agent and RAG stacks: where KV shows up twice
Coding agents are KV-heavy products even when the model is small. A single Claude Code or Codex session may carry system rules, tool schemas, file excerpts, linter output, and prior turns. Each category is still in the active context window, so the cache tracks the whole bundle. Trimming tool definitions helps as much as trimming chat — see MCP tool description reliability for why fat tool lists hurt both selection and memory.
RAG adds another twist. Retrieved chunks land in the prompt, so prefill cost and KV footprint jump on every query. A reranker that returns eight 800-token passages can add more than six thousand tokens before the model answers. Our reranker latency guide is about milliseconds, but the same design choice changes how many slots your inference server must keep warm. Agents that “read the whole repo” without retrieval often hit context limits first; agents that retrieve aggressively hit KV limits when many users run at once.
Multi-tenant SaaS has a third pattern: one model, many isolated caches. GPU memory is shared, but KV blocks are per request unless prefix caching explicitly deduplicates identical system prompts. That is why inference vendors quote both model memory and max concurrent users — the second number is really a KV scheduling constraint.
What practitioners should do
Size hardware with KV, not weights alone. A card that loads Q4 13B is not automatically a 128K-context card.
Cap context in products. Agent harnesses that stuff entire repos into prompts (context engineering) blow cache faster than chat UIs. Prefer retrieval (reranker guide) over infinite history.
Separate prefill and decode when scaling. Disaggregated serving moves prefill-heavy work to a different pool than decode-heavy chat — relevant once a single node's KV pool becomes the bottleneck.
Watch concurrent agents. Four coding agents on one local server mean four caches unless you serialize. If you run Hermes Agent or OpenClaw on a single GPU box, set per-agent context caps in the harness before you chase a larger quant.
Measure before buying hardware. Log max_num_seqs preemptions in vLLM, watch mlx wired-memory warnings, or track OOMs during long codegen sessions. Those signals are almost always KV pressure, not a bad checkpoint. Top laptop open models lists realistic context per RAM tier with this constraint in mind.
FAQ recap
The KV cache is the reason identical weights feel “heavier” after a long session. Quantization slimmed the checkpoint; it did not remove per-token state. Prefix caching and provider prompt cache discounts attack repeated prefill, not unbounded assistant length. For decode acceleration without changing cache math, see speculative decoding.
KV cache behavior, vLLM flags, and Hugging Face static-cache notes are accurate as of November 8, 2026. Always check your runtime version — block sizes and offload specs change between releases.