DeepSeek V4.1 Flash Cuts KV Cache HBM by ~75% — What Changed

The headline benchmark tables miss the engineering bet. When DeepSeek GA'd V4.1 Flash on September 10, 2026, most coverage focused on Terminal-Bench scores and the retirement of V4-Pro. DeepSeek's own Hugging Face model card titles the release differently: "Pushing the Limits of KV Cache Compression." The number that matters for anyone running long agent loops is 890 bytes per token — roughly one-quarter the HBM DeepSeek-V4-Flash needed for its global KV cache, and about one-eighth the persistent SSD footprint.

That is not a post-training patch you swap in at inference time. It is four architectural changes trained into the model from scratch — the inference-memory story behind the same release explainx.ai tracked from beta through GA, and a different angle from Venice's zero-retention access path, which covers trust boundaries rather than what happens inside the GPU memory hierarchy.

TL;DR

Why KV cache — not FLOPs — is the agent bottleneck

Standard transformer inference has two phases. Prefill processes the entire prompt in parallel and builds the KV cache — the stored keys and values every future token will attend back to. Decode generates one token at a time, growing that cache with each step. For a coding agent that ingests a repo, runs tools, and accumulates a 200K-token scratchpad, the cache is often what exhausts memory before raw compute does.

The math is brutal at long context. If each token's global cache costs C bytes, a 500K-token conversation holds roughly 500,000 × C bytes of KV state before you count weights, activations, or batching overhead. On datacenter GPUs where HBM is the scarcest resource — the same constraint Stanford's memory-price history tracks across AI capex cycles — shaving C by 4× is the difference between serving 40 concurrent agent sessions per node versus 10.

DeepSeek's framing in the September 10 announcement is explicit: "Cache-hit charges often account for a large share of agent costs. Compressing the cache cuts those costs significantly." That is why the KV story belongs in the same conversation as the 60% cache-hit price cut — smaller caches lower DeepSeek's cost to retain prefixes, and part of that saving shows up on the rate card.

Four mechanisms — how 890 bytes per token is possible

DeepSeek did not pick one trick. The Hugging Face card describes a stack where each layer removes a different chunk of redundancy.

1. Causal Encoder-Decoder (CED)

V4.1 Flash reorganizes the 40-layer transformer as 20 encoder layers followed by 20 decoder layers. During prefill, only the encoder path runs — 8B active parameters instead of the 16B used during decode. The critical cache insight: the decoder's global KV cache is projected from the final encoder hidden states, not independently materialized at every decoder layer from decoder hidden states.

For input-heavy agent workloads — ingesting a large diff, a document bundle, or tool output before generating a short reply — you pay less compute on the way in and store a more compact cache representation on the way out. This replaces the monolithic MoE decoder layout behind V4-Flash and the bolt-on vision tower pattern from V4-Flash-Vision-Exp.

2. Compressed Sparse Attention 2 (CSA2)

CSA2 assigns each attention layer one of three static modes — Full, Reindex, or Reuse — to share main KV and indexer keys across layers and reuse Top-K sparse-attention indices. In the decoder, a Hierarchical Sparse Indexer restricts later indexing layers to a candidate pool built by the first Full-mode layer, bounding deeper indexer cost independently of context length.

Cross-layer sharing is the opposite of the default transformer assumption that every layer owns its own KV tensors. If ten layers can reuse one layer's cache and sparse index, you stop paying linear storage growth with depth. DeepSeek trained sparse attention at 64K sequence length before extending to 1M tokens at 34T tokens into pretraining — the sparsity pattern is baked into weights, not retrofitted the way Microsoft's sliding-window post-training paper warns against for linear-attention conversions.

3. FP4 main KV caching

The main global KV cache is stored in FP4 (E2M1 format) with one E4M3 scale per 16 channels. That is half the width of FP8 cache storage and a quarter of FP16 — with quantization noise absorbed during training rather than applied as a serving-time afterthought.

Operators who self-hosted V4-Flash-Vision weights may already quantize KV cache manually (q4_1 in llama.cpp extends context at the cost of quality drift). V4.1 Flash treats 4-bit cache as a first-class training target, which is a stronger claim — but your engine must implement the exact FP4 layout to realize the savings.

4. SWA Bounded Replay

Sliding-window attention (SWA) limits how far back each layer looks, but historically the SWA KV state still had to be persisted to SSD for long-running sessions. V4.1 Flash's SWA Bounded Replay reconstructs missing SWA KV by replaying only the most recent n_win tokens, avoiding persistent SWA storage entirely.

DeepSeek reports this alone cuts the persistent KV cache footprint to roughly 1/8 of V4-Flash — the SSD line in the launch graphics, separate from the 4× HBM reduction. For agents that pause and resume across hours, the difference is whether your inference cluster needs a large NVMe tier dedicated to cache spillover or can keep hot state in HBM and DRAM pools.

The numbers in context

DeepSeek publishes a generational chart on the Hugging Face card: V4.1 Flash at 890 bytes/token global KV cache, approximately 4× smaller than V4-Flash and 437× smaller than DeepSeek-V1. Figure 1(b) on the model card visualizes the curve — useful because it separates "we quantized harder" from "we redesigned what gets stored."

Worked example: At 890 bytes/token, a 1M-token context window holds roughly 890 MB of global KV cache per sequence — before batching, vision tokens, or Engram conditional memory (196B parameters accessed via sparse lookup, separate from backbone KV). At the implied V4-Flash rate (~3.5 KB/token), the same context would need ~3.5 GB of cache — often the difference between fitting on one H100-class node versus sharding across two.

DeepSeek also pairs the cache work with DSpark speculative decoding (semi-autoregressive draft generation with confidence-scheduled verification) on the same card. Throughput and memory are coupled: a smaller cache leaves more HBM headroom for draft models and larger batches.

What this means for what you build or pay

On DeepSeek's API

If your agent reuses long system prompts, tool schemas, or document prefixes, cache-hit pricing is where the compression lands in your invoice. DeepSeek cut cache-hit input to $0.003/M tokens off-peak at GA — down from $0.007/M on the prior Flash tier. That is a list-price change enabled partly by cheaper storage, not a magic discount on uncached tokens.

Production checklist for API users:

1. Structure prompts for prefix reuse — static system + tool definitions first, volatile user content last.
2. Re-test after September 14 — legacy deepseek-v4-pro routes to V4.1 Flash at Flash rates; cache behavior may differ from old Pro sessions.
3. Do not conflate cache-hit savings with uncached input — cache-miss off-peak input is $0.15/M on the direct API; the 75% HBM story helps DeepSeek's margin most when prefixes actually hit.

Self-hosting and community engines

Open weights are on Hugging Face under MIT license, with reference code in the inference/ folder. The compression story does not transfer automatically — generic MoE runners that ignore CSA2 layer modes or store FP16 KV will show much higher memory than DeepSeek's 890-byte figure.

Until vLLM and SGLang ship full V4.1 cache support, treat DeepSeek's byte count as a first-party API specification, not a guarantee for local llama.cpp experiments. The same caution Raschka's Kimi K3 architecture note applies: confirm MLA or FP4 KV paths are actually enabled in your stack, or you are paying for weights without the efficiency story.

For a hands-on mental model of how KV cache drives agent cost, the runnable ablations in bojieli's ai-agent-book chapter2/kv-cache remain the best open tutorial — V4.1 Flash is the production-scale version of problems that chapter measures in miniature.

What people are asking

"Is 75% HBM reduction the same as 75% cheaper inference?"

No. HBM savings reduce memory-bound costs — how many sessions fit per GPU, how much SSD tier you provision, how aggressively you can batch. Prefill on the encoder path still burns FLOPs; decode still grows cache linearly with output length. Total cost is memory plus compute plus whatever your provider charges for cache hits. The Groq LPX / Rubin prefill-decode split is a useful analog: different bottlenecks need different hardware.

"Does CSA2 mean the model 'forgets' most of the context?"

It means attention is sparse by design — layers in Reuse mode do not independently store full dense KV for every token. DeepSeek trained for this at 64K before extending to 1M, which is stronger than swapping dense attention for sliding-window at serving time. Still run your own needle-in-haystack and repo-order tests on agent scaffolds before trusting 1M context for production coding agents.

"How does this compare to Kimi K3 MLA?"

Kimi K3's MLA compresses KV by projecting into a latent space — one family of techniques. V4.1 Flash stacks CED projection, cross-layer sharing, 4-bit storage, and SWA replay. Neither makes the other's approach obsolete; they reflect different labs betting on different compression menus. Compare on bytes per token at your context length and quality on your eval suite, not architecture aesthetics.

"Should I migrate from V4-Flash-0731 for memory reasons alone?"

If you are API-only and cache-hit spend dominates, yes — re-test and migrate before V4-Flash aliases fully retire. If you self-host 0731 weights with a stable FP8 KV setup that already fits your hardware, wait for engine support and benchmark memory and quality on your workloads. The open-weight Vision-305B release and V4.1 Flash are different architectures; migration is a replatform, not a weight refresh.

Operator checklist

Before you claim V4.1 Flash memory savings in a capacity plan:

• Confirm your inference engine implements CED, CSA2 layer modes, FP4 KV layout, and SWA Bounded Replay — not just MoE routing.
• Measure bytes/token at runtime with your tokenizer and vision inputs; multimodal prompts add visual embeddings outside the 890-byte headline number.
• Separate HBM (hot cache) from SSD (persistent tier); the 8× SSD reduction matters for multi-hour agent sessions with pause/resume.
• Re-run long-context evals after every engine upgrade — silent fallback to dense FP16 KV is common when kernels lag new architectures.
• Budget Engram (196B conditional memory) separately from backbone KV; sparse lookup has its own bandwidth profile.

Related reading

• DeepSeek V4.1 Flash: A Two-Day Beta With a New Multimodal Architecture
• Venice Adds DeepSeek V4.1 Flash — Private Access Without the Direct API
• DeepSeek V4 Flash 0731 Scores 89% on ARC-AGI at $0.02/Task
• DeepSeek V4 Pro: Agent Coding Benchmarks and API Economics
• DeepSeek V4-Flash-Vision-Exp: Multimodal Agent Launch
• DeepSeek Opens Its 305B V4 Flash Vision Model — Free Weights
• DeepSeek V4 Prices Just Went Up — Does It Match GPT-5.6?
• Sliding-Window Attention Beats Linear Attention — Post-Training Only
• Official sources: DeepSeek V4.1 Flash launch · API release notes · Hugging Face model card · DeepSeek API pricing

KV cache byte counts, architecture names, and pricing reflect DeepSeek's September 10, 2026 GA materials. Community inference engines may not implement the full compression stack on day one — verify memory measurements on your serving path before capacity planning.