ReplaySSM: NVIDIA and Tri Dao Cache SSM Inputs, Not State, for Faster Hybrid Decoding

NVIDIA and Tri Dao's Dao AI Lab at Princeton have published ReplaySSM, a way to serve hybrid models by caching the inputs of a state-space-model (SSM) layer instead of constantly rewriting its state. According to the NVIDIA technical blog post dated October 12, 2026, the change speeds SSM decode kernels by 1.46x to 1.80x at batch size 256, lifts end-to-end decoding by 1.20x to 1.47x, and makes speculative decoding on hybrid models roughly 2x faster.

If you run or fine-tune hybrid architectures such as Nemotron 3 or Qwen3.5, this is the kind of serving-engine change that eventually shows up as lower cost per token without touching the model. The paper is "ReplaySSM: Cache SSM Inputs, Not State" (arXiv 2610.13778), and the code lives at github.com/Johnny-Liou/ReplaySSM.

TL;DR: the questions people will ask

Why hybrid models have an SSM bottleneck

Hybrid models interleave a majority of SSM layers (Mamba-2, Gated DeltaNet, Kimi Delta Attention) with a few attention layers. The appeal is that SSM layers carry a fixed-size state instead of a KV cache that grows with context. NVIDIA names Nemotron 3, the Qwen3.5 to Qwen3.8 line, Kimi-K3 and GLM-5.3-Flash as hybrids in the wild. We have followed several of them, including Nemotron 3 Ultra, Nemotron 3.5 Lightning and the Qwen3.8 family.

SSM state flowing like a river through each decode step, the bottleneck ReplaySSM targets

The NVIDIA post argues that at realistic serving batch sizes the SSM layers, not attention, are often the decoding bottleneck for contexts up to about 100K tokens. It lists three reasons.

1. State memory traffic. Every decode step loads the full recurrent state, applies a small update, and stores it again. That is lots of bytes moved for very little arithmetic, so the layer is bound by HBM bandwidth. NVIDIA notes that one sequence's FP32 state for an SSM layer is as large as an attention KV cache for about 8K tokens.

2. Speculative decoding rollback. In attention, rejecting a draft token is cheap because you just move a pointer back. An SSM state has already been overwritten, so engines like vLLM store one state snapshot per draft token. That limits concurrency and means speculative decoding helps hybrids much less than it helps pure transformers. Our guide to DeepSeek's speculative decoding work covers why the technique matters so much for cost.

3. Quantization error. A KV cache is read but never fed back into itself, so it tolerates low precision. A quantized SSM state feeds the recurrence, so rounding error compounds over thousands of steps. That is why aggressive state quantization has been risky.

How ReplaySSM works

The core idea is to represent the state as a checkpoint plus a buffer of recent inputs, as described in the NVIDIA post.

• Non-flush steps. New inputs are appended to the buffer. The layer output is computed from the checkpoint plus the buffer. The full state is not written back.
• Flush steps. Once the buffer of L records fills, the buffered inputs are folded into the checkpoint, which is written to HBM once, and the buffer is cleared.
• Output-only evaluation. To verify several draft tokens, outputs for all of them come from a few small matrix products over cached keys, values and draft queries, without building intermediate states.
• Cheap rollback. Draft inputs go into the buffer and an accepted-prefix pointer advances. Rejected records are ignored and later overwritten, so no per-token snapshots exist.
• Separate precisions. A high-precision checkpoint drives the recurrence, while a low-precision copy serves reads. Because the low-precision copy never feeds the recurrence, its rounding error does not accumulate.

The method is described for Mamba-2, with buffered records of the decay term, value and key, and the authors say it also applies to Gated DeltaNet and Kimi Delta Attention because they share an outer-product update form. Performance runs in the post used an FP32 state and a window of L=16.

Speculative decoding in ReplaySSM, with a short fast draft arrow ahead of a longer verification arrow

The numbers NVIDIA reports

All of these come from NVIDIA's own post and paper, so treat them as vendor-reported until third parties reproduce them.

The models tested were Nemotron-3-Nano-4B (Mamba-2, one H100), Nemotron-3-Super-120B-A12B (Mamba-2, NVFP4 weights, one B300), Qwen3.5-4B (Gated DeltaNet, one H100) and Qwen3.5-122B-A10B (NVFP4, one B300), each on a single GPU without tensor parallelism.

Accuracy: what separate precisions buy

The GPQA-Diamond table in the post is the part to read closely, because state quantization is where hybrids usually lose quality.

A single shared NVFP4 state is damaging, notably a 31-point drop on the small Nemotron model. Keeping an FP32 or FP16 recurrence while serving low-precision reads brings three of the four models to within roughly 2 points of FP32. The smallest Nemotron still loses 5.6 to 6.7 points, so smaller hybrids are the most sensitive. The post also reports smaller losses on MMLU-Pro and Math-L5 and a long-input RULER test in the paper.

Where the gains do not show up

Be realistic about scope. The speedups are a serving-throughput story. At batch size 1, the post says gains are negligible, so a single local chat session will not feel faster. The benefit appears when many sequences share a GPU and HBM bandwidth is the limit, which is the regime of API providers and inference platforms. If you run models locally, such as in our Mac Studio local AI diary, expect the benefit later and mostly through engine updates.

It is also specific to SSM-style layers. Pure transformers gain nothing, and hybrids with very few SSM layers gain less. The Nemotron-3-Super model has 40 SSM layers and 8 attention layers per the post, while Qwen3.5-122B has 36 and 12, so both are SSM-heavy.

Open questions and caveats

• Production claims are mixed. The post says GLM-5.3-Flash serves with ReplaySSM, which fits our earlier coverage of GLM-5.3 inference infrastructure. For Kimi-K3, the post's wording is inconsistent: one passage describes production use, another cites ReplaySSM as a concurrent approach. We would not repeat the Kimi claim until it is clarified.
• Integrations are pull requests. The post lists TensorRT-LLM PRs 14203 and 16464, vLLM PRs 48018, 52506 and 52928, and FlashInfer PRs 3975 and 4081, and mentions SGLang. Open PRs are not releases. Check the merge status before planning around it.
• Vendor benchmarks. Comparisons are against vLLM kernels that the authors tuned. Independent numbers on other GPUs and workloads will tell us how general the speedup is.
• Window size trade-off. Larger L means fewer checkpoint writes but a bigger buffer to read on each step. The headline numbers use L=16.

What this means for builders

1. Hybrid models get cheaper to serve. If you buy tokens from a provider that runs Nemotron, Qwen3.5 or GLM-5.3-Flash, engine-level gains like this are what let prices fall, as seen in other inference work such as NVIDIA Dynamo's shadow engine.
2. Speculative decoding becomes viable for SSMs. Hybrids were handicapped here. Cheap rollback removes a reason to prefer pure transformers when latency matters.
3. Low-precision state is not off the table. The separate-precision trick suggests NVFP4 state can work if the recurrence stays high precision, which matters for fitting more concurrent sequences per GPU.
4. Watch the merges. If you operate vLLM or TensorRT-LLM, follow the listed PRs and re-benchmark on your batch sizes. The result at your concurrency, not the headline number, decides whether to upgrade.

For a broader sense of how far non-transformer sequence models go, see the Mamba on a BBC Micro experiment and our look at Cerebras serving Qwen3.8 27B at 1,500 tokens per second.

Related reading

• Nemotron 3 Ultra 550B open-weight MoE
• Nemotron 3.5 Lightning 30B-A3B
• DeepSeek speculative decoding guide
• GLM-5.3 inference infrastructure
• NVIDIA Dynamo shadow engine
• Qwen3.8 Max open weights
• Primary sources: NVIDIA technical blog, ReplaySSM code

Figures and pull-request numbers are accurate as of October 12, 2026 and come from NVIDIA's post; they may change as the integrations land.