NVIDIA says the same 288 GB of memory per GPU can now do up to 30 times the work per megawatt. In a technical blog published November 5, 2026, three NVIDIA authors argue that AI inference is no longer limited by how much memory a GPU has, but by how much useful work each byte delivers. They name five levers: a 4-bit number format, wide expert parallelism, a serving layer, a new CPU and a shared storage tier.
Every number below is NVIDIA's own unless we say otherwise. We found no independent replication in the sources we read. The goal of this post is to explain what each lever does, so you can judge the claim.
TL;DR: the five levers and NVIDIA's claims
| Lever | What it does | NVIDIA's stated result |
|---|---|---|
| NVFP4 | 4-bit values with block and tensor scaling | About 3.5x smaller weights than FP16, 1.8x than FP8; up to 4x throughput of FP8 on tested benchmarks |
| NVFP4 KV cache | Smaller cached attention data | Up to 50% smaller than FP8, about 2x more context on device |
| Wide expert parallelism | Spreads MoE experts over 8+ GPUs | More HBM per GPU for cache, higher arithmetic intensity |
| Dynamo | KV-aware routing, split prefill and decode, tiered cache | Turns recomputation into cache hits |
| Vera CPU | 88 cores, up to 1.5 TB LPDDR5X, 1.8 TB/s link to Rubin | Parks idle KV cache during tool calls |
| CMX | Shared pod-level flash tier for KV cache | Up to 5x tokens per second and power efficiency vs traditional storage |
| Combined | All five together | Up to 30x throughput per MW, Vera Rubin NVL72 vs GB300 NVL72 |
Why does memory, not compute, set the limit?
High bandwidth memory, HBM, is the fastest and most expensive memory in an AI server, and every byte is contested. The blog says capacity stayed at 288 GB per GPU from GB300 to Rubin while bandwidth rose to 19.2 TB/s, yet agentic throughput per megawatt grew far faster.
The 30x figure is measured on SemiAnalysis AgentX, which NVIDIA describes as replaying recorded coding-agent sessions with real tool-call gaps and growing context. The comparison is Vera Rubin NVL72 against GB300 NVL72 at 160 tokens per second per user. That is a specific workload at a specific interactivity level. Different workloads can show different ratios.
For earlier numbers on Rubin, see our coverage of Rubin NVL72 throughput and cost per SemiAnalysis and the September AI Infra Summit preview.
Lever 1: how does NVFP4 save bytes?
NVFP4 stores each value as a 4-bit E2M1 element. Every micro-block of 16 values shares an FP8 scale factor, and a per-tensor FP32 scale sits on top. The two levels of scaling are meant to keep accuracy where simpler 4-bit formats lose it. NVIDIA introduced the format in an earlier post, which reported 1% or less accuracy loss on key language tasks for DeepSeek-R1-0528 when quantized from FP8 with post-training quantization.
Applied to weights, the new blog says NVFP4 cuts footprint about 3.5x versus FP16 and 1.8x versus FP8. It says NVFP4 is the default precision for open agentic benchmarks with strict accuracy requirements, including AA-AgentPerf, SemiAnalysis AgentX and MLPerf Endpoints, and delivers up to 4x the throughput of an FP8 baseline at the same interactivity. One chart shows 4x throughput per megawatt over FP8 for DeepSeek R1 on GB300.
The same idea applies to the KV cache. NVIDIA says an NVFP4 KV cache is up to 50% smaller than FP8, so about twice the context stays resident, and keeps over 99% of BF16 baseline scores across 10 DeepSeek-V4-Pro benchmarks. For a local-model view of NVFP4, see our Qwen3 FP4 on RTX PRO 6000 and DGX Spark post.
Lever 2: why spread experts across many GPUs?
Mixture-of-experts models activate only some parameters per token, but every expert's weights must sit in HBM somewhere. Packed onto few GPUs, weights eat memory and each decode step pays a heavy loading cost.
Wide expert parallelism in TensorRT-LLM spreads experts across eight or more GPUs. NVIDIA's example: DeepSeek-R1 at EP64 puts four experts per layer on each GPU. That leaves more HBM for cache and batch size, raises arithmetic intensity by reusing each loaded weight across more tokens, and lets a load balancer move hot and cold experts between passes.
It works only with a fast link. On GB200 NVL72, NVIDIA cites 130 TB/s of aggregate NVLink bandwidth, and says large expert parallelism (EP32) gives up to 1.8x more output tokens per second per GPU than small (EP8) at 100 tokens per second per user. Vera Rubin NVL72 gives each GPU up to 288 GB of HBM4 and 3.0 TB/s of NVLink 6 bandwidth, roughly 20.7 TB of HBM4 across the rack, per the blog.
Lever 3: how does Dynamo stop recomputation?
KV cache is only valuable if reused. Agents reuse system prompts, tool schemas, shared documents and growing task history. If a request lands on a GPU without its prefix, the system pays for prefill again.
NVIDIA Dynamo treats KV cache as a fleet-wide resource through three parts:
- KV-aware routing, which tracks where blocks live and weighs prefix overlap against worker load.
- Disaggregated serving, which runs prefill and decode on separate GPU pools.
- KV Cache Runner, which tracks blocks across GPU, pinned CPU and storage tiers and restores a preserved prefix when needed, using the NIXL transfer library.
NVIDIA also reports vendor tests: VAST Data and WEKA showed 35 GB/s to one H100 and up to 270 GB/s aggregate read across eight H100s, respectively. A separate VAST test with vLLM and LMCache cut time to first token for Qwen3-32B with a 130K-token prompt. Those are storage-vendor tests on H100 systems, not Rubin. Our earlier post on Dynamo's shadow engine recovery covers another part of this software.
Lever 4: what does the Vera CPU add?
Agent sessions alternate between GPU steps and CPU work such as tool calls, code execution and retrieval. During a gap, the session's KV cache either sits in HBM doing nothing or is evicted and later rebuilt. NVIDIA says a slow CPU costs HBM twice: idle residency, then recomputation.
The Vera CPU has 88 Olympus cores, which NVIDIA says give 1.8x faster single-threaded performance and 40% lower peak loaded latency than x86 CPUs. Each CPU offers up to 1.5 TB of LPDDR5X, more than 5x the HBM4 of one Rubin GPU, linked to Rubin at 1.8 TB/s over NVLink-C2C, about 7x a PCIe Gen 6 x16 link. Dynamo can park an idle session's cache there and bring it back when the tool returns.
Lever 5: what is CMX?
Some context must survive across turns, agents and nodes. Local SSDs are tied to one node, and enterprise storage carries overhead that temporary cache does not need. CMX adds a "G3.5" tier: Ethernet-attached flash shared across a pod, built on the NVIDIA STX reference design with BlueField-4 STX, Spectrum-X Ethernet, DOCA Memos and Dynamo and NIXL integration. NVIDIA says it gives up to 5x higher tokens per second and up to 5x better power efficiency than traditional storage for long-context inference.
The gain comes from sharing: a prefix computed on one node can be reused across compatible model instances in the pod.
How should you read the 30x number?
Treat it as a ceiling on a favorable benchmark, not a promise.
- Vendor source. The post is NVIDIA's own technical blog. The page also carries an AI-generated summary and a note that it may be incomplete.
- One benchmark. The headline uses SemiAnalysis AgentX at 160 tokens per second per user. Other interactivity levels and workloads can differ.
- Stacked claims. Several component figures come from different models and hardware: DeepSeek R1 on GB300, GB200 for expert parallelism, H100 for storage tests.
- No price. Throughput per megawatt is not cost per token. Hardware cost, utilization and software maturity all matter.
- Software dependence. Gains need NVFP4 models, Dynamo, TensorRT-LLM and the right fabric. Teams on other stacks may see less.
What does this mean for builders and buyers?
If you use model APIs, the practical effect is indirect. Agentic workloads are long, repetitive and tool-heavy. If providers can keep more sessions' context resident or cheaply parked, per-session cost can fall. Whether that reaches prices depends on providers and competition.
If you run your own inference, the five levers are a checklist: quantize carefully and test accuracy on your tasks, measure your prefix reuse rate, separate prefill from decode if bursts hurt latency, and decide where idle context should live. You can try most of these ideas in open-source serving stacks without waiting for Rubin hardware.
For the data center angle, see our post on Vera Rubin Kubernetes recipes and the first CoreWeave Vera Rubin cluster. For how model makers shrink cache on the software side, see DeepSeek V4.1 Flash and KV cache reduction.
What is still unknown?
- Independent results for the 30x claim on other workloads.
- Price and availability of CMX and Vera Rubin NVL72 in volume.
- Accuracy of NVFP4 on your own models and tasks.
- How much of the gain is hardware and how much is software you could get on current GPUs.
Related reading
- Rubin NVL72 throughput and cost, SemiAnalysis
- NVIDIA AI Infra Summit: Vera Rubin and Groq 3 LPX
- Dynamo shadow engine recovery
- Vera Rubin Kubernetes recipes
- CoreWeave first Vera Rubin cluster
- DeepSeek V4.1 Flash KV cache reduction
- Official: NVIDIA technical blog and Introducing NVFP4
Figures are NVIDIA's own, accurate as of November 5, 2026, and not independently verified.