DGX Spark MoE Tuning: Ling-3.0-flash Hits 102 Tokens/s, but Only 54 on Real Tasks

NVIDIA's developer blog has a practical write-up on making a large mixture-of-experts (MoE) model faster on a single DGX Spark: 3 Ways to Boost MoE Inference on NVIDIA DGX Spark. The model is Inclusion AI's Ling-3.0-flash at batch size 1. The headline is a 185.1% decode speedup, from 36.09 to 102.88 tokens per second. The more useful number is lower down: on the full GSM8K test set the gain is 42.5%, from 37.81 to 53.89 tokens per second, with accuracy essentially unchanged.

This post walks through the three changes, why they work, and how to read the two sets of numbers, because the gap between them is the lesson for anyone tuning local inference.

TL;DR

What is Ling-3.0-flash, and why is decoding memory-bound?

According to NVIDIA, Ling-3.0-flash-fp4 has 124B total parameters and 5.1B active per token. It has 42 layers: 35 Kimi Delta Attention linear-attention layers and 7 multi-head latent attention layers. Each MoE layer routes every token to 8 of 512 routed experts plus one shared expert. The native context is 256K tokens, expandable to 1M.

At batch size 1, a single user chatting with a local model, NVIDIA's profiling found decoding is memory-bound: the time per token is dominated by reading weights, not by arithmetic. So the optimizations target two things: how many bytes of weights are read per step, and how many decode steps are needed. That framing explains why a seemingly small change like quantizing the LM head matters. If you are new to the idea, our quantization guide explains why smaller weights mean faster reads.

A mixture-of-experts router sending tokens to a few of many experts, illustrating DGX Spark MoE inference

Optimization 1: Humming MXFP4 MoE backend (plus 9.3%)

The baseline is the official W4A8 cookbook configuration with FlashInfer MXFP4 MoE kernels, NVIDIA CUTLASS dense GEMM and --enable-fp32-lm-head. The first change swaps the flashinfer_mxfp4 MoE backend for the Humming kernel library (humming-kernels on PyPI).

Humming uses NVRTC to compile kernels at runtime for the actual shapes, data types and GPU architecture, caching binaries in ~/.humming/cache. NVIDIA says individual kernels compile in seconds, while the FlashInfer CUTLASS path takes tens of minutes across 97 targets. On the synthetic test, time per output token drops from 27.71 ms to 25.35 ms, 39.45 tokens per second.

Optimization 2: online FP8 LM head (plus 21.7% cumulative)

The language-model head has a hidden dimension of 2,560 and a vocabulary of 157,184, and it is stored in BF16 and not quantized at load. Every decoded token reads all of it. Applying dynamic FP8 quantization only when constructing the LM head halves that weight-read volume. It is enabled with SGLANG_ENABLE_FP8_LM_HEAD=1, and it is mutually exclusive with --enable-fp32-lm-head. Cumulative time per token falls to 22.76 ms, 43.94 tokens per second.

Optimization 3: MTP speculative decoding with NEXTN (plus 185.1% cumulative)

Multi-token prediction uses the model's built-in NEXTN prediction layer (num_nextn_predict_layers: 1) as a draft model, so there is no extra training or download. NVIDIA used 3 draft steps, top-k 1 and 4 draft tokens. The big jump comes here: 9.72 ms per token, 102.88 tokens per second, with an accept length of 3.90 tokens per round.

There is a cost on the MoE side. The measured verification-cost ratio was about 2.26, mainly because verifying several draft tokens activates up to 32 experts versus 8 for a normal decode step. Speculative decoding is therefore less of a free lunch for MoE than for dense models. If you want a broader look at this technique, see EAGLE-4 speculative decoding on Qwen3.8-27B.

The full results table

Synthetic workload (32K input, 2K output, batch size 1, greedy decoding):

Full GSM8K test set (1,319 questions, batch size 1, five-shot, temperature 0.6, top_p 0.95, top_k 29, max_tokens 32,768):

Note that first-token latency is essentially unchanged: these tweaks help decode, not prefill.

Why the synthetic number overstates the gain

NVIDIA says so itself. Repetitive synthetic output and greedy decoding make draft tokens easy to accept: an accept length of 3.90 versus 2.68 on GSM8K. The 185.1% figure does not carry over to other workloads. Shorter real outputs (GSM8K median about 310 tokens) also spread fixed-overhead savings over fewer tokens. The real-workload gain is 42.5%, and that is the figure to plan around. When you evaluate any vendor speedup, ask whether the benchmark output resembles your own, a point we make in how to read AI benchmarks.

The silent-fallback trap

The most valuable warning is a failure mode. When FP8 LM-head quantization and MTP are enabled together, the draft model must share the entire LM-head module, not just its weight tensor. Otherwise, NVIDIA says, the framework silently falls back, accept length drops to 1.00, and throughput ends up lower than the unoptimized baseline. The recommendation: check at startup that every optimization is actually active, and do not infer it from the performance numbers. In practice, log the accept length: a value near 1.00 means speculative decoding is doing nothing.

How to try it

NVIDIA points to the Ling cookbook: a Humming plus FP8 plus MTP notebook and a standard MXFP4 baseline notebook. The ingredients are:

• humming-kernels from PyPI
• SGLANG_ENABLE_FP8_LM_HEAD=1
• MTP settings: steps 3, top-k 1, draft tokens 4

The post does not list software versions, full launch commands or driver and CUDA versions, so use the cookbook for those and expect to pin versions yourself.

A checklist for applying this to your own setup

The specific model matters less than the method NVIDIA used. If you run any MoE model locally, the same sequence applies:

1. Profile before tuning. NVIDIA began by establishing that decode was memory-bound at batch size 1. If your workload is batched or prefill-heavy, the answer may differ, and so will the right optimization.
2. Look for weights you read every step but never shrank. The LM head is the classic example: a large vocabulary multiplies a wide hidden dimension, and it is read for every token. Quantizing it is cheap relative to its share of memory traffic. Check what your runtime leaves in higher precision by default.
3. Treat compile time as a cost. The difference between kernels that compile in seconds and a path that takes tens of minutes across 97 targets is felt every time you change shapes or upgrade a driver, not just once.
4. Measure with your own prompts. Run a task that resembles your use, and record accept length alongside tokens per second. The 3.90 versus 2.68 accept lengths show how much the draft model's success depends on the text.
5. Verify every optimization is live. Add a startup check or a log assertion. A configuration that quietly disables speculative decoding looks like a working system that is simply slower.
6. Check quality on a held-out set. NVIDIA ran all 1,319 GSM8K questions with zero request errors or truncations and saw a 0.08 point accuracy difference. Speculative decoding is designed to preserve the output distribution, but sampling settings, quantization and fallbacks are where bugs hide, so measuring is cheap insurance.

One more detail deserves attention: P95 latency fell from 50.20 s to 38.60 s while P50 fell from 8.36 s to 6.11 s. The long tail of lengthy reasoning outputs benefits in absolute terms, which matters most for agent loops where a slow response blocks everything downstream.

Caveats

• One model, one hardware platform, batch size 1. Do not assume the same gains on other MoE models or at higher batch sizes, where weight reads are amortized across requests.
• The numbers come from NVIDIA's own post, which promotes its hardware. We did not reproduce them.
• GSM8K is a math benchmark with short outputs. Long-form coding or agent traces may behave differently.

Where this fits in our DGX Spark coverage

DGX Spark is increasingly the box people point local agents at. See our posts on the eight-node DGX Spark cluster running a 753B model, the DGX Spark price and memory change, Qwen3.8-Flash-Next in FP4 on DGX Spark and the inference engine TokenSpeed. The through-line: for single-user local inference, memory traffic and decode steps decide your speed, and the best wins come from reducing both.

Related reading

• What is AI model quantization? Complete guide
• EAGLE-4 speculative decoding on Qwen3.8-27B
• Eight-node DGX Spark cluster
• DGX Spark price and memory change
• Qwen3.8-Flash-Next FP4 on DGX Spark
• TokenSpeed inference engine
• External: NVIDIA's post · Ling cookbook

Numbers reflect NVIDIA's post of October 27, 2026 and may change with software versions.