NVIDIA Qwen3.8-Flash-Next FP4-Local: A 125B MoE in 96 GiB for One RTX PRO 6000

NVIDIA has published nvidia/Qwen3.8-Flash-Next-FP4-Local on Hugging Face: an FP4 build of Alibaba's Qwen3.8-Flash-Next, the 125B-parameter open-weight mixture-of-experts model. The checkpoint is about 96 GiB on disk, and NVIDIA says it holds within roughly 0.7 points of the BF16 original on the two benchmarks it reports. That is the headline for anyone who wants a frontier-class open model on a single workstation card.

The catch sits in the same model card: the weights use a new scale-factor compression format that upstream vLLM does not understand yet, so you have to run NVIDIA's linked community containers rather than a stock install.

TL;DR: what NVIDIA shipped

Why a 96 GiB checkpoint matters

Qwen3.8-Flash-Next is not a small model. Per the Qwen model card, it has about 125B total parameters, 48 layers and 512 experts, with 10 routed experts plus one shared expert active per token. Qwen describes it as an experimental, open-weight preview of the architecture it plans to use for Qwen4. Its selected scores include SWE-bench Pro at 62.5, LiveCodeBench v6 at 91.9 and GPQA Diamond at 91.7.

Running that in BF16 takes a multi-GPU server. The usual home-lab answer has been aggressive community quantization, as in our write-up of Strata running the same model on a 12 GB gaming GPU, which trades quality for fit. NVIDIA's release sits at the other end of that trade: a 96 GiB file that targets a 96 GB-class workstation card or a DGX Spark, with quality as close to BF16 as the vendor can get.

Because only 6B parameters activate per token, memory capacity, not compute, is the binding constraint. That is exactly the situation where 4-bit expert weights pay off.

Quantization compressing large weights into a compact form, as in the Qwen3.8-Flash-Next FP4 build

What NVIDIA actually did to the weights

This is not a plain post-training round-to-nearest conversion. The card describes three ingredients.

Mixed precision by tensor type. Decoder routed experts and the multi-token-prediction (MTP) experts use NVFP4, NVIDIA's 4-bit floating-point format. Decoder attention and some vision attention use MXFP8. The vision MLP layers use NVFP4, and the remaining vision tensors and some norms stay in BF16 or FP32. The idea is to spend 4 bits where the model is most tolerant, which is the huge pool of expert weights, and keep precision where errors compound.

Quantization-aware distillation (QAD). Instead of just calibrating, NVIDIA trained the quantized model against the BF16 model as a teacher. The card lists 2,500 trunk updates followed by 1,500 joint-refinement updates, on about 500M tokens over 3 epochs. Calibration and training data came from subsets of NVIDIA's Nemotron post-training and pretraining datasets. That is a small training budget next to pretraining, but enough to recover most of the loss from dropping to 4 bits.

The Model Optimizer toolchain. The work used NVIDIA Model Optimizer, version 0.47.0, the same library NVIDIA uses for other FP4 releases such as the Nemotron Labs Dory 4B looped model.

Two bit widths side by side, illustrating mixed NVFP4 and MXFP8 precision in the quantized Qwen3.8-Flash-Next

How much quality does it lose?

NVIDIA reports two numbers for the quantized build against the BF16 baseline:

Two cautions. First, these are the vendor's own measurements, and the BF16 GPQA figure (92.2%) is slightly above the 91.7% Qwen lists for its release model, so the two sets of numbers come from different evaluation setups and shouldn't be mixed. Second, the card says nothing about coding benchmarks such as SWE-bench, long-context retrieval at 500K+ tokens, or vision tasks for the FP4 build. Those are where 4-bit quantization tends to bite first, so test your own workload before replacing a BF16 deployment.

Hardware: what the card validates

NVIDIA lists two validated configurations, both on Blackwell with vLLM on Linux:

• NVIDIA RTX PRO 6000 Blackwell, single GPU. Uses PLE offload, meaning the large n-gram embedding tables are moved off the GPU, and supports up to 850K tokens of FP8 KV cache.
• NVIDIA DGX Spark. Runs with or without PLE offload, and with offload reaches up to 1M tokens of FP8 KV cache. For background on the box, see our coverage of the DGX Spark price and memory change and the RTX Spark local coding-agent setup.

Server towers and a green chip representing single-GPU and DGX Spark hosting for Qwen3.8-Flash-Next FP4

The card gives launch recipes for each. For DGX Spark, it uses the eugr/spark-vllm-b12x:latest container with HF_TOKEN set and vllm serve on port 8000, tensor parallel size 1, MTP speculative decoding with 4 tokens, FP8 KV cache, a 262,144-token maximum length, the qwen3 reasoning parser and the qwen3_xml tool-call parser. For the RTX PRO 6000 it points to the ghcr.io/local-inference-lab/vllm:release-10152026 container, with a one-line docker run and a longer manual path that uses 3-token MTP speculation. Both recipes require a long list of environment variables, so copy them from the model card rather than from a summary.

The catches before you try it

1. No stock vLLM. The NVFP4 scale factors use a new lossless compression format that upstream vLLM does not support yet. The linked containers are community builds, so treat them as third-party code and pin the image tag.
2. Blackwell only. The validated paths are Blackwell GPUs. Ampere and Hopper owners are not covered by this card.
3. Two licenses. NVIDIA's checkpoint falls under the NVIDIA Open Model Agreement, and the base model under the Qwen Community License 1.0. The card calls the model ready for commercial and non-commercial use, but compliance still means reading both.
4. Safety testing is on you. The card notes that outputs can be inaccurate, incomplete or biased, lists no bias-mitigation measures, and recommends application-specific safety testing before deployment.
5. Thin evaluation. Two benchmarks do not describe an agentic coding model's behavior. Run your own evals on tool calls, long contexts and code edits.

How this fits the local-model trend

The release lands in a month where big open-weight models are being pushed down to desk-sized hardware from several directions. Meta's open model has its own local-run guide, NVIDIA's own Nemotron line includes a 120B multimodal agent model, and enthusiasts keep pushing one-GPU setups, such as the FreeToken 753B local MoE hardware guide and the RTX 6000 Pro SOTA-locally write-up.

What is different here is who did the quantization. When the chip vendor ships a distilled FP4 checkpoint of a rival lab's model, trained against the BF16 teacher and validated on its own hardware, it becomes a reference point to compare community quants against. It also advertises NVFP4 as the format to build for on Blackwell, which is a commercial incentive worth keeping in mind as you read the numbers.

Should you use it?

• You own an RTX PRO 6000 Blackwell or a DGX Spark and want a strong open agentic model locally: this is the most vendor-supported route to Qwen3.8-Flash-Next at near-BF16 quality. Start with the DGX Spark or RTX recipe, serve with the reasoning and tool parsers, and run your own coding evals.
• You have a smaller GPU: this checkpoint does not fit. Look at the lower-bit community builds, and expect more quality loss.
• You need stock tooling for a production fleet: wait until upstream vLLM supports the format, or plan to maintain the container yourself.
• You are comparing quantizers: use the 0.7-point GPQA and ToolEvalBench gap as a baseline for QAD, then measure your own tasks. A 4-bit model that loses 0.7 points on multiple choice science questions can lose more on long code edits.

Practical checklist

1. Download from the model card with an HF_TOKEN and about 96 GiB of free disk.
2. Pick the matching container (DGX Spark or RTX PRO 6000) from the card and pin the exact tag.
3. Copy the environment variables and vllm serve flags from the card, including the parsers.
4. Start with the 262,144-token context. Only move toward the 850K to 1M FP8 KV settings after your base setup is stable.
5. Compare against your current model on a fixed set of your own prompts before any switch.

Related reading

• Qwen3.8-Flash-Next: the 125B MoE Alibaba teased
• Strata: Qwen3.8-Flash-Next 125B on a 12 GB GPU
• NVIDIA DGX Spark price and memory change
• NVIDIA RTX Spark: local coding agents
• Nemotron 3.5 Super VL: 120B multimodal open agent model
• FreeToken 753B local MoE hardware guide
• Official: NVIDIA model card, Qwen base model, NVIDIA Model Optimizer

Specs and container tags are as listed on the model card on October 20, 2026 and may change.