NVIDIA Dory 4B: A Looped Model You Can Dial Up at Inference

NVIDIA has published Nemotron-Labs Dory1.0-4B, a small language model that does something most open releases do not: it reuses the same middle layers several times, and you decide how many times at inference. The model card on Hugging Face says one checkpoint can serve different compute budgets, and NVIDIA reports 61.8 on SWE-bench Verified at four loops, close to the 61.0 it lists for Qwen3.5-9B. The repo we look at here is the NVFP4 (4-bit) build, aimed at DGX Spark-class local hardware.

This is NVIDIA's own data throughout. We have not run the model, and no independent benchmarks exist yet, so treat every score below as a vendor claim until someone reproduces it.

TL;DR: Dory at a glance

What is a looped language model?

A normal transformer stacks, say, 36 different layers and runs the input through each once. A looped model keeps some of those layers and runs them again. Dory splits its 36 layers into three groups: 8 input layers, 20 recurrent layers that are shared and reapplied, and 8 output layers. Run the recurrent group once and you have a shallow model. Run it four times and the effective depth grows without adding parameters.

That trade is the point. The memory cost is a 4B model, but compute per token scales with loop count. It gives you a dial between speed and quality on the same weights. Earlier work on recurrent-depth models explored the idea for reasoning; Dory is a vendor-grade release with long context, quantized builds and tooling.

Two design details matter for people who deploy models:

• Interleaved attention. The model mixes sliding-window attention layers with full-attention layers, which keeps long-context memory manageable.
• Last-loop KV caching. The key-value cache stores only the final loop's states, so cache size stays fixed no matter how many loops you run. Without that, a four-loop model would need four times the cache, which would defeat the purpose on a 256K context.

If you want a refresher on why the cache dominates long-context cost, see our earlier coverage of NVIDIA EdgeWise, which tackles the cost of agents from the opposite direction: a small local model that decides when to call a big cloud one.

Why does adjustable depth matter for local agents?

Most small-model deployments pick one model and live with its speed and quality. Agent workloads are uneven: a tool-call that fills in a JSON field needs little reasoning, while a multi-file bug fix needs a lot. With a conventional model you solve that by keeping two or three checkpoints loaded, which costs memory and adds routing logic. A looped model lets the same resident weights act as a fast model or a slower, stronger one, chosen per request.

That is attractive on unified-memory machines, where RAM is the binding constraint rather than raw compute. It also changes how you think about cost. Instead of asking which model to call, you ask how many loops a step deserves. Whether that works in practice depends on the serving stack exposing the loop count per request rather than per server, which the vLLM override on the card does not obviously do: it is set when the server starts. Treat per-request switching as something to verify, not something promised.

What do the benchmarks say?

Quantization compressing a large model into a smaller one, illustrating the NVFP4 build of Dory 4B

The card reports BF16 results at one, two and four loops, next to Qwen3.5-9B:

Three readings stand out. First, the jump from one loop to two is large on coding tasks (9.2 points on SWE-bench Verified), while two to four is modest (2.3 points). That supports NVIDIA's recommendation of two loops as the default. Second, GPQA Diamond does not improve from two to four loops, so extra looping is not a universal win. Third, a 4B model matching a 9B rival on SWE-bench Verified is notable, but Dory is behind on Terminal-Bench 2.1 and well behind on GPQA Diamond. The card's table also shows it behind the larger Qwen3.6-35B-A3B on most tasks, though it scores higher on IFBench (79 versus 57.44).

The card additionally says looping reduces the number of tokens the model generates, which would partly offset the extra compute per token. We did not find a table quantifying that, so we would not bank on it.

For another recent NVIDIA small-model comparison against Qwen and Gemma, see our Nemotron 3.5 Bolt 74B-A9B analysis.

How was it trained?

According to the card, Dory was pre-trained on 6T tokens, followed by 300B tokens of context extension, then post-trained with supervised fine-tuning and multi-environment GRPO reinforcement learning using NeMo RL and NeMo Gym, with Megatron-LM as the training framework. Separately, the model was pre-trained natively in NVFP4 for roughly 4.5T tokens before continued training in BF16.

Native low-precision pre-training is different from quantizing afterwards. Post-training quantization squeezes a finished model and hopes accuracy survives. Pre-training in NVFP4 lets the weights adapt to the format from the start. The card lists NVFP4 as W4A16: weights in 4 bits, activations in 16.

How fast is the NVFP4 build?

On DGX Spark the card reports NVFP4 at 2.17 to 2.44 times the speed of BF16, and MXFP8 at 1.69 to 1.84 times, in the two configurations NVIDIA tested. The card does not say which prompt lengths or batch sizes those were, so your results will vary. Decode on a small model is memory-bandwidth bound, which is why a 4-bit weight format helps so much on a unified-memory box. If you are choosing hardware, our look at RTX Spark as a one-machine code, test and deploy setup covers the other end of NVIDIA's local-AI push.

There is one oddity. The card text says 4B parameters, while Hugging Face's file metadata for this NVFP4 repo shows 2B. That is probably an artifact of how packed 4-bit tensors are counted, but NVIDIA has not explained it, so check the numbers yourself.

How to try it

Dory is not supported in stock releases yet. The card gives three routes:

1. vLLM. Install from pull request 60149 until it lands in a release, then serve with a Hugging Face override that sets the loop count, for example n_recurrent_loops to 2.
2. llama.cpp. Build from the pinned Dory-enabled commit in pull request 30306, load the GGUF Q5_K_M file, and set dory.num_loops through --override-kv.
3. Transformers. Use AutoModelForCausalLM or the text-generation pipeline.

The card recommends temperature 1.0 and top-p 0.95. Starting points on the Hugging Face side are the NVFP4 repo and the sibling BF16, MXFP8 and GGUF repos in the same collection.

A practical test plan: run your own coding or tool-use prompts at 1, 2 and 4 loops and plot quality against latency. If your workload looks like GPQA, expect the two-loop setting to be enough. If it looks like SWE-bench, four loops may justify the cost.

What are the limits and open questions?

• English only, according to the card, which rules it out for multilingual products.
• Vendor numbers. Every score here comes from NVIDIA's own evaluation. The card compares against Qwen3.5-9B, but harness settings and prompts matter a lot for SWE-bench and Terminal-Bench.
• Tooling lag. Needing unmerged pull requests in vLLM and llama.cpp is a real deployment barrier. Expect rough edges.
• Loop-count compute. Four loops means roughly four passes through the 20 shared layers per token. Fixed cache size helps memory, but not arithmetic.
• Agent use. The coding numbers suggest it could serve as a local agent model, but no agent-harness results appear on the card.

What this means for what you build or pay

If you run local agents on a DGX Spark or similar machine, Dory offers a way to get 9B-class coding scores from a 4B-class memory footprint, with a knob to trade latency for accuracy per request. That is useful for routing: cheap one-loop calls for simple steps, four loops for hard ones. You pay nothing in license fees for commercial use under OpenMDW, but you pay in engineering time to build from pull requests until support lands.

If you are on a laptop with a modest GPU and want something that works with stock tools today, a standard small model remains the lower-friction choice. Revisit Dory when the vLLM and llama.cpp PRs merge.

Context for the broader NVIDIA open-model push is in our post on Nemotron 3.5 Super VL and the Nemotron 3.5 reranker, which uses the same OpenMDW license family.

Related reading

• NVIDIA EdgeWise: local agent delegates to the cloud
• Nemotron 3.5 Bolt 74B-A9B vs Qwen and Gemma
• RTX Spark: code, test and deploy on one machine
• Nemotron 3.5 Super VL
• Nemotron 3.5 Rerank 8B
• Tencent DSpine speculative decoding drafter
• Official: OpenMDW license, vLLM pull request 60149, llama.cpp pull request 30306

Specifications, benchmark figures and pull-request status are accurate as of October 17, 2026 and may change.