Diffusion language models promised faster text generation by producing many tokens at once. A new paper says there is a faster route. Next-Concept Modeling (NCM), from Hazy Research and collaborators, generates text in a few coarse-to-fine steps and reports speedups of up to 31x over diffusion language models at similar quality. The arXiv paper was submitted on October 9, 2026, and the team released code, checkpoints and datasets on GitHub.
This post explains the idea in plain language, lists the reported numbers with their sources, shows how to run the released models, and flags what is not yet known. If you follow inference speed, it is one of the more interesting architecture papers of the month.
Quick reference
| Question | Answer |
|---|---|
| What is it? | A two-stage coarse-to-fine generator for text |
| How does it generate? | One forward pass per level of a code hierarchy, all positions in a level in parallel |
| Compared against | Diffusion language models (DLMs); autoregressive baselines on Sudoku |
| LM1B | Better generative perplexity than the strongest DLM at matched scale, 21x faster |
| OpenWebText | Competitive with the strongest DLM at about 10x the speed |
| XSum summarization | Matches the strongest DLM's ROUGE-L at 31x the speed |
| Sudoku | Beats autoregressive and diffusion baselines at all difficulty levels, with fewer function evaluations |
| Released | Code, tokenizer and generator checkpoints, test sets, datasets |
What problem is it solving?
Autoregressive models generate one token at a time, so a 1,000-token answer needs about 1,000 sequential forward passes. As inference grows into the dominant cost of running AI, that is a bottleneck. Diffusion language models try to fix it by refining the entire sequence over several denoising steps. The paper's abstract points out three weaknesses of that approach: DLMs degrade quickly when you cut the number of sampling steps, they process the full sequence at every step, and they need many passes over the dataset to reach good quality.
We have covered several attempts at faster text generation, including DiffusionGemma's 4x speedup, Inception's Mercury 2 at 1,000 tokens per second and Nvidia's TwoTower diffusion LLM work. NCM takes a different path from all of them.
How NCM works
The framework has two stages.
- A multiscale residual VQ-VAE compresses text into a hierarchy of discrete codes. Think of it as writing a summary at several zoom levels: a very short sequence of coarse codes captures the gist, and each further level adds a longer sequence of finer detail. VQ-VAE here means a model that maps its inputs to entries in a learned codebook.
- A transformer generates one level at a time. Within a level it predicts all codes in parallel. So the number of forward evaluations equals the number of levels, not the number of tokens.
The idea borrows from next-scale prediction in image generation, where models first draw a rough low-resolution picture and then add detail. The repository's description matches: decoding each level shows a sample being refined from a rough sketch to the full output in a few steps.
The speed advantage follows directly. If a hierarchy has a handful of levels, generating a sequence takes a handful of passes however long the text is, and each pass works on a short sequence at the coarse levels. The paper also reports that NCM needs roughly 4 to 5 times fewer training tokens than strong diffusion baselines on LM1B and OpenWebText, according to the paper's introduction. The models evaluated are around 100M parameters.
The reported results
All figures below come from the authors' abstract and README, not from our own runs.
- LM1B (unconditional generation): better generative perplexity than the strongest DLM baseline at matched parameter scale, running 21x faster.
- OpenWebText: competitive with the strongest DLM while running 10x faster.
- XSum summarization: matches the strongest DLM's ROUGE-L while being 31x faster.
- Sudoku: outperforms autoregressive and diffusion baselines across all difficulty levels, using fewer function evaluations.
The repository frames the overall range as generating 9x to 31x faster than diffusion baselines on different tasks and datasets. A notable secondary claim from the introduction is that spending the saved time on more candidates and choosing among them with minimum Bayes risk decoding gives about 7% higher ROUGE-L than the strongest summarization baseline within the same sampling-time budget, according to the paper's introduction.
What was released and how to try it
The README says weights and tokenized datasets are in a Hugging Face collection. Each released model has a frozen tokenizer and a next-concept generator, and there are entries for XSum, WMT14 German to English, Sudoku at three difficulty levels, LM1B at three sizes, TinyStories and OpenWebText.
Installation is a clone and pip install -r requirements.txt, with Python 3.10 or newer and a CUDA-capable PyTorch install recommended. The README gives download commands for the checkpoints and datasets with the Hugging Face CLI, then an evaluate.py entry point with one config per task. It runs one generation shard per visible GPU and prints task metrics: ROUGE and generative perplexity for XSum, sacreBLEU and chrF for WMT14, solve rates for Sudoku, and generative perplexity under GPT-2 large for the unconditional tasks. That makes the headline numbers checkable by anyone with a GPU, which is the best part of the release.
NCM, diffusion and autoregressive models side by side
The three families trade speed, quality and maturity differently. This table is a conceptual summary based on how each approach is described in the NCM paper and the posts linked above, not a benchmark we ran.
| Property | Autoregressive | Diffusion language model | NCM |
|---|---|---|---|
| Generation order | Left to right, one token per pass | Whole sequence refined over several denoising steps | Coarse to fine, one pass per level |
| Passes needed | About one per token | Tens of steps typically, quality drops when cut | A few, set by the number of levels |
| Work per pass | Attends to the prefix | Processes the full sequence each step | Short sequences at coarse levels |
| Training data need | Mature recipes | Reported to need many dataset passes | Reported 4 to 5 times fewer tokens on two benchmarks |
| Tooling maturity | Very high | Growing | Research code only |
| Evidence at scale | Extensive | Early | Not yet shown |
Two practical consequences follow. First, the comparison NCM wins is against diffusion models, so a team already happy with an autoregressive stack has no immediate reason to move. Second, the approach is most interesting where latency is critical and outputs are structured, such as summarization, translation and constraint puzzles like Sudoku, which are exactly the tasks in the paper.
Questions people will ask
Does a coarse-to-fine hierarchy lose information? Any compression can, and the paper's tokenizer must reconstruct text well enough for downstream metrics. The reported ROUGE and BLEU-style scores suggest the loss is small on the tested tasks, but long, rare-word or code-heavy text is untested in the material we read.
Is "concept" a literal meaning unit? The name suggests it, but the repository describes the codes as a hierarchy from a multiscale residual quantizer. Treat "concept" as a branding choice for the coarse levels, not a claim about interpretable semantics.
Could this combine with existing speedups? In principle, fewer sequential passes could stack with other optimizations such as quantization or better kernels, but the authors have not published such combinations, so that remains speculation.
What this does and does not mean
NCM is not a chatbot. The released models are research checkpoints trained for specific tasks and datasets, and the comparisons are with diffusion language models at matched small scale. Several limits are worth keeping in mind.
- Scale is unproven. Nothing in the abstract tests billions of parameters or open-ended instruction following, where autoregressive models dominate.
- Quality metrics are narrow. Generative perplexity, ROUGE and Sudoku solve rate are useful but are not the same as human judgments of long-form text.
- Compression is lossy by design. A discrete code hierarchy has to reconstruct text faithfully; how well it handles rare words, code and long documents is a question for the full paper and for replication.
- Ecosystem gap. Serving stacks, quantization tools and fine-tuning recipes are built for autoregressive transformers. A new decoding scheme starts with none of that.
- Self-reported results. The paper is a preprint; no independent replication yet.
Still, the direction matters. If coarse-to-fine decoding scales, it offers another way to cut inference cost beyond speculative decoding and diffusion approaches. For readers choosing infrastructure now, the practical stance is to keep serving autoregressive models and watch for larger NCM checkpoints.
Why it matters
Inference cost shapes everything from API prices to how many agent steps you can afford. Research that cuts the number of sequential passes without losing quality has outsized leverage, which is why labs keep funding speedups like the ones above. NCM's strongest evidence is that it holds up against diffusion models while using fewer sampling steps and fewer training tokens, two places where DLMs have struggled. The weakest point is the usual one for new architectures: it needs to be shown at scale, on general tasks, by people outside the lab.
Related reading
- DiffusionGemma: 4x faster text generation
- Mercury 2: a diffusion LLM at 1,000 tokens per second
- Nvidia's TwoTower diffusion LLM
- DeepSeek speculative decoding guide
- VRSD: a teacher-only prompt for self-distillation
- Last Secure Code Benchmark
Results are the authors' reported numbers as of October 14, 2026; the paper is a preprint.