Q Labs Dust: Pretraining Transformers Without Backprop, and the 4,000x Claim Checked

Q Labs Research has published Dust, a method for pretraining transformers without backpropagation, and a companion benchmark called Noise. Alongside it, a post spreading on X claims pretraining time dropped from 1.76 hours to 1.6 seconds, a roughly 4,000x speedup, and that Dust beats backprop with Adam. The Dust paper page and the Noise repository say something narrower and more interesting. This post separates what the authors show from what the headline implies.

The short version: the result is real as research. Dust trains a transformer using only forward passes and noise, and it beats an earlier method called EGGROLL by a wide margin. The 1.6-second figure also exists, in the archived 10M-token Noise speedrun table, but it is a community record to a loose loss target using newer methods, and the authors themselves say Dust is not yet a practical replacement for backpropagation.

Dust and Noise at a glance

Why training without backpropagation matters

Almost every modern model is trained with backpropagation: a forward pass computes the loss, then a backward pass computes exact gradients for every weight, and an optimizer such as Adam applies them. It works extremely well, but it needs the whole computation graph to be differentiable, and it stores activations for the backward pass.

Zeroth-order methods estimate the direction to move by trying random perturbations and seeing whether the loss improves. They need only forward passes, work with non-differentiable components, and map naturally to hardware where backward passes are costly. The catch has always been efficiency: you need huge numbers of samples to get a usable gradient estimate. That is why backprop won.

We have covered other attempts to loosen backprop's grip, including Sakana AI's PC-ALM, which trains deep nets without backpropagation. Dust attacks the same problem from the evolution-strategy side.

How Dust works in plain language

The paper describes the method in four moves.

1. Add noise to activations. At every token, Dust adds Gaussian noise to the outputs of linear layers, rather than perturbing the weights themselves.
2. Run one forward pass. Because each token carries its own noise, a single pass evaluates many perturbations at once. The authors call this a virtual population and say one pass evaluates at least three orders of magnitude more members than weight-space evolution strategies.
3. Score each token. Each token's reward is the change in its loss, plus discounted changes in later tokens that the noise can reach through attention.
4. Turn rewards into a gradient. Averaging reward-weighted noise estimates the error at each layer's output, and multiplying by the layer's input gives the weight gradient.

To limit interference, layer types are perturbed in separate passes, attention internals are scored by their effect on the attention output, and the language-model head is perturbed directly on cached logits. Hyperparameters were tuned by grid search.

What the paper actually shows

The setup is modest by frontier standards: GPT-style transformers on FineWeb with a 4,096-token tokenizer, batches of 16k tokens, one epoch, and a base model with 8 layers and width 512, three seeds per configuration.

Versus backprop with SGD. At 100k and 1M training tokens, Dust ends below backprop once it has a few hundred to about a thousand draws. At 10M tokens, the fitted limit lands just above backprop. At 20M tokens the power-law fit gives 4.431, with a wide 95 percent interval of 3.89 to 4.58, compared with 4.633 for backprop. The authors warn that this fit is loosely constrained because the curve is still falling.

With Adam. At 1M tokens, Adam improves both Dust and backprop. The paper's fitted limit is 5.248 for Dust versus 5.361 for backprop. The Noise README, in contrast, says Dust still trails backprop under Adam, so the two sources frame the Adam comparison differently. We read this as extrapolation versus measured runs and recommend checking the tables yourself.

Versus EGGROLL. EGGROLL at 16k draws still does not reach Dust at 64 draws. From 1M tokens upward the authors estimate Dust is roughly 1,000 to 10,000 times more efficient, and flag it as an extrapolation.

Model size. Across 2M, 7.3M, 38M and 243M parameters at 10M tokens, larger models were generally more population-efficient. That cuts against the usual belief that zeroth-order methods get worse as models grow.

Gradient alignment. Dust's gradient estimates align better with backprop's as population grows, and alignment stays fairly stable as tokens increase up to 1B, which the authors read as encouraging for scaling.

The claim: 1.76 hours to 1.6 seconds

The post circulating on X says the time to reach a fixed validation-loss target fell from 1.76 hours to 1.6 seconds in ten days, a 4,000x improvement, and that this came from the Noise speedrun. The arithmetic of the ratio checks out: 1.76 hours is about 6,336 seconds, and dividing by 1.6 gives roughly 3,960.

The endpoints do appear in the benchmark's records, with important context.

The README says that until October 22 the training set was 10M tokens, archived in a 10m/ folder, and the tracks now use 30M tokens with different targets. In the archive, 1 hour 46 minutes is about 1.77 hours, and dividing by 1.6 seconds gives about 3,975, so the headline ratio holds for that table.

Three caveats keep it from meaning "pretraining got 4,000x faster." First, the target was loss 4.2, far weaker than the 3.769 backprop with Adam reaches on the same data. Second, the fast entries are not Dust as published: their descriptions (stagewise forward-only training, closed-form and identity-path credit with exact values) point to newer, more structured credit-assignment methods that the rules allow, and the later records use token-sharded kernels and JAX. Third, the benchmark was reset to 30M tokens and a 3.5 target, where the best speedrun is still 1 hour 43 minutes. We also did not verify the "ten days" timeline in the viral post.

The honest limits the authors list

• Compute efficiency is not solved and needs orders-of-magnitude improvement.
• Many headline comparisons, including the efficiency multiples and the 20M-token limit, rely on extrapolation.
• Results come from one dataset (FineWeb) and models up to 243M parameters.
• The cosine analysis reaches 1B tokens, but the main training experiments do not.
• Looped or externally augmented architectures were not tested.
• Tuning needs grid searches, though settings largely transfer.

The authors also say Dust is not yet practical to replace backprop at current compute levels, and that it may surpass backprop with enough compute, a possibility rather than a demonstrated result.

What this means if you build or train models

For most readers, nothing changes this week. You will not swap Dust into a production training run. The value is in the direction:

• Research direction. If zeroth-order methods keep improving, training on hardware without efficient backward passes, or with non-differentiable pieces, becomes more realistic.
• Open benchmark. Noise gives a fixed model, data and score, with a rule that bans autograd, so progress is comparable. Submissions are pull requests that include logs, so claims are checkable. That is the right way to evaluate a speed claim.
• Scaling questions. The finding that larger models are more population-efficient is worth watching, since it runs against expectation.

If you work on small-model training or on-device learning, our posts on in-browser LLM fine-tuning with WebGPU and MicroLLM Lab's tiny browser models show where cheap training matters. For the wider debate about whether current methods are the path forward, see Yann LeCun on why LLMs are not AGI and Sakana's work on recursive self-improvement.

How to check the claim yourself

1. Open the Noise README and read the current leaderboard and the track definitions, including the 30M-token change.
2. Look for a pull request or log folder with config.json, metrics.jsonl and result.json for any claimed record. The README says the training script is the record.
3. Compare the validation-loss target in the claim with the target in the track; records are only comparable at the same target, data and hardware.
4. Read the paper's limitations section before quoting the efficiency multiples.
5. Treat any figure without a reproducible log as unverified.

What is still open

The authors say they are testing whether the efficiency gains scale by growing the training set from 10M to 30M tokens and finding which speed-up techniques fail under overfitting. We will revisit this post if the 30M-token leaderboard changes.

Related reading

• Sakana PC-ALM: training deep nets without backpropagation
• In-browser LLM fine-tuning with WebGPU
• MicroLLM Lab: tiny LLMs in the browser
• Yann LeCun on LLMs, AGI and JEPA
• Sakana MASS and recursive self-improvement
• Infinite-parameter LLMs and hypernetworks
• Official: Dust paper page and Noise repository

Figures are from the Q Labs paper page and the Noise README as of October 22, 2026, and may change as the leaderboard updates.