Q Labs Dust: Pretraining Transformers Without Backprop, and the 4,000x Claim Checked

Q Labs Research has published Dust, a method for pretraining transformers without backpropagation, and a companion benchmark called Noise. Alongside it, a post spreading on X claims pretraining time dropped from 1.76 hours to 1.6 seconds, a roughly 4,000x speedup, and that Dust beats backprop with Adam. The Dust paper page and the Noise repository say something narrower and more interesting. This post separates what the authors show from what the headline implies.

The short version: the result is real as research. Dust trains a transformer using only forward passes and noise, and it beats an earlier method called EGGROLL by a wide margin. The 1.6-second figure also exists, in the archived 10M-token Noise speedrun table, but it is a community record to a loose loss target using newer methods, and the authors themselves say Dust is not yet a practical replacement for backpropagation.

Update (October 23, 2026): the Noise benchmark moved to 30M tokens, and a forward-only record now edges Adam

On October 22, 2026 the Noise repository switched its training set from Dust's 10M tokens to 30M tokens of FineWeb, archiving the old 10M tracks in archive/10M/. The model is unchanged: the 37.7M-parameter transformer from the Dust paper. The speedrun target also changed, to a validation loss of 3.5.

The new headline number circulating on X, a forward-only method scoring 3.409 against 3.419 for backprop with Adam, is in the repository's records table. Here is what the README actually shows:

Three points keep this in proportion:

• It is a narrow win. 3.409 versus 3.419 is a gap of 0.010 on a single seed. The README itself says the goal is "to beat backprop by a large margin," and a better backprop optimizer in the same table (3.340) still beats the forward-only record.
• The time is not the 1.6-second story. The record takes 1 hour 43 minutes, so the earlier speedup figure discussed below belongs to the old 10M-token, loose-target table and should not be stacked on this result.
• It is not the 243M model. Some summaries pair this result with Dust's 243M-parameter scaling claim. The 3.409 entry is for the 37.7M model; the 243M result comes from the Dust paper's scaling runs.

The maintainers say they are enlarging the dataset to learn which efficiency gains survive without overfitting. Treat the number as evidence that community-built forward-only credit assignment is closing in on Adam at small scale, not as proof that backprop is obsolete. Other recent work on the same question is covered in our Sakana PC-ALM post.

Update (October 24, 2026): the 30M speedrun drops to minutes

The 1 hour 43 minute entry above is no longer the fastest way to reach the 3.5 target. The Noise README now lists a second speedrun record: validation loss 3.499 in 8 minutes 20 seconds on one 8xH100 node, credited to @RobertTLange, using "the identity path first, then Cut-lite at 8 rows per update with faster chain reruns." That is roughly 12 times faster than the 1 hour 43 minute entry.

A further claim is circulating: that a newer run reached loss 3.5 in 7.9 minutes, credited to the same researcher with CUDA graphs, fused kernels and a revised Cut estimator that reduces probe cost. A secondary summary calls it a 12.9x improvement over a 102.5-minute previous record. We could not find the 7.9-minute entry in the repository table we read (it lists 8 min 20 s), so treat 7.9 minutes as reported and check the records table and its logged run before citing it. Note the 3.499 here is a speedrun to a loose target, a different track from the best validation loss (3.409) in the table above, and none of this changes the caveat that a better backprop optimizer still scores lower loss.

Dust and Noise at a glance

Why training without backpropagation matters

Almost every modern model is trained with backpropagation: a forward pass computes the loss, then a backward pass computes exact gradients for every weight, and an optimizer such as Adam applies them. It works extremely well, but it needs the whole computation graph to be differentiable, and it stores activations for the backward pass.

Zeroth-order methods estimate the direction to move by trying random perturbations and seeing whether the loss improves. They need only forward passes, work with non-differentiable components, and map naturally to hardware where backward passes are costly. The catch has always been efficiency: you need huge numbers of samples to get a usable gradient estimate. That is why backprop won.

We have covered other attempts to loosen backprop's grip, including Sakana AI's PC-ALM, which trains deep nets without backpropagation. Dust attacks the same problem from the evolution-strategy side.

How Dust works in plain language

The paper describes the method in four moves.

1. Add noise to activations. At every token, Dust adds Gaussian noise to the outputs of linear layers, rather than perturbing the weights themselves.
2. Run one forward pass. Because each token carries its own noise, a single pass evaluates many perturbations at once. The authors call this a virtual population and say one pass evaluates at least three orders of magnitude more members than weight-space evolution strategies.
3. Score each token. Each token's reward is the change in its loss, plus discounted changes in later tokens that the noise can reach through attention.
4. Turn rewards into a gradient. Averaging reward-weighted noise estimates the error at each layer's output, and multiplying by the layer's input gives the weight gradient.

To limit interference, layer types are perturbed in separate passes, attention internals are scored by their effect on the attention output, and the language-model head is perturbed directly on cached logits. Hyperparameters were tuned by grid search.

What the paper actually shows

The setup is modest by frontier standards: GPT-style transformers on FineWeb with a 4,096-token tokenizer, batches of 16k tokens, one epoch, and a base model with 8 layers and width 512, three seeds per configuration.

Versus backprop with SGD. At 100k and 1M training tokens, Dust ends below backprop once it has a few hundred to about a thousand draws. At 10M tokens, the fitted limit lands just above backprop. At 20M tokens the power-law fit gives 4.431, with a wide 95 percent interval of 3.89 to 4.58, compared with 4.633 for backprop. The authors warn that this fit is loosely constrained because the curve is still falling.

With Adam. At 1M tokens, Adam improves both Dust and backprop. The paper's fitted limit is 5.248 for Dust versus 5.361 for backprop. The Noise README, in contrast, says Dust still trails backprop under Adam, so the two sources frame the Adam comparison differently. We read this as extrapolation versus measured runs and recommend checking the tables yourself.

Versus EGGROLL. EGGROLL at 16k draws still does not reach Dust at 64 draws. From 1M tokens upward the authors estimate Dust is roughly 1,000 to 10,000 times more efficient, and flag it as an extrapolation.

Model size. Across 2M, 7.3M, 38M and 243M parameters at 10M tokens, larger models were generally more population-efficient. That cuts against the usual belief that zeroth-order methods get worse as models grow.

Gradient alignment. Dust's gradient estimates align better with backprop's as population grows, and alignment stays fairly stable as tokens increase up to 1B, which the authors read as encouraging for scaling.

The claim: 1.76 hours to 1.6 seconds

The post circulating on X says the time to reach a fixed validation-loss target fell from 1.76 hours to 1.6 seconds in ten days, a 4,000x improvement, and that this came from the Noise speedrun. The arithmetic of the ratio checks out: 1.76 hours is about 6,336 seconds, and dividing by 1.6 gives roughly 3,960.

The endpoints do appear in the benchmark's records, with important context.

The README says that until October 22 the training set was 10M tokens, archived in a 10m/ folder, and the tracks now use 30M tokens with different targets. In the archive, 1 hour 46 minutes is about 1.77 hours, and dividing by 1.6 seconds gives about 3,975, so the headline ratio holds for that table.

Three caveats keep it from meaning "pretraining got 4,000x faster." First, the target was loss 4.2, far weaker than the 3.769 backprop with Adam reaches on the same data. Second, the fast entries are not Dust as published: their descriptions (stagewise forward-only training, closed-form and identity-path credit with exact values) point to newer, more structured credit-assignment methods that the rules allow, and the later records use token-sharded kernels and JAX. Third, the benchmark was reset to 30M tokens and a 3.5 target, where the best speedrun is still 1 hour 43 minutes. We also did not verify the "ten days" timeline in the viral post.

The honest limits the authors list

• Compute efficiency is not solved and needs orders-of-magnitude improvement.
• Many headline comparisons, including the efficiency multiples and the 20M-token limit, rely on extrapolation.
• Results come from one dataset (FineWeb) and models up to 243M parameters.
• The cosine analysis reaches 1B tokens, but the main training experiments do not.
• Looped or externally augmented architectures were not tested.
• Tuning needs grid searches, though settings largely transfer.

The authors also say Dust is not yet practical to replace backprop at current compute levels, and that it may surpass backprop with enough compute, a possibility rather than a demonstrated result.

What this means if you build or train models

For most readers, nothing changes this week. You will not swap Dust into a production training run. The value is in the direction:

• Research direction. If zeroth-order methods keep improving, training on hardware without efficient backward passes, or with non-differentiable pieces, becomes more realistic.
• Open benchmark. Noise gives a fixed model, data and score, with a rule that bans autograd, so progress is comparable. Submissions are pull requests that include logs, so claims are checkable. That is the right way to evaluate a speed claim.
• Scaling questions. The finding that larger models are more population-efficient is worth watching, since it runs against expectation.

If you work on small-model training or on-device learning, our posts on in-browser LLM fine-tuning with WebGPU and MicroLLM Lab's tiny browser models show where cheap training matters. For the wider debate about whether current methods are the path forward, see Yann LeCun on why LLMs are not AGI and Sakana's work on recursive self-improvement.

How to check the claim yourself

1. Open the Noise README and read the current leaderboard and the track definitions, including the 30M-token change.
2. Look for a pull request or log folder with config.json, metrics.jsonl and result.json for any claimed record. The README says the training script is the record.
3. Compare the validation-loss target in the claim with the target in the track; records are only comparable at the same target, data and hardware.
4. Read the paper's limitations section before quoting the efficiency multiples.
5. Treat any figure without a reproducible log as unverified.

What is still open

The authors say they are testing whether the efficiency gains scale by growing the training set from 10M to 30M tokens and finding which speed-up techniques fail under overfitting. We will revisit this post if the 30M-token leaderboard changes.

Related reading

• Sakana PC-ALM: training deep nets without backpropagation
• In-browser LLM fine-tuning with WebGPU
• MicroLLM Lab: tiny LLMs in the browser
• Yann LeCun on LLMs, AGI and JEPA
• Sakana MASS and recursive self-improvement
• Infinite-parameter LLMs and hypernetworks
• Official: Dust paper page and Noise repository

Figures are from the Q Labs paper page and the Noise README as of October 22, 2026, and may change as the leaderboard updates.