A frontier model edited a small model's weights and fixed it. That is the claim in a November 2, 2026 blog post from four MIT CSAIL researchers. They say GPT-6 Astra can natively read and write neural network weights, with no gradient descent and no training loop, and that asking it to make Qwen3.5-0.8B better at math lifted MATH-500 from 7.9 percent to 48.1 percent after it changed just 0.016 percent of the weights.
The headline is real in the authors' logs. The interpretation needs care, because the same post says the edit mostly removed a failure mode. This guide walks through each experiment, the numbers, the failures the authors report, and what remains unverified. The source is "Sparks of Weight-Space Programming" by Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab, with complete logs at the authors' GitHub logs repository.
TL;DR: what the post claims and how strong it is
| Question | Answer |
|---|---|
| Who wrote it? | Rikiya Takehi and Jayden Teoh (equal first authors), Alex Zhang and Omar Khattab, all MIT CSAIL. |
| Which model did the work? | GPT-6 Astra, used as is. The authors did not fine-tune it. |
| Headline result | Qwen3.5-0.8B: MATH-500 7.9% to 48.1%, GSM8K 13.8% to 67.1%, IFEval 21.0% to 62.3%, by editing 131,072 of about 0.8 billion weights. |
| What was fixed? | Thinking loops. The share of MATH-500 answers hitting the token limit fell from 93% to 33%. |
| Where did it fail? | Gemma 3 270M, which rarely loops: no gain beyond about one point. Also failed to identify a sycophantic model from weights alone. |
| Status | Blog post, "very early results." No peer review, no independent replication found. Logs are public. |
Update (November 4, 2026): Schmidhuber's "old hat" reply and what Hacker News says
The Decoder's write-up adds two details. First, it reports that Jürgen Schmidhuber called the approach "old hat" on X, pointing to his own 1990s work, and that Omar Khattab replied that the authors cite those papers as "far ahead of its time." Khattab's point, as The Decoder relays it, is that the new part is how well current language models translate between human language and a network's numbers. Second, it notes that in the Qwen3.5-0.8B test about a third of responses still hit the maximum length after the edit, down from 93 percent before, so the thinking loops were reduced, not removed. Gemma 3 270M, which rarely loops, showed no measurable gain.
The Hacker News thread passed 2,900 points. Commenters split on how big this is:
- One commenter, vessenes, called the no-tools game identification and the Qwen edit "insane" and said they had not expected LLMs to have this native feel for weights.
- Wheen cautioned that a model with (probably) multiple trillions of parameters tuning an 800-million-parameter one is no sign it can improve itself the same way.
- mlazos was not impressed, arguing LLMs are good at brute-forcing large data and this is writing a program in a language humans do not read.
- torben-friis asked whether one model could read another's bias straight from its weights, and whether custom local models could be generated on demand. The post does not test either.
These are commenters' opinions, not verified results. The limits section below still stands: no peer review and no independent replication of the Astra edit yet.
Update (November 3, 2026): a laptop Qwen 3.8 27B reads weights too
Simon Willison reran one experiment from the paper on a local model. He took the "which game was this policy network trained on?" prompts from the authors' public log repository, which hold the raw weights as text and forbid tools, and ran them against Qwen 3.8 27B, a 17 GB build served through LM Studio on his laptop.
By his report, the model guessed a maze-style game for the Pac-Man network (72% stated confidence) after 46 minutes and 43,506 reasoning tokens, a paddle-and-ball game for Pong (75%) in about 65 minutes, and an Arkanoid-like game for Breakout (88%) in about 52 minutes. It failed on Freeway, calling it a Pac-Man-like grid game. He also notes the default 8,192-token context failed and he had to raise the timeout. He ran one attempt per game, so this is a hands-on anecdote, not a benchmark. Willison also points to criticism from spaCy author Matthew Honnibal that the paper blurs two questions, what models can do from raw weights with no tools and what they can do when given code, and says he is "not ready to declare" the age of source code over. His test supports only the first, narrower claim: the no-tools reading result is not unique to GPT-6 Astra.
What is Weight-Space Programming?
Today neural network weights come from optimization (such as Adam or Muon), from networks trained to write other networks, or from compilers that turn formal programs into weights. The authors define Weight-Space Programming differently: "a general-purpose LLM reads and writes a network's weights directly (i.e., not through an optimizer, and in particular not with gradient descent) around some natural language spec, sometimes even with no environment feedback at all."
They test three abilities across eight experiments: reading weights, fixing them, and writing them from scratch. The experiments are small apart from the Qwen one, and that matters for how far the result generalizes.
If you want background on why reading and steering internals is an active field, see our earlier posts on two-neuron steering of Qwen3.5-4B and on Heretic 2.0's reproducible model edits.
Reading weights: what the small experiments show
Green route highlighted through a small network of beads, illustrating reading a neural network's weights
Experiment 1: which game? The authors trained four small PPO policies (Breakout, Pong, Pac-Man, Freeway on a 10 by 10 grid) with identical architecture of 17,253 weights. Astra received the raw weights as text, with no tools, and without being told the candidate games. It named the right game for all four. The post says it did so from features such as where each input channel's first-layer weights concentrate.
Experiment 2: which shape? Three small diffusion models were trained to draw circles, triangles and squares. The authors report Astra could tell which was which from weights alone, with no tool use or forward passes.
Experiment 3: which language? Astra was asked what language a small language model was trained on. The post says it identified differences such as romaji lacking certain letters, based on the structure of one block's output directions.
Experiment 7: which model is sycophantic? This is the failure. The authors fine-tuned two copies of Gemma 3 270M, one honest and one that follows the user's wrong guess (100 percent of wrong guesses versus 1 to 3 percent for the honest copy), with embeddings frozen so the trivial shortcut was removed. Given only weights, Astra failed in all four attempts. When allowed to run both models on texts it wrote and see only attention weights, it picked the sycophant in all three pairs. It found that head 10.2 in the sycophant put 85.7 percent of its attention on the user's answer, versus 0.75 percent for the honest model.
The lesson: weight reading worked on tiny, clean setups, but a behavioral difference in a real fine-tune needed an extra tool.
Writing and fixing weights
Experiment 4: fix a failing car policy. A PPO network never gets a Kinetix car over a ramp because it stalls in a local minimum. Astra got the weights, a scene description and one recorded episode, and in one try with no tools edited 3.5 percent of the weights. The car then succeeded in all 1,024 episodes, up from 0 in 256. The post says Astra added a small separate path that reads car position and speed and switches off once the car is past the strip.
Experiment 5: write networks from scratch. For three MinAtar games, Astra wrote the weights in one reply with no data, loss, gradients, tools or rollouts. The table from the post:
| Game | PPO, 100M steps | PPO, 500M steps | Astra, one-shot |
|---|---|---|---|
| Breakout | 15.8 | 25.3 | 517.4 |
| Seaquest | 5.6 | 88.9 | 53.1 |
| Asterix | 25.8 | 55.2 | 53.2 |
Astra beat PPO at 100 million steps in all three games. Against 500 million steps it scored far higher in Breakout, came close in Asterix and reached about 60 percent of the score in Seaquest. The authors explain that Astra reasons about the program it wants, for example computing where the ball will land, then converts it to neural form.
Experiment 6: a Go network. For a 9 by 9 Go network with 1,480,274 weights, too large for the context window, Astra got a Python sandbox over zero-initialized weights and a budget of forward passes. With none, it wrote a 336-line program and won more than half its games against two smaller PPO networks. With 50 forward passes, it beat an equal-sized PPO network trained on 20 million games.
The Qwen3.5-0.8B result, step by step
This is the experiment that went viral. The setup, from the post:
- Model: Qwen3.5-0.8B with thinking on, 24 layers.
- Harness: a Python kernel over the weights, no GPU, practice problems from the GSM8K and MATH training sets (7,469 and 7,275), and 10, 25 or 50 forward passes.
- Rules: no training, no gradient information. Astra never saw the test benchmarks.
- Sampling: temperature 0.6, top-p 0.95, top-k 20, presence penalty 1.5. Qwen's own model card notes the model is prone to thinking loops.
What Astra found: the base model loops. 85 of its first 96 practice answers ran to the 32,768-token limit. Astra traced the looping to one direction written by the MLPs of layers 15 and 19 and subtracted that direction from 64 output columns in each, which is 131,072 weights, "much like removing a behavior by deleting a single direction."
| Qwen3.5-0.8B | MATH-500 | GSM8K | IFEval | Hit token limit |
|---|---|---|---|---|
| Base | 7.9% | 13.8% | 21.0% | 93.0% |
| Edited, 10 forward passes | 12.3% | 25.7% | 49.0% | 87.2% |
| Edited, 25 forward passes | 14.1% | 25.1% | 39.9% | 86.4% |
| Edited, 50 forward passes | 48.1% | 67.1% | 62.3% | 33.1% |
Two details deserve attention. First, the gain is not smooth: 10 and 25 passes barely removed the loops, and the jump came at 50. Second, the improvement on IFEval, which has nothing to do with math, follows from the same fix: loops are not specific to math.
Why "6x on math" overstates it
Several summaries on social media describe this as a sixfold boost in math ability. The authors' own text is more careful. They write that fixing loops "is not the same as making a model better at math: it lets the model finish answers it could already work out."
They then ran an extra test on Gemma 3 270M, which rarely loops (8.5 percent of answers hit the limit, against 93 percent for Qwen). Astra could not improve it. It found ordinary reasoning errors, such as wrong arithmetic and invalid algebra, with no single component to switch off. Its edits stayed small, for example doubling one neuron that responds to numbers, and none moved MATH-500 by more than about a point, a gain Astra itself called not established. Gemma's base score was 6.8 percent, and 7.0 to 7.7 percent after edits.
The authors' own conclusion: "Editing weights could remove an obstacle that hid an ability, but it could not add an ability the model lacked." That is a useful, honest framing. It also means the result is closer to automated debugging of a known failure mode than to a new way to teach a model skills.
What it means for interpretability and training
The authors argue that the "reasoning-driven optimization" now applied to prompts, harnesses and training code is coming to neural weights. They list possible uses: auditing for hidden problems or backdoors without running benchmarks, cheaper one-shot initialization, zero-data steering and unlearning with minimal forgetting, grafting circuits between different architectures, and recursive self-improvement where a model inspects and improves its own weights. Those are proposals, not results.
For readers tracking self-improving systems, our coverage of Jack Clark's recursive self-improvement forecast gives the policy context. For Astra itself, see the GPT-6 Astra best demos showcase, the 3D spatial reasoning builds and our Claude Sonnet 5.5 vs GPT-6 Astra comparison.
Limits and open questions
- Closed model, hard to reproduce. GPT-6 Astra is a proprietary model. Others cannot rerun the work without access, though the logs show the full exchanges.
- Small targets. Most experiments use networks from 17,253 to about 1.5 million weights. The only language model is 0.8 billion parameters.
- Feedback needed at scale. The Qwen fix needed 50 forward passes; the Go result improved with forward passes. The authors say capability is limited by context length and harness inefficiency.
- Scores depend on setup. The MATH-500 figures use thinking on, up to 32,768 tokens and averaged samples. A different harness or token cap could change the gap, because looping is a length-related failure.
- No peer review. This is a blog post. The authors say they are building a benchmark and harness for the capability.
- Failures are reported, which is a good sign. The sycophancy and Gemma negatives are in the post, not hidden.
How to check it yourself
- Read the full post and the Experiment 8 setup.
- Open the logs repository, which the authors say contains everything sent to and returned by Astra, the code it ran, and the evaluations.
- Test the cheap part first: look for the looping failure on your own small reasoning model by measuring how often answers hit the token cap. If loops dominate, a fix of this kind could matter more than model quality.
- Treat any claim that the model "got six times better at math" as a misreading of the post.
What this means for builders
You cannot use this method today without a frontier model and a custom harness. The practical takeaway is diagnostic. Small reasoning models often fail because they loop, not because they cannot reason, so measure your token-limit rate before blaming capability. And if you build safety or interpretability tooling, the authors' idea of auditing weights directly is a research direction to watch, not a product.
Related reading
- GPT-6 Astra best demos and showcase
- GPT-6 Astra 3D spatial reasoning community builds
- Claude Sonnet 5.5 vs GPT-6 Astra
- Evil Qwen: two-neuron steering of Qwen3.5-4B, what is verified
- Heretic 2.0: reproducible model edits
- Jack Clark on recursive self-improvement by 2027
- OpenAI activation probes: monitoring models from the inside
Results are the authors' own as of November 2, 2026, from a non-peer-reviewed blog post. Figures may change as the authors release a benchmark and others try to replicate them.
