Goodfire Neurons as an Interpretable Basis: 2.9% Loss After Ablating 92% of MoE Units

Interpretability vendors need measurable sparsity, not just pretty dashboards. On November 10, 2026 Goodfire published “Neurons as an Interpretable Basis”, arguing that MoE neurons in Qwen3.6-35B-A3B are already functionally sparse: keep the strongest 320 routed experts per layer and loss rises only 2.9% while 92.2% of activations are zeroed. Ryan Schaeffer and others on social feeds highlighted the figure as a sign that most routed compute is dispensable; this post walks through what Goodfire did, how they recovered benchmarks, and what builders should cautiously take away.

We previously covered Goodfire’s Inside-Out monitors for rogue agents and the “neural networks are already clean” essay. The November paper adds ablation tables, a 100M-token recovery protocol, and a public neuron atlas for thousands of units.

TL;DR

Method in plain language

SwiGLU neuron diagram from Goodfire interpretability research Image: Goodfire

Goodfire combined three toolkit pieces on Qwen3.6-35B-A3B:

1. Sparsification — per layer, keep only the top 320 routed neuron activations out of 4,096 each token step.
2. Attribution — measure causal influence of neuron modes with their existing probe stack.
3. Research agents — LLM judges write one-sentence descriptions for firing patterns.

The headline 2.9% loss bump comes from the top-320 mask row in their public table (Δ loss +0.054 on base loss 1.886). Weaker ablation bands (P25–P75 trims) barely move loss, supporting the claim that weak activations contribute little.

Why benchmarks dipped before recovery

Goodfire reports the sparse model looped and over-checked in chain-of-thought—even when final answers appeared in text. Claude Opus 5.5 agents classifying rollouts saw correct facts buried under redundant verification.

Recovery fine-tuning used:

• SFT distillation — 91M tokens matching the dense teacher’s next-token distribution on thinking traces plus 20% general web text.
• RL with verifiable rewards — multiple-choice and short-answer sets (MMLU-auxiliary, MedMCQA, ARC) with group-relative advantages and a KL penalty toward the dense model.

Checkpoint step 100 reportedly beat the teacher on SuperGPQA while reducing “cap-hits” from endless reasoning. That is evidence the capacity loss is partly behavioral, not purely knowledge deletion.

Interpretability claims: single-mode neurons

Goodfire says 69% of 2,734 sampled neurons show one dominant causal mode, and short descriptions predict firing ~95% on strong activations in their binary judge setup. Examples in the paper include social “dance floor” tokens and camelCase code identifiers.

Skeptics should note: 95% on a forced two-choice test is not human-level understanding. It is a consistency check that descriptions align with activation clusters—useful for tooling UX, not proof of mechanistic completeness.

How this connects to safety and agent monitoring

If most routed neurons are expendable, monitoring strong activations becomes cheaper—a theme in our rogue agent monitor post. Teams running Anthropic strong safety evals might eventually ask vendors for sparse activation dashboards alongside logits-based tests.

Goodfire sells interpretability products; treat numbers as company-reported until third parties replicate on Llama, Mistral, or closed APIs.

What builders should do

Researchers: Download the neuron atlas linked from Goodfire’s page and try describing units in your own fine-tunes—compare judge accuracy against random baselines.

Pruning enthusiasts: Do not equate research masking with production pruning. Goodfire retrained with 100M tokens and RL loops; your serving graph may lack that budget.

Compliance teams: Sparse interpretability may help document internal concepts for EU AI Act technical files, but it does not replace red-team logs or ExploitBench-style evals.

Reading the ablation table like a systems engineer

Goodfire chart of neuron ablation damage and restoration on Qwen MoE Image: Goodfire

Goodfire’s public numbers compare several masking strategies:

Two lessons jump out. First, random ablation destroys the model, so the surviving signal is structured, not noise. Second, aggressive percentiles (P5–P95 bands) still move loss less than random masks—supporting functional sparsity rather than arbitrary width.

When Ryan Schaeffer and others call a 2.9% loss figure “brutal” on social feeds, they usually mean training compute was mostly wasted on weak experts, not that serving can drop 92% of neurons for free without retraining.

Replication checklist for skeptics

Before you cite Goodfire in a board deck:

1. Download Qwen3.6-35B-A3B weights and reproduce top-320 masking on a small eval slice.
2. Compare SuperGPQA before and after the 100M-token recovery recipe—without RL, loops may persist.
3. Test description accuracy on 50 random strong neurons; report your own judge agreement rate.
4. Repeat on a non-Qwen architecture Goodfire has not published.

Until step four succeeds externally, treat cross-model generalization as an open hypothesis.

Links to mechanistic interpretability debates

The paper pushes back on superposition pessimism: if neurons are already sparse and describable, feature dictionaries might be optional for some monitoring tasks. That challenges parts of the broader sparse-autoencoder narrative—but does not obsolete SAEs overnight, because polysemantic neurons may still appear in weaker activation bands Goodfire discards.

For safety teams, the actionable slice is monitor the top activations cheaply. Our Goodfire inside-out monitor post covers a different product surface with similar philosophy.

Product and licensing context

Goodfire sells interpretability tooling to labs and enterprises. Their research posts double as capability advertisements. That is not disqualifying, but it means incentives favor positive results on Qwen3.6. Peer review and third-party replication remain the filter.

Neuron atlas and downstream products

Goodfire links a neuron atlas with thousands of per-unit pages. For interpretability engineers, that is the practical deliverable: browse strong activations, read proposed descriptions, file false positives. For executives, it is evidence the company can productize monitoring beyond blog posts.

Compare with Gemini activation probes and Anthropic R&D automation metrics—different stacks, same buyer question: can we see what the model is doing before users complain?

MoE-specific lessons for routing research

Mixture-of-experts models route tokens to subsets of experts. Goodfire’s mask keeps 320 of 4096 routed experts per layer. That implies MoE routing is extremely peaked: most experts rarely matter for a given token. Routing research teams should ask whether training cost could target those peaks earlier, though Goodfire does not claim training recipes—only observational ablations after training.

When sparsity claims fail

The paper is Qwen-specific. Dense transformers, conv models, and Mamba-style state models may not show 92% disposable mass. Treat the 2.9% headline as one architecture snapshot, not a scaling law.

Enterprise monitoring workflow sketch

1. Export top-activation neurons weekly from production fine-tunes.
2. Auto-generate descriptions; humans audit 5% sample.
3. Alert when high-influence neurons fire on policy-sensitive prompts.
4. Correlate with agent incident trackers when actions misfire.

That workflow is aspirational—Goodfire sells pieces of it—but the November numbers give FinOps a reason to fund pilot budgets.

SuperGPQA and reasoning-specific recovery

Goodfire emphasizes SuperGPQA gains after RL because the sparse model initially over-thought without committing to answers. That pattern matches user complaints about reasoning models that loop. If your product wraps chain-of-thought, monitor token loops separately from loss—ablation may increase verbosity even when loss looks flat.

Relationship to pruning and distillation hype

Vendors marketing 50% sparsity speedups should not cite Goodfire’s 2.9% loss without also citing the 100M-token recovery and RL pass. Serving speed was not the paper’s claim; interpretability was.

Academic context

The work extends interpretability beyond SAE feature dictionaries toward native neurons. Critics may demand comparisons on GPT-style dense models and open Llama checkpoints. Until those land, cite the paper as MoE evidence, not universal law.

Glossary for non-specialists

Ablate means force an activation to zero during the forward pass. Routed neurons in MoE are expert units selected per token. Loss here is standard language-model cross-entropy on held-out text, not user-facing quality. Sigma in Goodfire tables expresses how unlikely a layout match is under chance—higher is stronger evidence in their framework.

What Ryan Schaeffer’s social posts emphasized

Aggregators surfaced Schaeffer commentary on the 2.9% loss figure as “brutal” training inefficiency—interpreted as most routed expert compute being expendable. Treat social paraphrases as engagement, not independent replication. Always cite the primary research page tables when making procurement decisions.

Peer review may arrive later; until then, treat neuron descriptions as hypothesis generators for red-teamers, not compliance proof. Run your own spot checks on neurons flagged “high influence” before wiring them into production monitors.

If your team benchmarks interpretability vendors, ask whether they report holdout neuron accuracy the way Goodfire reports 95% on strong activations—without that, marketing slides reuse the same headline metrics forever.

Related safety tooling on explainx.ai

• Inside-out monitors
• Anthropic strong safety evals guide
• OpenAI Daybreak red-team factory
• AI2 XYBench goal miss benchmark

Related reading on explainx.ai

• Goodfire Inside-Out rogue agent monitors
• Neural networks already clean essay coverage
• GLM local vLLM tensor parallel guide
• AI2 XYBench goal-miss benchmark
• Apple alignment limits paper
• Shanghai Sage-A1.5 open MoE

Primary: Goodfire research page. Model background: Qwen3.6-35B-A3B model card (35B total parameters, 3B active per token, mixture-of-experts).

Tables and training protocol summarized from Goodfire’s November 10, 2026 publication.