Evolving Machines published the Echelon optimizer on November 16, 2026, arguing that Muon-quality spectral updates can run at low-rank prices—and backing the claim with layer timing tables, 1.3B training seeds, MoE projections on DeepSeek V4.1 Flash, and a Lean-checked convergence proof. The full post is aimed at teams whose optimizer step, not matmul, has become the training tax on wide MoE stacks.
Muon climbed the leaderboard because it replaces each singular value in momentum with 1 while keeping singular vectors, approximated via Newton–Schulz iterations. Echelon’s pitch: keep the alignment benefits of polar-style updates but spend O(mnr) work by flattening only the top r singular values and scaling the tail—cutting the cubic term that dominated 70B-class steps.
TL;DR numbers from Evolving Machines
| Scenario | Muon | Echelon | Notes |
|---|---|---|---|
| Optimizer step, 70B-class layers | 33.9 s | 3.23 s | One RTX PRO 6000, batched, rank 64 |
| Optimizer inside 1.3B run | baseline | 2.5× faster optimizer | 14% shorter training steps |
| Validation @ 1.3B (3 seeds) | 4.3757 ± 0.0109 | 4.3751 ± 0.0115 | Within seed noise |
| DeepSeek V4.1 Flash step | baseline | ~35% less time | 16× B300, rank 32, comm excluded |
Why optimizer cost re-enters the chat in late 2026
Matrix optimizers like Muon, SOAP, and Kron won mindshare because they exploit layer structure. Independent work such as Raschka’s rope guide and community skepticism pieces on 2× speedup myths remind readers that reported optimizer gains shrink with scale. Echelon’s authors lean into that debate with explicit tables for GPT-2 small through 500B+ MoE shapes, separating optimizer share f from speedup s so readers can compute expected step savings instead of trusting a headline.
Mixture-of-experts routing diagram for MoE optimizer updates
For MoE, the post stresses a structural fact: every expert gets an optimizer update even though each token touches few experts, pushing optimizer share f toward 30–50% on large routed models. That is why DeepSeek V4.1 Flash rows show ~35% step-time reductions while tiny dense models might save only ~1%.
How Echelon differs mathematically (plain language)
Muon targets the polar factor of momentum—directions preserved, all singular values forced to one—implemented with costly iterations. Echelon only fully flattens the top r singular values and down-weights the rest, aiming for most of Muon’s alignment without 30n³ FLOPs per wide layer.
The authors connect the update to steepest descent in spectral norm, citing standard polar-factor optimality: among matrices with ‖O‖₂ ≤ 1, the polar factor maximizes alignment ⟨M,O⟩. Echelon trades exact polar maps for bounded low-rank estimates they prove stay stable even when internal guesses are wrong—then they machine-check the proof in Lean, rare for a productized optimizer blog.
Hardware splits matter
The same rank can yield 10.7× optimizer speedup on an RTX PRO 6000 but only 3.7× on a B300 for DeepSeek V4.1 Flash shapes, because Muon’s dense matmuls benefit more from Blackwell silicon. Echelon’s savings shrink when the baseline is already fast—an important planning note for teams choosing cloud GPU SKUs on Baseten-style efficiency guides versus bare-metal training.
Open models called out in the tables
Evolving Machines timed real layer shapes for several 2026 open stacks—Qwen3.8-27B, Nemotron 3.5 Lightning 30B-A3B, Mistral Small 4 119B-A6.5B, Reflection Beam 501B-A23B, and DeepSeek V4.1 Flash 552B-A16B—with GPU counts chosen as the fewest cards that hold training state. These are projections tied to measured optimizer microbenchmarks, not full retrains sponsored by each lab.
Still, the list gives budget owners a shortcut: if you fine-tune Beam-class MoE, optimizer share near 28–32% means a ~10× microbenchmark can shave double-digit percent off wall-clock before accounting for communication—which Echelon explicitly excludes in several rows.
1.3B end-to-end experiment
The dense 1.3B run is the integrity check: three seeds each for Echelon and Muon, 32k tokens/step on one GPU. Echelon reports 14.2% training-step savings when optimizer share is 23.5% and optimizer speedup 2.5×. Validation loss lands inside seed noise, while AdamW baselines in the same table look worse (5.184 with one seed), consistent with why teams adopted Muon in the first place.
Who Evolving Machines is
The company sells continuous evaluation infrastructure for self-improving agents—closer to Together’s agent eval narrative than to a foundation lab. Publishing a training optimizer signals they eat their own long-horizon simulation costs and want open trainers to afford the same loops.
What to watch before you swap optimizers
- GitHub drop — post promises fused kernels “soon”; production teams should wait for tagged releases.
- Communication — MoE rows exclude all-reduce time; multi-node runs may dilute savings.
- Rank selection — tables show speedups fall as rank approaches width/16; auto-tuning may be manual at first.
- Community replication — independent repros on Moonshot science pivots scale models will test whether 1.3B loss matching holds at frontier width.
The savings formula in practice
Evolving Machines gives a simple identity: if optimizer work is fraction f of a step and Echelon speeds the optimizer by s, expected step-time savings are f(1 − 1/s). Plug in their DeepSeek V4.1 Flash row (f ≈ 0.473, s ≈ 3.7 on B300) and you land near 34.6%, matching the blog table.
Use the formula before you believe 10.5× microbench headlines:
| If optimizer share f is… | And speedup s is… | Step-time saved |
|---|---|---|
| 10% (small dense) | 2.5× | ~6% |
| 25% (1.3B run) | 2.5× | ~15% |
| 32% (large MoE) | 10× | ~29% |
| 47% (DeepSeek Flash) | 3.7× | ~35% |
Small models see small wins; MoE sees large wins because optimizer share grows with expert count.
When Echelon is worth trying first
Prioritize Echelon experiments when:
- Optimizer profiling shows more than 20% step time in AdamW/Muon on wide layers.
- You train MoE with many experts per token and frequent optimizer passes.
- You rent RTX PRO class GPUs where their 10× microbenchmarks peak.
- You need Lean-checked theory for internal ML safety reviews.
Deprioritize when:
- Communication dominates (multi-node all-reduce bound).
- Sequence length experiments are data-loader bound, not optimizer bound.
- You lack release tags and cannot audit fused kernels in your environment.
Newton-Schulz versus low-rank: intuition without the cube
GPU cluster training concept for optimizer step comparisons
Muon’s Newton-Schulz iterations approximate polar(M) with O(n³) work per wide layer. Echelon truncates which singular directions get flattened, so cost tracks rank r instead of full width.
Think of Muon as forcing every singular direction to carry equal step energy; Echelon spends big moves only on top directions and dampens the tail. The Lean proof tries to guarantee steps stay bounded even when rank estimates mis-rank noisy gradients—important when MoE routing makes gradients spiky.
Open-source release expectations
The post promises GitHub code, Python package, and a fused kernel “soon.” Until tags ship, treat Echelon as research-preview: replicate 1.3B loss curves on your hardware before touching production pretraining.
Watch Evolving Machines’ feeds alongside Edward Witten lattice work with Claude as examples of small labs publishing rigor while frontier pretraining stays expensive.
Classroom connection: optimizers in agent fine-tuning
Not every team runs 500B MoE, but agent fine-tuning on 8–70B models still hits optimizer tax when LoRA ranks climb. Echelon is not LoRA—it changes base weight updates—yet the rank knob language overlaps. If you fine-tune coding agents, profile whether optimizer or backward wins before adopting new methods; Replica skill harnesses often hide training configs from app developers.
Rank selection cheat sheet from Evolving Machines tables
Their DeepSeek V4.1 Flash optimizer speedup table shows diminishing returns as rank grows:
| Rank | B300 speedup vs Muon | RTX PRO 6000 speedup |
|---|---|---|
| 32 | 3.7× | 10.7× |
| 64 | 3.4× | 7.3× |
| 128 | 2.8× | 3.0× |
| 256 | 2.0× | 2.3× |
Takeaway: higher rank is not free quality—after ~64, you pay more compute per step for smaller relative gains on Blackwell. Start low rank on B300 clusters, higher rank on workstation GPUs if their data holds on your shapes.
Communication-aware expectations
Several Echelon rows say communication not counted. On multi-node MoE, all-reduce and expert parallel traffic can erase optimizer wins if you only microbench single-GPU steps. Profile whole step with Nsight or PyTorch profiler before planning 33% shorter runs.
Lean proof: why it matters for procurement
Enterprise AI labs increasingly ask vendors for formal guarantees on anything touching safety-critical fine-tunes (finance, biotech). A Lean-checked convergence claim does not replace empirical loss curves, but it gives review boards a checkable artifact—rare in optimizer marketing. Treat it like SLA appendices: useful in RFPs, irrelevant if code never ships.
When GitHub tags land, pin commit hashes in your training run manifests so compliance can reproduce exact optimizer semantics months later.
Share profiler screenshots with your team when Echelon wins; optimizer shifts are invisible in loss logs but obvious in step-time histograms.
Related reading
- Rope and training-stability explainers
- MMFlash speculative decoding costs
- Moonshot Kimi benchmark flattening
- DeepSeek chat client limits
- Baseten frontier inference efficiency
- Together AI agent evaluations
Benchmarks and proofs described here come from Evolving Machines’ November 16, 2026 Echelon post unless noted.
