explainx.ai350K+ learn from us
TrendingAI News TodayPathwaysSkills
Pricing
explainx.ai

Upskill in AI — 16 free pathways, live workshops & bootcamps, and 50+ courses from practitioners. Plus the skills, tools, and MCP servers to practice on.

follow us

follow on google

Add explainx.ai as a preferred source

corporate training

support@explainx.ai

get started

Find your pathTake Free Evaluation

community

Join the community

learn

mind: share how you thinkpathways — start freeworkshopsbootcampscoursescompare Explainxcertificationsmock testsexplainx universitycorporate traininglearn skills & mcp

discover

skillsmcp serversexplainx mcpestacktoolsmdx readeragentsllmsdesignsdictionarypeopleagi trackerfelony benchranks

company

aboutvisionmissionteaminstructorsteach on explainxpartnershipscommunityhackathonscareers

content

daily AI newsstate of AI — live resultsblogreleasespromptsgeneratorsresource libraryfor LLMsexplainx.ai kids

solutions

all solutionsdeveloper upskillingmarketing upskillingproduct manager upskillingleadership upskilling

newsletter · weekly

Get AI news, tools, and insights in your inbox.

supportcontactprivacytermsdata rightshow we create contentsubmission guidelines

© 2026 AISOLO Technologies Pvt Ltd

explainx.ai

On this page

  • TL;DR numbers from Evolving Machines
  • Why optimizer cost re-enters the chat in late 2026
  • How Echelon differs mathematically (plain language)
  • Hardware splits matter
  • Open models called out in the tables
  • 1.3B end-to-end experiment
  • Who Evolving Machines is
  • What to watch before you swap optimizers
  • The savings formula in practice
  • When Echelon is worth trying first
  • Newton-Schulz versus low-rank: intuition without the cube
  • Open-source release expectations
  • Classroom connection: optimizers in agent fine-tuning
  • Rank selection cheat sheet from Evolving Machines tables
  • Communication-aware expectations
  • Lean proof: why it matters for procurement
  • Related reading
← Back to blog

explainx / blog

Echelon Optimizer: 10× Faster Muon Steps With a Lean Proof

Training, Optimizers, Research, Open Source, MoE

Part of AI Research

Evolving Machines published Echelon on Nov 16, 2026: a low-rank Muon-style optimizer with up to 10.5× faster steps on 70B layers, 35% shorter MoE training steps, and a machine-checked Lean convergence proof.

Nov 16, 2026·8 min read·Yash Thakker
add explainx.ai
go deep
Echelon Optimizer: 10× Faster Muon Steps With a Lean Proof

Evolving Machines published the Echelon optimizer on November 16, 2026, arguing that Muon-quality spectral updates can run at low-rank prices—and backing the claim with layer timing tables, 1.3B training seeds, MoE projections on DeepSeek V4.1 Flash, and a Lean-checked convergence proof. The full post is aimed at teams whose optimizer step, not matmul, has become the training tax on wide MoE stacks.

Muon climbed the leaderboard because it replaces each singular value in momentum with 1 while keeping singular vectors, approximated via Newton–Schulz iterations. Echelon’s pitch: keep the alignment benefits of polar-style updates but spend O(mnr) work by flattening only the top r singular values and scaling the tail—cutting the cubic term that dominated 70B-class steps.

TL;DR numbers from Evolving Machines

table · 4 cols
ScenarioMuonEchelonNotes
Optimizer step, 70B-class layers33.9 s3.23 sOne RTX PRO 6000, batched, rank 64
Optimizer inside 1.3B runbaseline2.5× faster optimizer14% shorter training steps
Validation @ 1.3B (3 seeds)4.3757 ± 0.01094.3751 ± 0.0115Within seed noise
DeepSeek V4.1 Flash stepbaseline~35% less time16× B300, rank 32, comm excluded
Daily briefFrom Yash · 350K+ students taught

Catch up on AI

One AI story explained in plain English, plus one thing to try. Every day, free. Unsubscribe anytime.

Why optimizer cost re-enters the chat in late 2026

Matrix optimizers like Muon, SOAP, and Kron won mindshare because they exploit layer structure. Independent work such as Raschka’s rope guide and community skepticism pieces on 2× speedup myths remind readers that reported optimizer gains shrink with scale. Echelon’s authors lean into that debate with explicit tables for GPT-2 small through 500B+ MoE shapes, separating optimizer share f from speedup s so readers can compute expected step savings instead of trusting a headline.

Mixture-of-experts routing diagram for MoE optimizer updatesMixture-of-experts routing diagram for MoE optimizer updates

For MoE, the post stresses a structural fact: every expert gets an optimizer update even though each token touches few experts, pushing optimizer share f toward 30–50% on large routed models. That is why DeepSeek V4.1 Flash rows show ~35% step-time reductions while tiny dense models might save only ~1%.

How Echelon differs mathematically (plain language)

Muon targets the polar factor of momentum—directions preserved, all singular values forced to one—implemented with costly iterations. Echelon only fully flattens the top r singular values and down-weights the rest, aiming for most of Muon’s alignment without 30n³ FLOPs per wide layer.

The authors connect the update to steepest descent in spectral norm, citing standard polar-factor optimality: among matrices with ‖O‖₂ ≤ 1, the polar factor maximizes alignment ⟨M,O⟩. Echelon trades exact polar maps for bounded low-rank estimates they prove stay stable even when internal guesses are wrong—then they machine-check the proof in Lean, rare for a productized optimizer blog.

Hardware splits matter

The same rank can yield 10.7× optimizer speedup on an RTX PRO 6000 but only 3.7× on a B300 for DeepSeek V4.1 Flash shapes, because Muon’s dense matmuls benefit more from Blackwell silicon. Echelon’s savings shrink when the baseline is already fast—an important planning note for teams choosing cloud GPU SKUs on Baseten-style efficiency guides versus bare-metal training.

Open models called out in the tables

Evolving Machines timed real layer shapes for several 2026 open stacks—Qwen3.8-27B, Nemotron 3.5 Lightning 30B-A3B, Mistral Small 4 119B-A6.5B, Reflection Beam 501B-A23B, and DeepSeek V4.1 Flash 552B-A16B—with GPU counts chosen as the fewest cards that hold training state. These are projections tied to measured optimizer microbenchmarks, not full retrains sponsored by each lab.

Still, the list gives budget owners a shortcut: if you fine-tune Beam-class MoE, optimizer share near 28–32% means a ~10× microbenchmark can shave double-digit percent off wall-clock before accounting for communication—which Echelon explicitly excludes in several rows.

1.3B end-to-end experiment

The dense 1.3B run is the integrity check: three seeds each for Echelon and Muon, 32k tokens/step on one GPU. Echelon reports 14.2% training-step savings when optimizer share is 23.5% and optimizer speedup 2.5×. Validation loss lands inside seed noise, while AdamW baselines in the same table look worse (5.184 with one seed), consistent with why teams adopted Muon in the first place.

Who Evolving Machines is

The company sells continuous evaluation infrastructure for self-improving agents—closer to Together’s agent eval narrative than to a foundation lab. Publishing a training optimizer signals they eat their own long-horizon simulation costs and want open trainers to afford the same loops.

What to watch before you swap optimizers

  1. GitHub drop — post promises fused kernels “soon”; production teams should wait for tagged releases.
  2. Communication — MoE rows exclude all-reduce time; multi-node runs may dilute savings.
  3. Rank selection — tables show speedups fall as rank approaches width/16; auto-tuning may be manual at first.
  4. Community replication — independent repros on Moonshot science pivots scale models will test whether 1.3B loss matching holds at frontier width.

The savings formula in practice

Evolving Machines gives a simple identity: if optimizer work is fraction f of a step and Echelon speeds the optimizer by s, expected step-time savings are f(1 − 1/s). Plug in their DeepSeek V4.1 Flash row (f ≈ 0.473, s ≈ 3.7 on B300) and you land near 34.6%, matching the blog table.

Use the formula before you believe 10.5× microbench headlines:

table · 3 cols
If optimizer share f is…And speedup s is…Step-time saved
10% (small dense)2.5×~6%
25% (1.3B run)2.5×~15%
32% (large MoE)10×~29%
47% (DeepSeek Flash)3.7×~35%

Small models see small wins; MoE sees large wins because optimizer share grows with expert count.

When Echelon is worth trying first

Prioritize Echelon experiments when:

  • Optimizer profiling shows more than 20% step time in AdamW/Muon on wide layers.
  • You train MoE with many experts per token and frequent optimizer passes.
  • You rent RTX PRO class GPUs where their 10× microbenchmarks peak.
  • You need Lean-checked theory for internal ML safety reviews.

Deprioritize when:

  • Communication dominates (multi-node all-reduce bound).
  • Sequence length experiments are data-loader bound, not optimizer bound.
  • You lack release tags and cannot audit fused kernels in your environment.

Newton-Schulz versus low-rank: intuition without the cube

GPU cluster training concept for optimizer step comparisonsGPU cluster training concept for optimizer step comparisons

Muon’s Newton-Schulz iterations approximate polar(M) with O(n³) work per wide layer. Echelon truncates which singular directions get flattened, so cost tracks rank r instead of full width.

Think of Muon as forcing every singular direction to carry equal step energy; Echelon spends big moves only on top directions and dampens the tail. The Lean proof tries to guarantee steps stay bounded even when rank estimates mis-rank noisy gradients—important when MoE routing makes gradients spiky.

Open-source release expectations

The post promises GitHub code, Python package, and a fused kernel “soon.” Until tags ship, treat Echelon as research-preview: replicate 1.3B loss curves on your hardware before touching production pretraining.

Watch Evolving Machines’ feeds alongside Edward Witten lattice work with Claude as examples of small labs publishing rigor while frontier pretraining stays expensive.

Classroom connection: optimizers in agent fine-tuning

Not every team runs 500B MoE, but agent fine-tuning on 8–70B models still hits optimizer tax when LoRA ranks climb. Echelon is not LoRA—it changes base weight updates—yet the rank knob language overlaps. If you fine-tune coding agents, profile whether optimizer or backward wins before adopting new methods; Replica skill harnesses often hide training configs from app developers.

Rank selection cheat sheet from Evolving Machines tables

Their DeepSeek V4.1 Flash optimizer speedup table shows diminishing returns as rank grows:

table · 3 cols
RankB300 speedup vs MuonRTX PRO 6000 speedup
323.7×10.7×
643.4×7.3×
1282.8×3.0×
2562.0×2.3×

Takeaway: higher rank is not free quality—after ~64, you pay more compute per step for smaller relative gains on Blackwell. Start low rank on B300 clusters, higher rank on workstation GPUs if their data holds on your shapes.

Communication-aware expectations

Several Echelon rows say communication not counted. On multi-node MoE, all-reduce and expert parallel traffic can erase optimizer wins if you only microbench single-GPU steps. Profile whole step with Nsight or PyTorch profiler before planning 33% shorter runs.

Lean proof: why it matters for procurement

Enterprise AI labs increasingly ask vendors for formal guarantees on anything touching safety-critical fine-tunes (finance, biotech). A Lean-checked convergence claim does not replace empirical loss curves, but it gives review boards a checkable artifact—rare in optimizer marketing. Treat it like SLA appendices: useful in RFPs, irrelevant if code never ships.

When GitHub tags land, pin commit hashes in your training run manifests so compliance can reproduce exact optimizer semantics months later.

Share profiler screenshots with your team when Echelon wins; optimizer shifts are invisible in loss logs but obvious in step-time histograms.

Related reading

  • Rope and training-stability explainers
  • MMFlash speculative decoding costs
  • Moonshot Kimi benchmark flattening
  • DeepSeek chat client limits
  • Baseten frontier inference efficiency
  • Together AI agent evaluations

Benchmarks and proofs described here come from Evolving Machines’ November 16, 2026 Echelon post unless noted.

Spotted something out of date? Let us know.
Daily briefFrom Yash · 350K+ students taught

Catch up on AI

One AI story explained in plain English, plus one thing to try. Every day, free. Unsubscribe anytime.

  • ✓ Free
  • ✓ One email a day, no spam
  • ✓ Unsubscribe anytime
Yash Thakker

Written by

Yash Thakker

Yash is an AI expert with over 300K learners. Join his workshops →

View Yash Thakker in People in AI →

Related posts

Nov 11, 2026

InfraRSI: Agent Harness That Tunes LLM Training for 2.75× Speedups

InfraRSI applies recursive-self-improvement ideas to distributed training infra: an agent locks invariants, profiles noise, co-optimizes launch knobs and code patches, and re-verifies against baselines—lifting mean MFU from 17.4% to 30.0% in the paper.

Nov 9, 2026

StepFun P²-MOPD: Multi-Teacher Distillation With Parameter Priors on Qwen3

The P²-MOPD paper on Hugging Face and arXiv adds expert task vectors as learnable parameter priors during multi-teacher on-policy distillation. Authors report faster, more balanced domain recovery on Qwen3 1.7B and 4B models, with code at SinuoWang/P2-MOPD.

Nov 16, 2026

PaperOrchestra: Google Research's Multi-Agent LaTeX Paper Writing Framework

PaperOrchestra is a Google Cloud AI Research open-source pipeline that converts raw research materials into conference-template LaTeX, with specialized agents for outline, literature, writing, refinement, and plotting.