Prime Intellect released Prime Mega MoE on October 29, 2026, a training version of DeepSeek's fused Mega MoE kernel. In its announcement, the company says it "extended DeepSeek's Mega MoE kernel with a fast, overlapping backward pass, making it usable for training," and reports up to 3.11x faster MoE kernels versus the default PyTorch-based implementation and up to 1.34x faster end-to-end training in its Prime-RL framework.
For anyone training or post-training mixture-of-experts (MoE) models, this is one of the more concrete open optimizations of the month. It also shows how fast open-source kernel work moves: DeepSeek published the forward kernel in its open-source DeepGEMM repository, and a lab outside DeepSeek has now added the backward pass. We explain the idea, the numbers and the limits below.
TL;DR
| Question | Answer |
|---|---|
| What is it? | DeepSeek's fused forward Mega MoE kernel plus a new overlapping backward kernel from Prime Intellect |
| Headline gains | Up to 3.11x faster MoE kernels; up to 1.34x (34 percent higher throughput) end-to-end vs the Prime-RL baseline |
| Vs TorchTitan | 48 percent faster steps on Qwen3-30B-A3B, 55 percent on Qwen3-235B-A22B (Prime Intellect's chart) |
| Hardware | Hopper and Blackwell GPUs |
| Data types | bfloat16; MXFP8 (Blackwell only); blockwise FP8 (Hopper only) |
| Source of numbers | Prime Intellect's own benchmarks, not independently reproduced |
Why MoE training is communication-bound
In an MoE model, each token is routed to a small subset of experts, and those experts live on different GPUs. Prime Intellect explains the cost: in every layer, tokens are sent to the GPU that holds their expert, the expert math runs, and the results are sent back. That happens in the forward pass and again in the backward pass. Normally these are separate operations, so the GPU sorts tokens, hands them to the transport layer and waits, runs the expert computation, hands results back and waits again. "All of that waiting wastes GPU time."
If you want background on routing and why MoE models are efficient, our posts on Meta's open-source dropless MoE kernels and Cursor's MoE megakernel work cover neighboring ideas.
What does the Mega MoE kernel do?
Mega MoE is a fused, overlapping kernel from DeepSeek's open-source DeepGEMM collection. Instead of a chain of separate operations, one kernel invocation handles both computation and communication directly, without round trips from the GPU to the CPU. In a single pass it routes tokens, runs the expert MLP math and combines results, overlapping compute with communication. While one group of tokens is in transit, the kernel is already computing on a group that just arrived, so the GPU is not idle waiting for data. A fused kernel can also skip intermediate tensors, because it uses GPU-only buffers as workspace.
A router hub linked to a ring of small expert blocks
DeepSeek's release, Prime Intellect notes, "provides only a forward implementation," optimized for Hopper and Blackwell GPUs. Training needs gradients, so the team built a fast overlapping backward kernel and swapped the standard MoE layers for calls to these custom kernels.
The benchmarks and what they measure
Prime Intellect publishes two setups. Read them before you extrapolate.
Full training step, Prime-RL with Mega MoE on vs. off. The setup was 8xH200 per node, Qwen3-30B-A3B on 4 nodes and Qwen3-235B-A22B on 16 nodes, sequence length 16,384, expert parallelism of 8 within a node plus FSDP over all GPUs, FP8, full activation checkpointing, fp32 master weights with AdamW, torch.compile with full graph, no offloading, real data and real routing. It reports the mean of steps 6 to 20 over 20 steps, excluding warmup. The headline result is up to 1.34x faster end-to-end.
Versus TorchTitan. In Prime Intellect's chart, time per training step falls from 3.07 s to 2.08 s on Qwen3-30B-A3B (48 percent faster, 5,337 to 7,886 tokens per second per GPU) and from 14.98 s to 9.65 s on Qwen3-235B-A22B (55 percent faster, 1,094 to 1,697 tokens per second per GPU).
In short, the TorchTitan comparison shows the larger gain (48 to 55 percent faster steps), the full Prime-RL step shows the smaller, more realistic gain (up to 1.34x), and the per-layer test isolates the kernel itself. Keep the three apart when you compare against your own training setup, because the surrounding model, parallelism and precision all change the result.
Per-layer test. On one node with 8 H200s, a single MoE layer (dispatch, experts and combine, without the router), 16,384 tokens per GPU, 128 experts, top-8 routing with random routing identical on both sides, FP8. This is where the 3.11x kernel figure comes from.
A subtle point: the 55 percent figure compares a Prime-RL stack using Mega MoE against TorchTitan, a different framework, so it mixes the kernel's effect with other differences between the stacks. The cleaner like-for-like number is the 1.34x versus the same framework with the kernel off. Both are vendor-run, and we have not seen third-party reproductions yet.
Who should care?
- Teams post-training open MoE models, such as Qwen3 variants, especially with RL where step time dominates. Prime Intellect builds Prime-RL and its stack for open models, so this slots into its tooling.
- Anyone with Hopper or Blackwell clusters and an MoE workload bottlenecked on expert-parallel communication.
- Framework maintainers who can compare how fused communication kernels land across stacks, as in the SGLang plugin system for vendor kernels.
You probably do not need it if you train small dense models, use older GPUs without the supported data types, or already run with low communication overhead.
What people are asking
Is this the same as speculative decoding or other inference tricks? No. Those speed up serving a trained model. Mega MoE targets the training step, where gradients flow back through the expert layers, and where each layer's all-to-all communication happens twice.
Why does overlapping compute and communication help so much? On a large cluster, a GPU that finishes its expert math has to wait for tokens from other GPUs, and then wait again to send results back. If the kernel starts computing on tokens that have already arrived while others are still in flight, the idle time shrinks. The effect grows as models get larger and span more nodes, which matches the larger gain Prime Intellect reports on the 235B model than on the 30B model in the TorchTitan comparison.
Should I switch frameworks for this? Not on this evidence alone. The cleanest gain is the 1.34x versus the same framework with the kernel off. Whether the kernel can be ported into your existing stack, such as TorchTitan, depends on the integration work and licensing, which the announcement does not detail.
How do I check a claim like this? Reproduce a short run in your own configuration. Match sequence length, expert parallelism, precision and activation checkpointing; skip warmup steps; compare tokens per second per GPU and also confirm that loss over the first few hundred steps tracks the baseline. A kernel that is faster but changes numerics is a different trade.
Caveats
- Setup-specific. Reported gains depend on intra-node expert parallelism, FP8, long sequences and this exact stack.
- Hardware limited. Hopper and Blackwell only; MXFP8 requires Blackwell and blockwise FP8 requires Hopper.
- Training-quality effects. The post focuses on throughput. Anyone adopting a new kernel should check that loss curves and numerics match their baseline.
- Credit chain. The core fusion idea is DeepSeek's. Prime Intellect's contribution is the backward pass and integration into Prime-RL.
Context: Prime Intellect's recent releases
Prime Intellect has been shipping quickly: Prime Inference for serving open models on October 2, the Moondream acquisition on October 13. DeepSeek's own infrastructure releases continue too; see our coverage of its open-source inference engine.
Why this matters beyond one kernel
Open-weight MoE models have become the default shape for large releases, from Qwen3 to DeepSeek and others, which means the cost of training and post-training them is increasingly a communication problem rather than a pure arithmetic one. A faster step time translates directly into cheaper reinforcement-learning runs: if a run takes 34 percent fewer GPU-hours for the same number of steps, a lab can iterate more or spend less. For small teams fine-tuning open models, this is the kind of low-level gain that quietly widens what is affordable.
There is also a pattern worth noting. Frontier-lab infrastructure work, such as DeepSeek's kernels, is being released openly and then extended by outside groups. That compounds quickly, and it narrows the infrastructure gap between large labs and everyone else. The flip side is maintenance: custom fused kernels tie your stack to specific GPU generations and data types, so teams should weigh the speedup against portability.
How to try it
We could not find a repository link in the sources we checked, so follow the links in Prime Intellect's post and its Prime-RL documentation for installation. In general, adoption means building the kernel for your GPU architecture, replacing your MoE layer with the fused call, and benchmarking a short run with the same measuring approach (skip the compile warmup steps) before committing. Start with the smaller model setup to validate correctness.
Related reading
- Meta's dist-moe dropless MoE kernels
- Cursor's mixture-of-kittens MoE megakernel
- Prime Inference launch
- Prime Intellect acquires Moondream
- SGLang plugin system and vendor kernels
- DeepSeek's open-source inference engine
Benchmark figures are Prime Intellect's own, as of October 29, 2026.