PyTorch 2.15 on Mac: FP8, Faster MoE Decoding and an 8x Softmax on Apple Silicon

If you run models on a Mac through PyTorch, the October 20 post "MPS in PyTorch 2.15" on the Hugging Face blog is the most detailed account yet of what changes. Its authors, Irakli Salia and Khushi Agrawal, list FP8 tensors, native mixture-of-experts (MoE) kernels, new Metal implementations of softmax and padding, and a pile of linear algebra fixes. The headline claim is that MoE models decode 5 to 6.5 times faster on Apple Silicon than before. We read the post closely, because the numbers inside it do not all tell the same story.

This is a report on a release described by its maintainers. We have not benchmarked 2.15 ourselves, so every figure below is attributed to the Hugging Face post.

TL;DR: what changes for Mac users

FP8 arrives on MPS

The release adds torch.float8_e4m3fn on MPS. An FP8 value takes one byte, half of FP16 or BF16, so weights and activations shrink accordingly. F.scaled_mm handles tensorwise and rowwise scaling, and the post's example quantizes a 1 by 1024 input and a 1024 by 1024 weight with per-tensor scales and returns a BF16 output.

For local-model users this is the foundation, not yet the payoff. The post says Metal 4.1 shaders with faster matmuls and native FP8 matmul are being prepared for later. Today, FP8 mostly halves storage and bandwidth; the compute-side win depends on that future work. If you want the wider picture of how people squeeze large models onto Macs, see our posts on local inference with DwarfStar and DFlash 2 on MLX with an M5 Max.

Why decoding got faster

Decoding generates one token at a time, so the work per step is a pile of tiny matrix-vector products. Two changes target that.

Dense layers

F.linear now uses GEMV kernels for single-row inputs, with eligible bias additions fused in. Across 96 BF16 layer shapes the geometric-mean speedup is 1.23x. Gains concentrate in small projections such as MoE gates; large layers and LM heads stay near 1x, which is expected because those are bandwidth-bound already.

Mixture-of-experts

Grouped matmul now has a native MPS implementation. Group offsets are read on the GPU instead of being copied to the CPU, and a dedicated kernel handles the small groups typical of decoding. The post also lists fewer GPU-to-CPU syncs and faster top-k routing. Because MoE models route each token to a few experts, the grouped matmul is the hot path, and avoiding a CPU round trip on every step is a large part of the gain.

Here is the post's 2.14 to 2.15 table on an M3 Ultra with torch.compile:

That is roughly a 1.3x improvement. The headline figures are bigger: Qwen3.6-35B-A3B from 9.4 to 61.3 tokens per second, and LFM2.5-8B-A1B from 23.6 to 123.6. The post does not clearly say why the two sets differ; the likely explanation is a different baseline, such as eager mode on an older release versus the compiled path, but the post does not say so. Read "5 to 6.5x" as the best case against a weak starting point, and "1.3x" as the like-for-like kernel gain until the authors clarify. Either way, a 35B-parameter MoE running above 50 tokens per second on a desktop Mac is a useful result. Our write-up of a consumer-GPU run of a 125B MoE and the guide to Qwen 3.6 27B locally give context for what other setups achieve.

Softmax, padding and the 270 GiB bug

Several fixes are less glamorous but matter more for people running image and video models.

• Softmax and log-softmax run on dedicated Metal kernels. A FP32 softmax over a row of 128,256 values drops from 57 to 7 microseconds, about 8x faster than MPSGraph, and the speedup is 7.1x to 8.0x across vocabularies from 128,256 to 201,088. Softmax over the vocabulary happens on every generated token, so this helps all LLM decoding.
• Constant padding moved to Metal kernels. In a Wan2.2-TI2V-5B-Diffusers VAE decode, tiled over 28 tiles with 8,764 padding calls, peak total driver memory fell from about 270.1 GiB to 8.1 GiB, per the post's table. Allocator-reserved memory was 8.07 GiB in both cases, so the old path was leaking through driver-level allocations that were not visible in the usual counters.
• Image corruption fix. On 2.14, the Qwen-Image-2.1 VAE encoder corrupted images on MPS, a problem reported in ComfyUI. With 2.15, an encode-decode round trip matches CPU at 29.95 dB PSNR, up from 10.06 dB.
• Other kernels: specialized conv1d forward kernels, Metal reflection and replication padding (also fixing backward correctness), a gather kernel for upsampling backward, and the fused dropout kernel for out-of-place dropout.

Linear algebra and new operators

For scientific and vision workloads, the release brings complex64 LU factorization, LU solve and triangular solve to MPS, which unlocks complex determinants. Batches of FP32 matrices up to 4 by 4 are inverted in one Metal kernel, aimed at camera geometry and image transforms. QR uses blocked updates, and there is a fast path for 16, 32 and 64 sized triangular systems used by Gated DeltaNet style layers. SVD handles tiny singular values and extreme magnitudes more robustly, which improves pseudoinverse and least-squares results.

New operators on MPS include torch.frexp, torch.logspace, torch.take, Tensor.put_, torch.heaviside and unsigned-integer comparisons. One stated limitation: logspace does not yet support integer or boolean outputs with more than one element. The math implementation of scaled dot-product attention now handles batch and head broadcasting correctly.

Streams and events

torch.mps.Stream and the torch.mps.stream context manager let Python code choose which stream runs work, and Tensor.record_stream lets the allocator track a tensor used on another stream. This is the same pattern CUDA developers know for overlapping data loading and compute, and it is what pipelines and custom schedulers need. Event timing is also described as more reliable. The post includes a short example that runs work on a separate stream, records the tensor use and synchronizes before reading the result.

Preview: what PyTorch 2.16 may bring

The authors are explicit that some of these are on main or in review and may change.

The dense Qwen3.8-27B result is the telling one: only 4 percent. Dense models at that size are memory-bandwidth limited, so scheduling tricks help MoE models far more than dense ones.

What this means for what you build

• Running MoE models locally on Mac is getting practical. If you prototype with Hugging Face transformers on MPS, 2.15 is worth the upgrade, provided you can move to macOS 15.
• Do not expect miracles on dense models. Gains scale with how much of the step is launch overhead and syncing, not raw bandwidth.
• Check your OS. The macOS 15 floor drops older machines from MPS support.
• Benchmark your own model. The headline and the table differ by about 4x, so measure with and without torch.compile on your workload before promising anything to a team.
• Try the release candidate in a clean environment. The post shows installing from PyTorch's test index with uv pip in a separate environment, and invites bug reports on GitHub for slow, missing or broken MPS behavior. The PyTorch MPS documentation covers the basics of the backend.

For a different corner of the PyTorch ecosystem moving to native stacks, see our take on Amazon TorchNeuron on Trainium and NVIDIA's cuTile Mamba-2 kernels.

Related reading

• DwarfStar DS4: local inference
• DFlash 2 on MLX with an M5 Max
• Strata: 125B MoE on a consumer GPU
• Qwen 3.6 27B local development guide
• Amazon TorchNeuron on Trainium: claimed vs verified
• NVIDIA cuTile Mamba-2 kernels
• Run MoE models larger than RAM by streaming experts from SSD

Figures are from the Hugging Face post published October 20, 2026 and may change before the final PyTorch 2.15 release.