A 26B MoE that fits in a ~2 GB process budget is not magic — it is SSD I/O overlapping GPU work.
On July 29–30, 2026, Andrey Mikhaylov’s TurboFieldfare hit Show HN: run Gemma 4 26B-A4B IT on any M-series Mac, including 8 GB machines, by keeping the shared ~1.35 GB core + FP16 KV in RAM and streaming only the routed experts needed per token from disk. Installed weights ~14.3 GB; process RSS ~2 GB (4K context, 16 expert-cache slots). Measured decode: ~5–6 tok/s on M2 Air, ~31–35 tok/s on M5 Pro.
This sits in the same local-inference lane as AirLLM-style offload, ESP32 per-layer flash tricks, and Apple-focused stacks we cover in MacBook vs dedicated GPU — but it is a purpose-built Metal runtime, not “turn down llama.cpp layers.”
TL;DR
| Question | Answer |
|---|---|
| Model | Gemma 4 26B-A4B IT (~3.88B active/token) |
| RAM budget | ~2 GB (+ OS page cache helps a lot) |
| Disk | ~14.3 GB .gturbo install (~15 GB download) |
| Stack | Swift 6.2 · Metal 4 · macOS 26+ |
| M2 8 GB | 5.1–6.3 tok/s decode |
| M5 Pro 24 GB | 31–35 tok/s decode |
| Trick | Resident shared MoE path + pread expert cache |
| API | Loopback OpenAI-compatible server |
| License | Apache-2.0 code; weights under Gemma terms |
| Stars (at writeup) | ~1.2k |
What people are asking
“Why not just mmap the whole GGUF?”
HN asked. Author’s first version used mmap. On 8 GB M2, a cold ~3.36 MB expert took ~10 ms with mmap vs ~2.8 ms with pread; full simulation ~0.50 vs ~4 tok/s. mmap pages reactively and does not know which experts the router picked or when reads can overlap GPU work. TurboFieldfare: router → LFU 16-slot cache → miss fills via bounded parallel pread into Metal-visible buffers while Metal runs the shared expert branch.
llama.cpp can run huge MoEs under tight RSS with offload — often much slower when the OS is IOPS-bound on small page faults. Explicit large-chunk reads + compute overlap is the product.
“Is 5 tok/s usable?”
For interactive coding against Claude/Codex APIs — usually no. For offline chat, batch jobs, privacy-sensitive notes, or “keep 12 GB free for Resolve” — often yes. Author: with enough RAM you can raise expert-cache slots (e.g. 32 → ~3.5 GB) for better hit rate. Community: M4 Max 64 GB reported ~48 tok/s largely because the packed experts stay in page cache (iostat showed far less true disk than cold-miss math).
Unused RAM is disk cache. Starve the machine and tok/s falls toward raw SSD.
“macOS 26 only?”
Official requirements: macOS 26, Metal 4, Swift 6.2, Xcode 26, Apple Silicon. Apple10 GPU family gets a documented ~2.4× prefill path. M1 users on macOS 15 reported building after commenting languageVersion = .version4_0 — workable but without that prefill win.
“Will this destroy my SSD?”
Reads do not meaningfully wear NAND. Sustained overnight decode will heat fanless Airs; alternate I/O and GPU phases leave brief idles, but thermal throttle is still expected.
How the engine works
per layer:
Metal: attention + router (resident weights)
CPU: top-8 expert IDs → LFU 16-slot plan
miss → parallel pread into Metal buffers
Metal: shared-expert branch (overlaps reads)
Metal: combine shared + routed outputs
prefill: chunks ≤128 tokens (one fetch serves many rows)
decode: token-by-token routed loop
Weights: MLX affine 4-bit (group 64) for embedding/attention/experts; 8-bit router. KV: FP16 with circular storage for sliding-window layers and linear for full-attention layers. Installer streams Hugging Face ranges straight into .gturbo — no second full checkpoint on disk.
Expert reuse (author): ~41% selected again next token, 57% within two — enough for cache to cut I/O roughly in half on M2 logs (250–320 MB pread per token during I/O phase at ~3 GB/s bursts; hit rate ~59–69%).
Curated 103 experiments in-repo — most failed (mmap, madvise, F_RDADVISE, Markov expert predictors, disk reordering). That lab notebook is the real Show HN value.
Quick start
git clone https://github.com/drumih/turbo-fieldfare.git
cd turbo-fieldfare
swift build -c release
.build/release/TurboFieldfareMac
First launch: Download (~15 GB), then Load Model, prompt, Generate. CLI / repack / verify:
swift run -c release TurboFieldfareRepack \
--output scratch/gemma4.gturbo --overwrite
swift run -c release TurboFieldfareCLI \
--model scratch/gemma4.gturbo \
--messages-file messages.json
Loopback server:
swift build -c release --product TurboFieldfareServer
.build/release/TurboFieldfareServer --model scratch/gemma4.gturbo
# http://127.0.0.1:8080/v1 — Chat Completions, streaming, tools
Run one model owner at a time (app, decode service, CLI, server, tests). Check memory_pressure -Q before long runs.
When to use TurboFieldfare vs MLX
| Need | Prefer |
|---|---|
| 8–16 GB Mac, want 26B-class MoE offline | TurboFieldfare |
| Max tok/s, RAM to spare (~14 GB+) | MLX (author ~75 tok/s M5 Pro full-resident) |
| Multi-model zoo / GGUF ecosystem | llama.cpp / MLX community |
| Coding SOTA locally | Often Qwen MoE elsewhere — Gemma is “good enough” generalist |
| Non-Apple GPU | Not this repo (Metal-specific) |
Related Gemma coverage: Gemma 4 updates, Gemma 4 12B multimodal, DiffusionGemma, how to run open models locally.
Sibling SSD-stream MoE projects HN pointed at: Flash-MoE, Colibri, antirez/DwarfStar-class DeepSeek engines for 64 GB+ machines.
What the experiments teach (beyond this repo)
Mikhaylov’s inventory is a case study in local MoE systems engineering:
| Idea | Outcome (author) |
|---|---|
| mmap experts | Too slow on cold 8 GB M2 |
pread + overlap shared GPU | Primary win (~0.5 → ~4 tok/s class) |
| madvise / F_RDADVISE | Mixed; long decodes sometimes worse |
| Predict next experts (Markov / cross-layer) | Did not stick |
| Reorder experts on disk | Overfit to one prompt |
The HN wish for MTP heads that prefetch experts is harder than it sounds: expert choice is per layer, and layer n depends on layers below — you cannot freely speculate the whole stack from a draft token. Prefetch wrong experts and you burn the SSD slack you do not have.
Memory-bandwidth tables for M2→M5→rumored M6/M7 matter here. As base chips gain bandwidth and SSDs keep leaping (M5 laptop SSD uplift was already huge vs M4), streamed MoE becomes more mainstream for “workstation with big disk, not enough RAM for the full quant.”
Products in the package
| Product | Role |
|---|---|
| TurboFieldfare | Swift library + Metal kernels |
| TurboFieldfareMac | Native install / generate app |
| TurboFieldfareDecodeService | One-shot local Metal owner for the app |
| TurboFieldfareCLI | Instruction chat + raw completion |
| TurboFieldfareServer | Loopback OpenAI-compatible API |
| TurboFieldfareRepack | Streaming installer + verify |
Prompting defaults: temperature 0.2, Top-K 64, Top-P 0.95; set temperature 0 for greedy. The Mac app owns chat formatting; --prompt is raw completion for reproducible benches. Multi-turn chat was still landing at Show HN time — check releases.
Builder checklist
- Free ≥20 GB disk before install.
- Close browsers / IDEs; note page-cache effects when benchmarking.
- Report tok/s + RSS + free RAM + SSD model via the community benchmark guide.
- Treat local OpenAI server as loopback-only.
- Do not expect Claude-class agent coding from Gemma 4 26B — harness + model both matter (agent harness guide).
- If you port to Qwen 35B-A3B, reuse the router stop → pread experts → resume pattern; kernels won’t copy-paste.
- Raise expert-cache slots when you have RAM headroom; document the new RSS.
- Keep MLX installed in parallel so you can A/B quality and speed on the same prompt set.
For “everything models vs something models” HN debates: MoE streaming is a systems answer to memory, not a substitute for training smaller specialists. You still pay for experts on disk and in transfer time; you just refuse to keep all of them hot.
Honest limitations
- macOS 26 gate keeps many users out today.
- Gemma-only — not a general MoE runtime.
- Text-only; tools are server-side declarations for the client to run.
- Speed is SSD- and cache-bound; marketing tok/s needs cold vs warm disclosure.
- Quality ≠ frontier API for hard coding/agent work.
- Independent security review still on you (Swift/Metal local stack; no HF token required in default path).
Closing
TurboFieldfare is a clean demonstration that MoE sparsity + explicit I/O planning beats “hope mmap is fast enough” on memory-poor Macs. If you have an 8–16 GB Apple Silicon box and want Gemma 4 offline without swapping yourself into oblivion, clone it, run the app, and send back numbers. If you have 64 GB and need max speed, stay on MLX — and steal the experiment log either way.
Follow @explainx_ai for more local-inference engineering writeups.
Related on explainx.ai
- AirLLM — 70B-class on 4GB GPU patterns
- MacBook vs dedicated GPU for local LLMs
- How to run open-source models locally
- Gemma 4 updates — flash attention & tools
- Gemma 4 12B multimodal local
- DiffusionGemma
- ESP32 tiny LLM + flash embeddings
- Open-weights American AI letter
Sources
- drumih/turbo-fieldfare
- Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM
- Gemma 4 model card
Benchmarks and requirements as published in the TurboFieldfare README around July 29–30, 2026. Re-measure on your hardware; page cache and macOS version dominate outcomes.