TurboFieldfare: Gemma 4 26B in ~2 GB RAM on Apple Silicon

Update — September 2, 2026: If you have the RAM to spare instead of streaming experts from disk, a community MLX leaderboard has since pushed straight-in-memory Gemma 4 26B A4B inference 130%+ faster on Mac — Google's own account rounded it to "2x."

A 26B MoE that fits in a ~2 GB process budget is not magic — it is SSD I/O overlapping GPU work.

On July 29–30, 2026, Andrey Mikhaylov’s TurboFieldfare hit Show HN: run Gemma 4 26B-A4B IT on any M-series Mac, including 8 GB machines, by keeping the shared ~1.35 GB core + FP16 KV in RAM and streaming only the routed experts needed per token from disk. Installed weights ~14.3 GB; process RSS ~2 GB (4K context, 16 expert-cache slots). Measured decode: ~5–6 tok/s on M2 Air, ~31–35 tok/s on M5 Pro.

This sits in the same local-inference lane as AirLLM-style offload, ESP32 per-layer flash tricks, and Apple-focused stacks we cover in MacBook vs dedicated GPU — but it is a purpose-built Metal runtime, not “turn down llama.cpp layers.”

TL;DR

What people are asking

“Why not just mmap the whole GGUF?”

HN asked. Author’s first version used mmap. On 8 GB M2, a cold ~3.36 MB expert took ~10 ms with mmap vs ~2.8 ms with pread; full simulation ~0.50 vs ~4 tok/s. mmap pages reactively and does not know which experts the router picked or when reads can overlap GPU work. TurboFieldfare: router → LFU 16-slot cache → miss fills via bounded parallel pread into Metal-visible buffers while Metal runs the shared expert branch.

llama.cpp can run huge MoEs under tight RSS with offload — often much slower when the OS is IOPS-bound on small page faults. Explicit large-chunk reads + compute overlap is the product.

“Is 5 tok/s usable?”

For interactive coding against Claude/Codex APIs — usually no. For offline chat, batch jobs, privacy-sensitive notes, or “keep 12 GB free for Resolve” — often yes. Author: with enough RAM you can raise expert-cache slots (e.g. 32 → ~3.5 GB) for better hit rate. Community: M4 Max 64 GB reported ~48 tok/s largely because the packed experts stay in page cache (iostat showed far less true disk than cold-miss math).

Unused RAM is disk cache. Starve the machine and tok/s falls toward raw SSD.

“macOS 26 only?”

Official requirements: macOS 26, Metal 4, Swift 6.2, Xcode 26, Apple Silicon. Apple10 GPU family gets a documented ~2.4× prefill path. M1 users on macOS 15 reported building after commenting languageVersion = .version4_0 — workable but without that prefill win.

“Will this destroy my SSD?”

Reads do not meaningfully wear NAND. Sustained overnight decode will heat fanless Airs; alternate I/O and GPU phases leave brief idles, but thermal throttle is still expected.

How the engine works

per layer:
  Metal: attention + router (resident weights)
  CPU: top-8 expert IDs → LFU 16-slot plan
  miss → parallel pread into Metal buffers
  Metal: shared-expert branch (overlaps reads)
  Metal: combine shared + routed outputs
prefill: chunks ≤128 tokens (one fetch serves many rows)
decode: token-by-token routed loop

Weights: MLX affine 4-bit (group 64) for embedding/attention/experts; 8-bit router. KV: FP16 with circular storage for sliding-window layers and linear for full-attention layers. Installer streams Hugging Face ranges straight into .gturbo — no second full checkpoint on disk.

Expert reuse (author): ~41% selected again next token, 57% within two — enough for cache to cut I/O roughly in half on M2 logs (250–320 MB pread per token during I/O phase at ~3 GB/s bursts; hit rate ~59–69%).

Curated 103 experiments in-repo — most failed (mmap, madvise, F_RDADVISE, Markov expert predictors, disk reordering). That lab notebook is the real Show HN value.

Quick start

git clone https://github.com/drumih/turbo-fieldfare.git
cd turbo-fieldfare
swift build -c release
.build/release/TurboFieldfareMac

First launch: Download (~15 GB), then Load Model, prompt, Generate. CLI / repack / verify:

swift run -c release TurboFieldfareRepack \
  --output scratch/gemma4.gturbo --overwrite

swift run -c release TurboFieldfareCLI \
  --model scratch/gemma4.gturbo \
  --messages-file messages.json

Loopback server:

swift build -c release --product TurboFieldfareServer
.build/release/TurboFieldfareServer --model scratch/gemma4.gturbo
# http://127.0.0.1:8080/v1 — Chat Completions, streaming, tools

Run one model owner at a time (app, decode service, CLI, server, tests). Check memory_pressure -Q before long runs.

When to use TurboFieldfare vs MLX

Related Gemma coverage: Gemma 4 updates, Gemma 4 12B multimodal, DiffusionGemma, how to run open models locally.

Sibling SSD-stream MoE projects HN pointed at: Flash-MoE, Colibri, antirez/DwarfStar-class DeepSeek engines for 64 GB+ machines.

What the experiments teach (beyond this repo)

Mikhaylov’s inventory is a case study in local MoE systems engineering:

The HN wish for MTP heads that prefetch experts is harder than it sounds: expert choice is per layer, and layer n depends on layers below — you cannot freely speculate the whole stack from a draft token. Prefetch wrong experts and you burn the SSD slack you do not have.

Memory-bandwidth tables for M2→M5→rumored M6/M7 matter here. As base chips gain bandwidth and SSDs keep leaping (M5 laptop SSD uplift was already huge vs M4), streamed MoE becomes more mainstream for “workstation with big disk, not enough RAM for the full quant.”

Products in the package

Prompting defaults: temperature 0.2, Top-K 64, Top-P 0.95; set temperature 0 for greedy. The Mac app owns chat formatting; --prompt is raw completion for reproducible benches. Multi-turn chat was still landing at Show HN time — check releases.

Builder checklist

1. Free ≥20 GB disk before install.
2. Close browsers / IDEs; note page-cache effects when benchmarking.
3. Report tok/s + RSS + free RAM + SSD model via the community benchmark guide.
4. Treat local OpenAI server as loopback-only.
5. Do not expect Claude-class agent coding from Gemma 4 26B — harness + model both matter (agent harness guide).
6. If you port to Qwen 35B-A3B, reuse the router stop → pread experts → resume pattern; kernels won’t copy-paste.
7. Raise expert-cache slots when you have RAM headroom; document the new RSS.
8. Keep MLX installed in parallel so you can A/B quality and speed on the same prompt set.

For “everything models vs something models” HN debates: MoE streaming is a systems answer to memory, not a substitute for training smaller specialists. You still pay for experts on disk and in transfer time; you just refuse to keep all of them hot.

How to Report Your Own Numbers

The post's headline figures come from the README and the author's logs, and it states that page cache and macOS version dominate outcomes. That makes the way you measure as important as the result. If you try TurboFieldfare, record the following so your numbers are comparable to the ones quoted above:

1. Hardware and OS. Chip, installed RAM, SSD model, and macOS version. The post's own spread, from 5 to 6 tok/s on an M2 Air to 31 to 35 tok/s on an M5 Pro, shows how much this matters.
2. Cold versus warm disk. The post notes that a reported M4 Max figure of about 48 tok/s was largely explained by experts staying in page cache. Say which state your run started in.
3. Memory. Process RSS, free RAM, and the expert-cache slot count you used (the defaults quoted are 16 slots at 4K context).
4. Prompt settings. Temperature and mode. Use temperature 0 and raw completion if you want a reproducible bench.
5. A baseline. Run the same prompts on MLX if you have the RAM, since the post suggests keeping it installed for A/B comparison of speed and quality.

Also check memory_pressure -Q before a long run and keep only one model owner active at a time. Reports that include these fields are far more useful to the project than a single tok/s number, and they make it easier to tell a real regression from a cache effect.

Honest limitations

• macOS 26 gate keeps many users out today.
• Gemma-only — not a general MoE runtime.
• Text-only; tools are server-side declarations for the client to run.
• Speed is SSD- and cache-bound; marketing tok/s needs cold vs warm disclosure.
• Quality ≠ frontier API for hard coding/agent work.
• Independent security review still on you (Swift/Metal local stack; no HF token required in default path).

Closing

TurboFieldfare is a clean demonstration that MoE sparsity + explicit I/O planning beats “hope mmap is fast enough” on memory-poor Macs. If you have an 8–16 GB Apple Silicon box and want Gemma 4 offline without swapping yourself into oblivion, clone it, run the app, and send back numbers. If you have 64 GB and need max speed, stay on MLX — and steal the experiment log either way.

Follow @explainx_ai for more local-inference engineering writeups.

Related on explainx.ai

• Spark-X2.6-4B: a 4B Apache 2.0 on-device model, claims vs verified

• Newer: Janas streams MoE experts from SSD to run an 80B model on 32 GB

• Gemma 4 26B A4B runs 2x faster on Mac — mlx.fast leaderboard

• AirLLM — 70B-class on 4GB GPU patterns

• MacBook vs dedicated GPU for local LLMs

• How to run open-source models locally

• Gemma 4 updates — flash attention & tools

• Gemma 4 12B multimodal local

• DiffusionGemma

• ESP32 tiny LLM + flash embeddings

• Open-weights American AI letter

Sources

• drumih/turbo-fieldfare
• Show HN: Open-source engine running Gemma 4 26B in 2 GB RAM
• Gemma 4 model card

Benchmarks and requirements as published in the TurboFieldfare README around July 29–30, 2026. Re-measure on your hardware; page cache and macOS version dominate outcomes.