Deltafin: Run Kimi K3 (2.8T MoE) on One Apple Silicon Mac

Deltafin answers a ridiculous question honestly: can you run Kimi K3 (~2.8T MoE) on one Apple Silicon laptop?

Yes — at about 16 seconds per token on an M1 Max 64 GB with a full install. Not a daily driver. An existence proof with open notes, fused kernels, and an OpenAI-compatible server. HN framed it as “seconds per token, not tokens per second” — and still upvoted it (thread).

For real serving, use explainx.ai’s multi-GPU / hosted K3 guide. For why K3 is shaped for sparse I/O, see Raschka’s architecture notes.

TL;DR

Why MoE makes the joke possible

K3’s weights are ~1.56 TB. A Mac cannot hold that in RAM. Per token, the router only needs 16 experts × 92 layers** of expert data (25.8 GB**) plus a resident spine (attention, shared experts, latents, embeddings — ~114 GB on disk I/O path).

Deltafin:

1. Downloads the spine once; optionally converts to int8 (convert_spine_int8.py) to cut I/O.
2. Either installs all 82,432 routed experts locally (--full) or HTTP range-fetches them into a growing cache (--stream).
3. Runs Moonshot’s modeling code, with a pure-PyTorch shim for CUDA-only fla kernels (KDA recurrence on CPU at decode).

That is the same sparse-I/O philosophy as projects they credit: colibri, ds4 / DwarfStar, llama.cpp MXFP4 priors.

Install (three commands)

python3 -m venv venv
./venv/bin/pip install torch numpy safetensors tiktoken ml_dtypes blobfile \
  "transformers==4.56.2" einops tokenizers

clang -O3 -mcpu=native -shared -DNO_MAIN -o tools/libmxfp4gemv.dylib tools/fused_gemv.c

./venv/bin/python tools/setup_k3.py --full   # or --stream

Needs Xcode CLT, Python 3.12+, Hugging Face access. No flag → auto-picks full if disk allows.

Upgrade stream → full later:

./venv/bin/python tools/fetch_experts_all.py

Run and serve

./venv/bin/python tools/kimi_run.py --chat --prompt "Three largest moons of Saturn?"
./venv/bin/python tools/serve_openai.py --port 8000

Point OPENAI_BASE_URL at http://127.0.0.1:8000/v1. Caveats from the README (believe them):

• Client timeouts in hours
• Greedy only (temperature / top_p ignored)
• One request at a time (else 429)
• Streaming installs make chat prefill brutal
• Agents with huge system prompts are a demo, not a workflow

Router picks land in router_trace.jsonl if you study expert reuse (~31% consecutive-token overlap in their holdout).

What “OpenAI-compatible” means here

serve_openai.py is enough to point Cursor, OpenCode, or a custom client at http://127.0.0.1:8000/v1 with a dummy API key. That does not mean you get cloud Sol latency, streaming UX, or multi-tenant safety. It means the request shape matches enough of the Chat Completions API for local experiments. Keep timeouts in hours, expect greedy-only decoding, and never point a production webhook at this process.

If you want a less heroic laptop path, compare run K3 on real GPU hosts and the architecture map so you know which blocks (LatentMoE, MLA, KDA) are burning your SSD.

Where the 16 seconds go

On quiet M1 Max, full install (approx):

Decode is disk-bandwidth bound on the spine. 128 GB RAM can keep the spine in page cache — README says that cost largely vanishes. Antirez publicly teased faster SSD-streaming numbers on M5 Max 128 GB; treat third-party clips as directional, not Deltafin’s own benchmark table.

Hardware shopping list (honest)

Disk: budget ~1.7 TB for full experts vs ~200 GB streamed. If your laptop SSD is 1 TB with OS + Xcode, you do not have a full install — stop arguing with the README and use --stream.

Why fused MXFP4 GEMV matters

The clang -O3 … fused_gemv.c step is not cargo-cult. On Apple Silicon, naive Python dequant + matmul leaves performance on the table; a small native library for the MXFP4 path is how Deltafin makes “one token per ~16s” a measurable demo instead of a multi-minute joke. If that .dylib fails to build, fix Xcode CLT before you blame the model.

Failure modes we see in the wild

1. HF auth missing → download dies mid-expert.
2. Wrong transformers pin → loader incompat; stick to the README’s 4.56.2.
3. Client 60s timeout → false “server hung” while prefill still runs.
4. Second concurrent request → 429; this is single-flight by design.
5. Huge system prompts → agents look broken; shorten context for laptop demos.
6. Thermal throttle on a closed MacBook → tokens stretch beyond the quiet-desk table.

How to use Deltafin without lying to yourself

• Goal A: Prove an open 2.8T-class MoE can emit tokens on a Mac → Deltafin is perfect.
• Goal B: Daily driver coding agent → use hosted Kimi / Modal / GPU box (OpenCode + Modal write-up).
• Goal C: Study expert routing → enable router_trace.jsonl and compare consecutive-token expert overlap to the ~31% holdout note.
• Goal D: Architecture literacy → pair with Raschka’s LatentMoE / NoPE notes.

Deltafin is a teaching instrument that happens to be a real inference path — not a replacement for a workstation.

Techniques that actually moved the needle

Worth stealing for other MoE streamers:

• Coalesced expert fetch (one 17.55 MB range / expert) ~6.4× vs per-tensor
• pread + F_NOCACHE thread pool — cold expert path 40 s → 4.3 s
• Double-buffered spine load; previous-token expert prefetch
• Fused MXFP4 NEON gemv; Metal dequant prototype
• Template-layer GPU buffer reuse for KDA/MLA shape families
• Lossless n-gram speculation when resident I/O dominates

Failures they published (good science): low-rank expert approx useless, MXFP4 barely compressible, HTTP/2 slower than H1 keep-alive here, no MTP head, two-Mac TP arithmetic doesn’t close.

Honest framing (HN consensus)

• Fun / resilience / measurement — not “local AI wins the datacenter war tomorrow.”
• Email / overnight interfaces fit the latency better than chat.
• Reads do not murder SSD endurance the way writes do.
• Title should say M1 Max, not base M1 — the HN title was corrected for a reason.

Teaching lab outline (90 minutes)

1. Clone Deltafin; build the fused GEMV dylib.
2. --stream install on a 64GB Mac; generate one token; time it.
3. Inspect router_trace.jsonl for expert reuse.
4. Hit serve_openai.py from a tiny curl Chat Completions call.
5. Discuss when to abandon laptop inference for Modal / GPU hosts.
6. Map observed latency back to LatentMoE / MLA / KDA.

Students leave understanding memory hierarchy + MoE routing, not just “AI on Mac.” That is Deltafin’s real syllabus value.

FAQ for Mac owners

Can I code with this daily? Not realistically on 64GB stream installs — use hosted Kimi or a GPU box.

Is 16s/token a bug? No — spine disk bandwidth dominates until RAM caches it.

Why transformers==4.56.2? Pin breakage is common; follow README.

Can I raise temperature? Greedy-only today; ignored params are documented.

Is this the same as MLX ports? Different stack — Deltafin’s fused MXFP4 path is its own engineering object.

Treat Deltafin as a lab microscope for MoE-on-Apple, then graduate to serious local serve when you need productivity.

Closing note for explainx.ai readers

This launch moves fast; verify primary docs before you standardize tooling or brief a client. Re-check availability, pricing, and model IDs on the official pages linked above, then tell us what broke in production so we can update the guide. Follow @explainx_ai for follow-ups when the vendors ship the next patch — and prefer measured evals on your own traffic over screenshot economics. If you only needed a headline, you already have it; if you are implementing, the checklists above are the part that saves a weekend.

Related on explainx.ai

• Kimi K3 architecture — Raschka / LatentMoE / NoPE
• How to run Kimi K3 locally (GPU / hosted)
• Kimi K3 open weights release
• Open model feels surprisingly good — Modal + OpenCode
• Fermion Neutrino — tiny ternary on-device
• What is llama.cpp?
• PrismML Bonsai phone ternary

Sources

• gavamedia/deltafin
• Hacker News — Running Kimi K3 on a M1 Max
• Moonshot Kimi K3

Speeds and disk sizes are from Deltafin’s README measurements on one M1 Max as of the Jul 2026 publish. Your NVMe, RAM, and chip will change the floor.