Colibrì: Run GLM-5.2 on 25 GB RAM by Streaming MoE Experts From Disk

On July 10, 2026, Colibrì hit 453 points on Hacker News — Show HN: Getting GLM 5.2 running on my slow computer. Author vforno (JustVugg) set a deliberately modest goal: make GLM-5.2 — Z.ai's 744B MoE frontier open model — answer correctly on a 12-core laptop with 25 GB RAM, even if speed is measured in minutes per paragraph, not tokens per second.

The result is not a faster llama.cpp fork. It is a disk-streaming MoE runtime: ~9.9 GB dense weights stay in RAM; ~370 GB of routed experts live on NVMe and load on demand. Pure C, zero runtime deps, token-exact against a transformers oracle.

TL;DR — what people are asking

The idea — why only ~11 GB changes per token

GLM-5.2 activates ~40B parameters per token, but MoE routing means only ~11 GB of expert weights are new each decode step — the rest is shared dense stack (attention, embeddings, shared experts).

Colibrì splits the model physically:

The engine (c/glm.c, ~1,300 lines) implements faithful glm_moe_dsa forward — validated 32/32 teacher-forcing and 20/20 greedy against a tiny-random oracle with the real architecture.

MLA attention with compressed KV: 576 floats/token vs 32,768 naive — critical for long sessions on RAM-starved boxes (GLM-5.2 specs).

What's actually implemented (not vapor)

From the README as of the HN launch:

• DeepSeek-V3-style router — sigmoid, noaux_tc, routed_scaling_factor, shared expert, first-3-dense layers
• MTP speculative decoding — GLM-5.2's layer-78 head drafts; main model verifies in one batched forward
• MTP head must be int8 — at int4, draft acceptance collapses to 0–4%; int8 hits 39–59% acceptance, 2.2–2.8 tokens/forward (community #8)
• Honest MTP caveat — cold cache adds expert loads per verified draft; speculation can lose until cache warms; DRAFT=0 disables it
• Integer-dot kernels — Q8_0-style activations, AVX2; int8 matmuls 1.4–2.5× faster than f32 paths measured
• MLA weight absorption on decode — DeepSeek trick; exact match with absorption forced
• Async expert readahead — WILLNEED while multiplying current block
• Learning cache — .coli_usage records routed experts; startup auto-pins hottest in spare RAM — "gets faster the more you use it"
• Batch-union MoE on prefill — each unique expert read once per batch
• BPE tokenizer in C — 320k merges, no Python at runtime
• DSA sparse attention — in progress (indexer weights ~108 GB extraction downloading)

Honest speed numbers — read before you clone

Colibrì is explicit: this is not fast. It is a 744B-class model talking on hummingbird rations.

Author dev box (WSL2, 12 cores, 25 GB RAM, VHDX NVMe ~1 GB/s random)

Community benchmarks (stock setup.sh, greedy, --ngen 32)

HN consensus: 0.05–0.1 tok/s is a proof-of-life demo, not a daily driver. ~1 tok/s on Apple Silicon with fast SSD starts to resemble "walk away overnight" agent workflows walrus01 noted on HN. 5–15 tok/s remains a back-of-envelope target for 128–256 GB RAM + many cores + warm pin — not yet measured at scale.

SSD, swap, and the corrected wear story

Early README drafts worried about SSD wear from page cache writes. The July 10 README clarifies:

• Expert streaming = read-only — does not wear NAND meaningfully
• Swap under memory pressure does write — colibrì sizes expert cache from MemAvailable to stay out of swap
• Thermals — hours at full read duty cycle heat cheaper drives; monitor health

For soldered-SSD laptops, HN commenters treat this as an experiment, not default daily tooling — external NVMe or a desktop with replaceable storage is saner for multi-hour runs.

Quick start — commands that work

cd c
./setup.sh                      # gcc + OpenMP, self-test (expects 32/32)

# One-time convert (needs python: torch safetensors huggingface_hub numpy)
./coli convert --model /nvme/glm52_i4     # ~400 GB free, resumable

# Or skip convert — download prebuilt:
# https://huggingface.co/jlnsrk/GLM-5.2-colibri-int4

COLI_MODEL=/nvme/glm52_i4 ./coli chat

Useful knobs: --temp 0.7 (int4-tuned), --topp 0.7 (adaptive expert top-p, ~1.6× speedup in #2 benchmark), THINK=1 reasoning block, DRAFT=n MTP depth, PIN=stats.txt PIN_GB=20 hot-expert pinning.

Quality sweep (needs fast disk):

./coli bench                    # hellaswag, arc_challenge, mmlu — 40 each
./coli bench mmlu --limit 200

The project has not published int4 accuracy loss yet — the harness exists; a full run on the dev box disk is ~a day. That benchmark is the highest-value community contribution right now.

Colibrì vs the rest of the GLM-5.2 local stack

Colibrì does not replace Perplexity's GLM orchestrator or Ollama's $88M hybrid cloud bet. It extends the sovereign extreme: frontier open weights when you refuse to rent GPUs and refuse to buy 256 GB RAM.

Hacker News themes worth keeping

Ticket / async interface — several commenters argued chat UI is wrong for 0.1 tok/s models; queue a task, check back in hours (loop engineering mindset without cloud).

RAM is the hidden cap on 24 GB machines — even with 2.7× faster disk, expert cache auto-caps to 2 slots/layer, so decode stays cold. More RAM → more pin → fewer reads.

Parallel NVMe / RAID0 — untested multiplier; author invites benchmarks.

OpenCode integration — in-flight PR per OP; until merge, wire agents manually or wait.

Related experiments — flash-moe reported 5+ tok/s on M3 Max (different approach); hypura for general SSD streaming — same design space, different targets.

Who should try Colibrì — and who should not

Try it if:

• You have ≥25 GB RAM, ≥400 GB NVMe, Linux or WSL2, AVX2
• You want to validate int4 GLM-5.2 quality on hardware faster than the author's laptop
• You are researching MoE disk streaming for local LLM economics

Skip it if:

• You need interactive coding agents today — use API, Cline, or 256 GB Unsloth
• You have only soldered 256 GB Mac SSD and fear thermals — treat as read-only experiment
• 0.07 tok/s would frustrate you — cloud GLM-5.2 harness is cheaper time-wise

Star the repo, run iobench, open an issue with numbers — the README's scaling table is explicitly waiting for datapoints from better machines.

How to read these numbers before you test it

The figures in this post come from three different kinds of evidence, and they should not be mixed. The 0.05-0.1 tok/s result is the author's own cold-decode measurement on a WSL2 box with a VHDX-backed NVMe. The 0.07 and 0.11 tok/s results come from a community run on a 24 GB machine, and the 1.06 tok/s figure comes from an M5 Max with 14.2 GB/s O_DIRECT disk. The 5-15 tok/s range, by contrast, is a back-of-envelope target that has not been measured at scale.

When you run your own test, record the same fields the community reports use: the iobench disk numbers, the config (default, --topp 0.7, MTP off), the cache hit rate, and peak RSS. A tok/s figure without disk throughput and hit rate tells the next reader very little, because the post shows that disk speed and RAM-limited cache slots drive the result. Also note that int4 accuracy loss has not been published yet, so treat any output as a proof of life rather than a quality verdict until coli bench results exist.

Related on explainx.ai

• Newer: Janas streams MoE experts from SSD to run an 80B model on 32 GB

• 28.9M LLM on $8 ESP32 — flash PLE table

• GLM-5.2 MIT open source and Code Arena adoption

• Run GLM-5.2 locally with Unsloth — 256 GB path

• What is llama.cpp? GGUF and local APIs

• How to run GLM 5.2 in agent harnesses

• Fix local LLM looping — samplers and sandboxing

• Ollama $88M — open models runtime

• Perplexity GLM 5.2 orchestrator — hosted contrast

• Qwen 3.6 27B local — when dense beats MoE

Official: Colibrì on GitHub · GLM-5.2-colibri-int4 weights · Show HN thread

Benchmarks and README details reflect the July 10, 2026 colibrì launch and community issues #2, #4, #5, #8. Tok/s varies enormously by NVMe, RAM pin budget, and MTP settings — run coli bench and iobench on your hardware before assuming interactive speed.