28.9M Params on an $8 ESP32 — How It Fits

Most “edge LLM” demos still assume a phone SoC or a Jetson. slvDev/esp32-ai does something meaner: a 28.9 million parameter TinyStories model that generates text on the ESP32-S3 itself — roughly an $8 microcontroller class — writing tokens to a small OLED at about 9.5 tok/s, with no server.

The previous public bar on a chip like this was ~260K params. This one stores about a hundred times more by refusing to keep the whole model in fast RAM. Google’s Per-Layer Embeddings (Gemma 3n / Gemma 4) move the giant table into flash; the ESP32’s memory hierarchy becomes the product.

This explainx.ai decode is for embedded and edge builders: the memory map, the RESULTS.md ablations, the honest TinyStories ceiling, and how it rhymes with disk-backed Colibri and Moonshine Micro on RP2350 — different modalities, same “bandwidth over capacity” mindset.

TL;DR — What People Are Asking

The Numbers (Headline Deploy Config)

Core is matched across ablations at ~559K dense params — small enough that the design constraint was “fits the thinking budget,” not “fill the flash.” Vocab 32768, d_model=96, 6 layers, ple_dim=128.

Why MCU LLMs Usually Stay Tiny

Internal SRAM on the S3 is 512KB. If every weight must be hot, you stay in the sub-million-param ghetto. DaveBben’s earlier esp32-llm (~260K) is the comparison context the README cites — independent prior work, not a dependency.

The fix is philosophical: stop treating “model size” as “RAM footprint.” Embedding-style tables are read, not multiplied through every step the way a dense FFN is. Leave 25M rows in flash; fetch 6 rows/token (450 B).

SRAM  (fast, tiny)   thinking core — used every token
PSRAM (medium)       output head + working memory / KV
FLASH (huge, slow)   25M-param PLE table — sparse reads

On-chip bandwidth benches (RESULTS.md): flash random 512B row 20.3 µs; six rows ≈ 0.12 ms per token for the table — about 0.7% of per-token memory time in the isolated bandwidth test. The output head dominates traffic (17.3 ms for a 1.5MB PSRAM scan in that bench). PLE’s bet is confirmed: the big table is nearly free; the head is the tax.

Per-Layer Embeddings, MCU Edition

PLE (Gemma) injects embedding information per layer rather than only at the bottom. Ablations matter:

• PLE beats same-core SRAM-fitting baseline by ~16× seed noise.
• Per-layer injection beats bottom-only fat embedding by 0.046 nats.
• Control ple_notable (plumbing, no table) is worse than baseline at small vocab — the table does the work, not the adapters.
• Vocab matters: at 4096, PLE’s edge is tiny (+0.025); at 32768 it is 4× larger — more rows, still sparse.

Quote carefully: 28.9M is stored parameters via a memory-hierarchy split — not a claim that this thinks like a 28.9M dense model.

4-bit PTQ: PLE’s edge is fully retained (even slightly larger vs baseline after quant) because a large redundant table with group scales is more robust than a tiny dense core where every weight is precious. No QAT required for the headline flash artifact.

Speed Ladder on Silicon

Portable C runtime matched PyTorch golden to max abs diff = 0.00001 across 32,768 logits before device compile.

Profile (int8 path, ms/token wall): head 57.6 | attn 25.6 | ple 8.5 | ffn 6.9 | input 4.4. The head is PSRAM-bandwidth-bound (~40 ms read floor at 60.7 MB/s for 2.43 MB int8 weights/token). SIMD alone maybe ~15% more; real levers are fewer bytes (int4 head + unpack) or a smaller/factorized head.

Author left the messy history in-repo — including a parameter-accounting bug that inflated an early number, then the correction. That honesty is part of why HN trusted it.

What It Does / Does Not

Does: coherent TinyStories continuations on-device (example from RESULTS: Once upon a time, there was a little girl named Lily…).

Does not: Q&A, instructions, code, world knowledge. Ceiling is the dense core; PLE does not move that.

HN immediately floated Inflect-class TTS on the same board for offline read-aloud — plausible param budgets, separate engineering. Another thread: layer-per-MCU clusters with RP2350 PIO — speculative; bandwidth between chips is the hard part.

Same Idea, Different Scale

Karpathy’s llama2.c is spiritual ancestry (tiny LM in plain C) — RESULTS.md says no code/checkpoints derived from llama2.c or DaveBben; credit TinyStories + Gemma PLE design.

Builder Checklist

1. Read RESULTS.md before citing tok/s — use end-to-end 9.5, not the 58 tok/s bandwidth ceiling.
2. Flash from firmware/esp32_llm/ — wiring notes for the 1.3" OLED (SH1106, pins 18/46 in recent commits).
3. Train/ablate from src/ + experiments/ if you change vocab or ple_dim.
4. Do not market as “local ChatGPT.” Market as memory-hierarchy LLM.
5. If you port PLE to another MCU, measure random flash row latency first — the thesis dies if row fetch isn’t cheap vs compute.

For product thinking, treat this as a sensor-side narrative generator: a badge, toy, or offline kiosk that prints a short story without a radio. Pairing with Wi-Fi sensing on ESP32 is a stretch conceptually — different RF stack — but the BOM overlap is real: same chip family, same flash discipline, same “ship without a cloud bill” constraint. If you need daily-driver chat quality, stay on llama.cpp; if you need “proof the hierarchy works,” flash this firmware and watch the OLED.

Honest Limitations

• Domain = TinyStories only.
• S3 SIMD unused in shipping dots — head still scalar int8.
• Not a speed crown vs phones; it’s a fit crown.
• Flash wear / XIP details matter in production firmware.
• HN’s “why bother on MCUs” critique: fair for products; wrong for education and offline sensors.

The useful mental model: capacity lives where bandwidth is sparse-friendly; compute lives where latency is low. Once you internalize that, the ESP32 demo stops looking like a stunt and starts looking like a design pattern you can steal for any flash-backed table — embeddings, codebooks, or expert routing stubs — as long as you measure the row fetch before you celebrate the param count.

HN Takeaways Worth Keeping

Fun and learning are legitimate. Damascene constraint projects teach memory hierarchy better than slides.

Milk-V / RP2350 comments are about adjacent price points — Linux boards with more RAM change the problem; this post’s claim is specifically “PLE on ESP32 SRAM/flash,” not “best $5 AI board.”

Scale-up to NVMe + CPU is Colibri territory: same sparse-weight intuition, brutal seconds-per-token economics. Don’t confuse MCU demo tok/s with desktop disk-offload tok/s.

TTS on the same chip is a product idea, not shipped here — Inflect-Micro-class models are in the right param ballpark but need their own flash/PSRAM budget and audio path.

Related on explainx.ai

• BitNet on a 1975 6502 / BBC Micro — ~52K ternary params in ~25KB, no multiply instruction
• Moonshine Micro — voice on RP2350 microcontrollers
• Colibri — GLM-5.2 streaming from disk on 25GB RAM
• Inflect-Micro-v2 — local TTS under 10M params
• What is llama.cpp?
• Closed-source AI vs local open-source alternatives
• RuView — Wi-Fi sensing on ESP32
• MiniCPM / tiny model breakthroughs

Primary sources: GitHub — slvDev/esp32-ai · in-repo README.md / RESULTS.md · TinyStories (Eldan & Li, arXiv:2305.07759) · Google Gemma Per-Layer Embeddings docs · Hacker News discussion (July 2026).

SRAM, tok/s, and parameter splits are as published in RESULTS.md (validated ~2026-07-21). Re-check the repo before shipping a board — firmware pins and accounting notes move.