BitNet on a 1975 6502: A Language Model Inside 25KB

Matt Beton trained a tiny Mamba language model, quantized it to ternary BitNet weights, and ran next-token inference on an 8-bit MOS 6502 from 1975 — including on his dad’s BBC Micro Model B. The public sample is childish and misspelled. That is the point: the milestone is fitting an autoregressive stack into ~25KB of userspace on a CPU with no multiply instruction, not beating GPT on MMLU.

The June 2026 write-up hit Hacker News in early August. explainx.ai’s read: this is the extreme end of the same edge-AI story we covered with 28.9M params on an ESP32 and ternary Neutrino-1 8B — except the constraint is a machine from the Apple II / BBC Micro era.

TL;DR — what people are asking

The constraint that forces every design choice

The MOS 6502 powered the BBC Micro and Apple II. Beton’s budget is brutal:

• ~25KB for model and inference in userspace (of ~32KB RAM)
• 8-bit integer datapath only
• No multiply in the ISA — an 8×8 MAC costs ~150 cycles via shift-add; a ternary accumulate is ~30

Final split he reports: 9KB inference code (CC65 C → 6502) and 13KB packed weights.

Loading onto the real machine is delightfully period-correct: train on a MacBook, build a UEF tape image, play it through a DIY 3.5mm-to-tape cable with PlayUEF so the BBC thinks a cassette drive is talking. Parity checks run through sim65; full UX dry-runs on jsbeeb before the physical board.

Sample output (calibrate expectations)

From Beton’s post, generated on the BBC Micro:

once upon a time tom and lily saw things lily were sad her house he heartd them ilily and tom said yes she saw a little girl smiled tom was so excited her mom said yes

That is coherent enough to prove next-token generation is working — and bad enough to kill “ChatGPT of 1975” headlines. Beton trains for spelling and simple grammar forms, not instruction following.

Why BitNet: multiply is the enemy

BitNet-style ternary weights live in {-1, 0, +1}. Matrix multiply collapses to skip / add / subtract of activations — perfect when the CPU cannot multiply.

Storage math he uses:

• theoretical ~1.58 bits/param (log2(3))
• practical pack: 4 params/byte (2 bits each) so unpack is a right-shift
• 5-per-byte is denser but needs floor-divide-by-3 — painful on 6502
• 13KB × 4 ≈ 52K ternary parameters

Training uses the usual straight-through estimator: float32 master weights, ternary forward, full-precision backward. The LM head stays int4 so the 27-way character distribution still has enough resolution; other matrices are ternary.

This is the same quantization family as modern ternary releases like Fermion Neutrino-1 — just scaled down until it fits beside a cassette port.

Why Mamba: memory must not grow

Transformers pay for attention with a KV cache that grows every token. On 32KB RAM, that cache steals the bytes you wanted for weights. Beton does not need needle-in-a-haystack recall — only short-term memory for spelling and toy grammar.

Recurrent / SSM models keep a fixed-size state h:

(token_i, h)  ->  (token_{i+1}, h')

Same compute shape every step. He tried GRU-style recurrence; under ternary weights the spectral radius blows up and training diverges unless the big matrices become int4 — which shrinks the model. Mamba uses a per-channel decay in [0, 128)/128, so the update magnitude stays ≤ 1 by construction. That is why Mamba wins the hardware lottery here.

Integer plumbing that keeps it alive

The core kernel is a packed ternary linear: for each 2-bit code, skip / += x / -= x, then shift_sat_int8. That primitive composes into the Mamba block. Full C lives in the repo’s inference tree.

One subtle training detail: shr is extremely sensitive — bumping it by one doubles or halves post-activation magnitude. Beton lets scale parameters move in the first half of training, then freezes them. That is the kind of detail that never shows up in a GPU-first recipe and dominates when every bit of dynamic range is rented from an int16 accumulator.

Sampling is another hardware negotiation. Softmax wants exp. The 6502 gets a precomputed lookup table of round(255 * e^{-d/T}) for temperature ≈ 0.9, subtract-max for stability, then a 16-bit pseudo-random draw. There is no entropy source without user input, so the seed is fixed and every run emits the same “random” story — which is actually helpful when you are debugging an emulator parity gap.

What Hacker News argued about

A few recurring threads from the August discussion are worth keeping:

Banked memory. Could you stream weights from banked RAM while keeping only the hot inference core resident? Era-appropriate, yes — and it would change the “everything must fit in 25KB” framing into a paging problem. Beton optimized for a single flat userspace footprint; banking is a natural sequel, not a gotcha that invalidates the demo.

Hand-written assembly. Several commenters who have shipped NES / cc65 projects guessed that hand assembly could reclaim both bytes and cycles. Fair. The 6502 is a famously awkward C target. Starting in C with a Python parity harness is still the right order: correctness first, then cycle-shaving.

Miniaturization fantasies. “Edge LLM in glasses” is the hopeful leap. BitNet6502 shows that some autoregressive loop can run on absurd hardware. It does not show that useful personal assistants will. The gap from toy story tokens to reliable offline agents is data, evals, sensors, and silicon — not one more clever packing scheme.

The 1975 counterfactual. Would someone in 1975 have recognized this as “language modeling”? Probably not in today’s vocabulary. They would have seen a slow program emitting English-ish characters. The modern punchline is that the algorithms we now treat as inevitable (attention + float matmul) are partly lottery winners of GPU-era hardware.

How to try it

1. Read the primary post: Autoregressive Language Model on the 6502 Processor
2. Clone / browse MattBeton/BitNet6502
3. Use Beton’s in-browser BBC Micro link (UEF from GitHub, auto-typed commands) — generation takes minutes
4. Or build UEF locally (make bbc-uef) for PlayUEF / real hardware

If you only have five minutes, watch the sample text and skim the ternary_linear C kernel. If you have an afternoon, run the browser demo and compare sim65 parity notes in the repo — that harness is the real teaching artifact.

Where this sits on the edge-AI spectrum

BitNet6502 is not competing on quality. It is a stress test of co-design: when the machine changes, the “default” architecture (transformer + float matmul) is no longer rational. Beton explicitly cites Sarah Hooker’s hardware lottery — SOTA algorithms win partly because they match available silicon. Force 1975 silicon, and Mamba + BitNet suddenly look obvious.

That same lesson applies upward: glasses, sensors, and offline gadgets will keep inventing their own lotteries. The quantization guide and local llama.cpp stack are the practical cousins for machines that do have multiply — but still care about bits and cycles.

What builders should steal (and what not to)

Steal:

• Start from the ISA, then pick architecture (not the reverse)
• Prefer fixed-state recurrence when RAM cannot grow a cache
• Match quantization to available ops (ternary when multiply is fake)
• Keep a Python reference + emulator parity harness (sim65) before real iron
• Character vocab when embed matrices would otherwise dominate

Don’t steal as product claims:

• “AI of 1975” marketing without showing the sample text
• Assuming CC65 output is optimal (assembly headroom exists)
• Treating 52K toy LM quality as a miniaturization roadmap to glasses-grade assistants without new data, evals, and silicon

Bottom line

BitNet6502 is a working autoregressive language model on a 1975 8-bit CPU: ~52K ternary parameters, Mamba state, ~25KB userspace, cassette-tape deployment, browser emulator to play along.

It will not replace your coding agent. It will reset your intuition about how tightly model, quantization, and silicon have to be designed together — and why “just run a small transformer” is sometimes the wrong sentence to start with.

Related on explainx.ai

• 28.9M LLM on an $8 ESP32 — Per-Layer Embeddings
• Fermion Neutrino-1 8B ternary weights
• What is llama.cpp?
• What is AI model quantization?
• PrismML Bonsai / phone ternary models
• Transformer architecture & attention
• Inflect-Micro-v2 — local TTS under 10M
• Moonshine Micro — voice on RP2350

Primary sources: Matt Beton — Autoregressive Language Model on the 6502 · GitHub — MattBeton/BitNet6502 · Hacker News discussion (Aug 2026) · Sarah Hooker, “The Hardware Lottery”

Figures for memory split, cycle counts, parameter packing, and architecture choices follow Beton’s June 2026 write-up and public repo as of August 3, 2026. Generation quality remains toy-scale; verify current browser demo links on the author’s site.