explainx / blog
Jul 23, 2026: Hugging Face + BigCode ship The Stack v3 — ~5T deduped code tokens, 114 TB crawl, 770 languages, contents inline. v2→v3 for code LLMs.
Jul 23, 2026
A 2022 Hugging Face/BigScience project called Petals — run large language models at home, BitTorrent-style — hit the Hacker News front page again in 2026, reigniting a debate about whether peer-to-peer LLM inference is finally viable now that models are smaller, quantization is better, and newer projects like Mesh LLM and AI Horde have taken different approaches to the same problem.
Jul 24, 2026
Buzz is Block’s open-source hive: one community, one signed event log, agents as members with their own keys — not chat bots glued to Slack. explainx.ai maps what works today, the Rust relay stack, and when to self-host vs wait.
Jul 23, 2026
Marcel Rød open-sourced GigaToken, a drop-in-replacement tokenizer claiming gigabytes-per-second throughput and up to 1000x faster encoding than HuggingFace Tokenizers on the same hardware — nearly 1,300 GitHub stars within a day. The gains come from SIMD pretokenization and aggressive caching, and matter more for pretraining data pipelines and time-to-first-token than most people assume.
On July 23, 2026, Hugging Face and BigCode released The Stack v3 — billed as the largest open code dataset yet: on the order of ~5 trillion deduplicated, filtered training tokens, distilled from a ~114 TB GitHub crawl spanning 770 languages and 224 million repositories.
The killer UX fix: contents are inline. v2’s “cool dataset, where’s the code?” complaint is gone.
| Question | Answer |
|---|---|
| Train size? | ~15.9 TB / ~4.9–5T tokens · 713 langs |
| Full corpus? | ~114 TB · 770 langs · 224M repos |
| Crawl cutoff? | GitHub re-crawl through Aug 2025 |
| vs v2 train? | v2 ~550B tokens / 2 TB → ~10× tokens |
| Contents? | Inline (no SWH ID treasure hunt) |
| License stance? | Permissive focus · restrictive licenses excluded |
| Primary cards? | stack-v3-train · stack-v3-full bucket |
| Use for? | Code LLM / agent / cyber-defense pretraining |
| Stack v1 | Stack v2 | Stack v3 | |
|---|---|---|---|
| Cutoff | 2022 | 2023 | 2025 |
| Full | 6.4 TB | 67.5 TB | ~113.7–114 TB |
| Train | ~200B tok | 2.0 TB / ~550B | 15.9 TB / ~4.9–5T |
| Languages (train / full) | 358 | 618 | 713 / 770 |
Language growth examples called out in the release: C++ ×15, TypeScript ×7.5, Rust ×7, Python ×4.8 vs prior filtered sets — partly two more years of open source, partly a dedup bug fix discovered in v2.
| Dataset | Best for |
|---|---|
| HuggingFaceCode/stack-v3-train | Point load_dataset and train — near-dedup, heuristic filters, PII redaction, contents inline |
| HuggingFaceCode/stack-v3-full | Research on raw corpus, custom mixes, cluster IDs, stubs for excluded files (~114 TB bucket) |
Citation (Lozhkov et al., 2026) ships on the dataset card — keep it in papers and model cards.
Very few organizations will pretrain a 100B+ coder on all ~5T tokens end-to-end. More realistic patterns:
| Pattern | What you do | When it fits |
|---|---|---|
| Continual pretrain | Start from an open base (GLM, Kimi, Qwen, DeepSeek-class) and mix Stack v3 shards | You need domain/language lift without full retrain cost |
| Repo-pack SFT | Sample full repositories, pack file trees into long contexts | Agent harnesses that already reason over multi-file edits |
| Language specialty mixes | Upweight Rust/TS/Python slices; downweight noise languages | Product only ships in a few stacks |
| Cyber / vuln corpora | Filter for security-relevant paths and test code | Defense models — pair with Fugu-style orchestration |
| Distillation targets | Use Stack v3 to generate synthetic traces, then train a small student | Edge / cost-sensitive serving |
Always hold out your evals — SWE-bench-like suites, internal PR fix rates, compile/test loops. Chartography-style public benches for vision taught the same lesson as code: tooling and data mix often beat “wait for the next closed model.”
The Stack v3 release emphasizes permissive licenses and exclusion of restrictively licensed code, plus PII redaction on the train split. That is necessary but not sufficient for regulated buyers:
stack-v3-train first. Prove your tokenizer, packing, and loss curves on the filtered set before burning weeks on the full bucket.Open weights alone are no longer the scarce story — Echo, Fireworks routing studies, and Asia-origin models already push quality/cost frontiers. What remains scarce is transparent, large, permissively framed code data with repository structure. Stack v3 is that substrate. Closed labs will keep proprietary crawls; open labs finally have a default answer when someone asks “what did you train the coder on?”
Pair Stack v3 with honest evals and a router, and you get closer to the “right model per task” thesis without renting a frontier default for every autocomplete.
| v2 pain | v3 fix |
|---|---|
| Fetch contents via Software Heritage IDs | Sources embedded |
| Stale 2023 snapshot of older crawls | Fresh Aug 2025 crawl |
| ~550B train tokens | ~5T filtered tokens |
| Dedup quirks | Release notes call out a v2 dedup bug discovered and fixed |
Still run your own license scanners and PII checks for regulated deployments. “Permissive release” ≠ “your counsel waived review.”
Can I train a commercial coding agent only on stack-v3-train? Many teams will, but you still need instruction data, tool-use traces, and product evals. Stack v3 is the pretraining substrate, not the whole recipe.
Should I include the full 114 TB bucket? Only if you are studying dedup science or rebuilding filters. Production trainers almost always start from the filtered train split and optionally blend a few custom private corpora.
How does this interact with Asia open-weight models? Strongly. Models like GLM and Kimi already compete on coding; Stack v3 gives Western and open labs a shared public crawl to close data gaps without pretending proprietary GitHub dumps are reproducible (GLM context).
Is v3 “done”? Hub datasets revise. Pin revisions in your training manifests the way you pin Docker digests.
The Stack v3 is the open-code community’s self-contained answer to “we need 2025-scale GitHub without black-box crawls.” If you train code models or agents in the open, stack-v3-train is the new default starting point — and stack-v3-full is the research substrate.
Download the train split, run a smoke continual-pretrain on a single language slice, and measure your internal coding agent suite before you celebrate the five-trillion headline. The number is real; the win is whether your harness gets better. Teams already routing open weights through Echo or Fireworks-style studies finally have a public corpus that matches the ambition of those stacks.
Sources: Hugging Face / BigCode Stack v3 release notes · HuggingFaceCode/stack-v3-train · stack-v3-full bucket card · Lozhkov et al. 2026 citation
Token counts, language tallies, and license filters reflect the July 23, 2026 Stack v3 dataset cards. Re-check Hub revisions before starting a multi-week pretrain.