The Stack v3: 5 Trillion Open Code Tokens from Hugging Face

On July 23, 2026, Hugging Face and BigCode released The Stack v3 — billed as the largest open code dataset yet: on the order of ~5 trillion deduplicated, filtered training tokens, distilled from a ~114 TB GitHub crawl spanning 770 languages and 224 million repositories.

The killer UX fix: contents are inline. v2’s “cool dataset, where’s the code?” complaint is gone.

TL;DR — What People Are Asking

v1 → v2 → v3 at a Glance

Language growth examples called out in the release: C++ ×15, TypeScript ×7.5, Rust ×7, Python ×4.8 vs prior filtered sets — partly two more years of open source, partly a dedup bug fix discovered in v2.

Which Split to Use

Citation (Lozhkov et al., 2026) ships on the dataset card — keep it in papers and model cards.

Why Builders Care

1. Repository context — v3 is built for full-repo pretraining, not only single-file soup. That matches how coding agents actually work.
2. Reproducibility — open weights without open data is half a story; Stack v3 is the data half for code (Echo-style open stacks need corpora this big).
3. Cyber + agents — dense multi-language code is also fuel for defense models and static-analysis assistants — see Sakana Fugu cyber orchestration for the harness side.
4. Cost reality — 5T tokens is a cluster problem. Most teams will continual-pretrain or distill, not train from scratch.

Practical Training Recipes (Not From-Scratch Fantasy)

Very few organizations will pretrain a 100B+ coder on all ~5T tokens end-to-end. More realistic patterns:

Always hold out your evals — SWE-bench-like suites, internal PR fix rates, compile/test loops. Chartography-style public benches for vision taught the same lesson as code: tooling and data mix often beat “wait for the next closed model.”

License, PII, and Compliance Reality Check

The Stack v3 release emphasizes permissive licenses and exclusion of restrictively licensed code, plus PII redaction on the train split. That is necessary but not sufficient for regulated buyers:

1. Re-scan — Run your own license classifiers; do not outsource judgment to a dataset card paragraph.
2. PII residual risk — Redaction is heuristic. Treat accidental secrets in comments/history as expected noise and scrub again if you fine-tune for customer support or enterprise code.
3. Opt-out / takedown — BigCode-era datasets historically honored developer opt-outs; confirm the v3 process on the Hub card before you assume yesterday’s list still applies.
4. Training canaries — If you publish models, document Stack v3 provenance so auditors can reproduce claims.
5. Geographic policy — Some enterprises ban training on public GitHub crawls entirely; Stack v3 does not change that policy — it only makes the crawl transparent.

Storage, Networking, and Pipeline Tips

• Prefer stack-v3-train first. Prove your tokenizer, packing, and loss curves on the filtered set before burning weeks on the full bucket.
• Plan object storage early. 15.9 TB train is awkward on a single workstation SSD; use parallel readers and sharded caches.
• Watch Hub rate limits. Enterprise mirrors or regional replicas save wall-clock when twenty GPUs starve waiting on HTTPS.
• Dedup again for your mix. Near-dedup globally ≠ optimal for your language mix; re-cluster if you oversample popular mono-repos.
• Log data cards. Record commit hashes / Hub revisions; Hub datasets move.

How Stack v3 Fits the 2026 Open Stack

Open weights alone are no longer the scarce story — Echo, Fireworks routing studies, and Asia-origin models already push quality/cost frontiers. What remains scarce is transparent, large, permissively framed code data with repository structure. Stack v3 is that substrate. Closed labs will keep proprietary crawls; open labs finally have a default answer when someone asks “what did you train the coder on?”

Pair Stack v3 with honest evals and a router, and you get closer to the “right model per task” thesis without renting a frontier default for every autocomplete.

Migration Notes from v2

Still run your own license scanners and PII checks for regulated deployments. “Permissive release” ≠ “your counsel waived review.”

FAQ-Style Edge Cases Builders Hit

Can I train a commercial coding agent only on stack-v3-train? Many teams will, but you still need instruction data, tool-use traces, and product evals. Stack v3 is the pretraining substrate, not the whole recipe.

Should I include the full 114 TB bucket? Only if you are studying dedup science or rebuilding filters. Production trainers almost always start from the filtered train split and optionally blend a few custom private corpora.

How does this interact with Asia open-weight models? Strongly. Models like GLM and Kimi already compete on coding; Stack v3 gives Western and open labs a shared public crawl to close data gaps without pretending proprietary GitHub dumps are reproducible (GLM context).

Is v3 “done”? Hub datasets revise. Pin revisions in your training manifests the way you pin Docker digests.

Honest Limitations

• Training cost — storage + GPU for 15.9 TB train is non-trivial; budget networking and object storage before GPUs.
• Web-sourced charts/code provenance — respect dataset card and original rights holders.
• Not a chat model — this is pretraining fuel; you still need SFT/RL and product evals.
• Fork filter (5+ stars) — niche private-style forks may be underrepresented.
• Hub bandwidth — plan mirrors / regional caches for full-bucket pulls.
• License residual risk — permissive framing still requires your own classifiers for regulated training.

Bottom Line

The Stack v3 is the open-code community’s self-contained answer to “we need 2025-scale GitHub without black-box crawls.” If you train code models or agents in the open, stack-v3-train is the new default starting point — and stack-v3-full is the research substrate.

Download the train split, run a smoke continual-pretrain on a single language slice, and measure your internal coding agent suite before you celebrate the five-trillion headline. The number is real; the win is whether your harness gets better. Teams already routing open weights through Echo or Fireworks-style studies finally have a public corpus that matches the ambition of those stacks.

Related on explainx.ai

• Echo TraceML — open-weight ensembles
• Kimi K2.7-Code open coding model
• Fireworks — Kimi K3 + Fable routing study
• Sakana Fugu cyber orchestration
• What is an agent harness?
• GLM-5.2 China open-weight response
• OpenRouter Fusion MoA debate
• Wafer GLM-5.2 on AMD MI355X

Sources: Hugging Face / BigCode Stack v3 release notes · HuggingFaceCode/stack-v3-train · stack-v3-full bucket card · Lozhkov et al. 2026 citation

Token counts, language tallies, and license filters reflect the July 23, 2026 Stack v3 dataset cards. Re-check Hub revisions before starting a multi-week pretrain.