MiniMind is a trending open-source project that trains a 64M-parameter chat model from scratch in roughly two hours on a single RTX 3090, according to the jingyaogong/minimind README. The repo had on the order of 65,000 GitHub stars when candidates were ranked on November 17, 2026.
If you have only used LoRA on a 70B instruct model, MiniMind is the opposite end of the curriculum: tokenizer, pretrain, SFT, preference optimization, and agent RL with native PyTorch loops instead of one-line Trainer calls. explainx.ai covers adjacent builder paths in Baseten’s frontier inference guide, Qwen Code’s harness, and how labs distill small models.
TL;DR
| Item | minimind-3 (current line) |
|---|---|
| Dense params | ~64M |
| MoE variant | ~198M total, ~64M active |
| Structure | Aligned with Qwen3 / Qwen3-MoE |
| Tokenizer vocab | 6,400 (custom minimind_tokenizer) |
| License | Apache 2.0 |
| Fast path data | pretrain_t2t_mini.jsonl + sft_t2t_mini.jsonl |
| Claimed 3090 cost | ~3 RMB rent for mini path (author estimate) |
Why this repo exists
The README argues most “learn LLMs” content stops at inference or LoRA, not full-stage training. MiniMind keeps model code, tokenizer training, datasets, and RL scripts in one tree so you can see how attention, MoE routing, and chat templates connect.
That educational goal matters for agent builders. When you later debug why a 32B tool-calling model drops JSON, it helps to have once watched loss curves on a 64M SFT run you controlled end to end.
Model lineup
Published checkpoints on ModelScope and Hugging Face include:
- minimind-3 — 64M dense (April 2026 release date in README table)
- minimind-3-moe — sparse MoE with top-1 on four experts
- Older minimind2 and minimind-v1 lines for comparison
The minimind-3 config in README uses 8 layers, 768 hidden size, 8 query heads / 4 KV heads, and 32K max positions with RoPE theta 1e6 — deliberately small so embedding and lm_head do not dominate parameters the way a 150k vocab would on a 26M width model.
Training pipeline (what each script does)
All trainers live under ./trainer/:
train_pretrain.py— Next-token prediction on large text jsonl; output checkpoints likepretrain_768.pth.train_full_sft.py— Supervised chat tuning on conversational jsonl with tool-call examples mixed in.train_dpo.py— Preference pairs fromdpo.jsonl.train_ppo.py/ GRPO variants — RLAIF onrlaif.jsonland agent trajectories.train_agent.py— Multi-turn tool-use RL (GRPO / CISPO) onagent_rl.jsonl.
The README also documents distillation, YaRN long-context extension, and experimental diffusion LM plus linear attention branches — optional reading after the mini path works.
Checkpoint resume
Every script accepts --from_resume 1 to pick up ./checkpoints/<name>_resume.pth, including wandb/swanlab run continuity — useful on rented GPUs that preempt.
Quick start (mini Zero chat)
From the README’s “Step 0” flow:
git clone --depth 1 https://github.com/jingyaogong/minimind
cd minimind && pip install -r requirements.txt
Download weights or train:
modelscope download --model gongjy/minimind-3 --local-dir ./minimind-3
# or train:
cd trainer && python train_pretrain.py
cd trainer && python train_full_sft.py
python eval_llm.py --weight full_sft
Multi-GPU uses torchrun --nproc_per_node N train_xxx.py without DeepSpeed required, though DeepSpeed is supported.
For a Streamlit WebUI, the README points to scripts/web_demo.py after copying transformers-format weights into ./scripts/.
Data files you actually need
The dataset README lists multi-gigabyte pretrain_t2t.jsonl and sft_t2t.jsonl for full fidelity, but the mini pair (~1.2 GB + ~1.6 GB) is the author’s recommended fast replicate.
Formats are plain jsonl:
- Pretrain:
{ "text": "..." } - SFT:
{ "conversations": [ { "role": "user", "content": "..." }, ... ] }including tool_call turns
Tool-call capability is baked into default SFT data for minimind-3, so you do not run a separate tool SFT stage unless you customize.
How minimind-3 differs from minimind2
The April 2026 minimind-3 update (per changelog) aligns blocks with Qwen3, removes a shared-expert MoE pattern, adds agentic RL scripts, merges reasoning tags via <think> instead of a standalone train_reason.py, and ships OpenAI-compatible serve_openai_api.py with reasoning_content and tool_calls fields.
Weights convert to transformers / llama.cpp / vllm / ollama per README — relevant if you want to compare your 64M run against hosted APIs in the same harness as OpenCode.
Cost and time claims (what is verified)
The README’s table lists ~2.31 h and ~3 RMB for minimind-3 on a single 3090 for one epoch each on mini pretrain and SFT. Those numbers are author benchmarks, not independent audits. Your cost depends on cloud vendor, epoch count, and whether you train MoE.
The “3 块钱” marketing line is rent only — it excludes your electricity, time, and failed runs.
Extensions: vision and Omni
The same organization points to MiniMind-V and MiniMind-O in GitHub Discussions for multimodal lines. This post focuses on the text minimind-3 path trending this week.
Limitations honest builders should expect
- Quality ceiling: A 64M Zero model will not compete with frontier APIs on reasoning benchmarks.
- Chinese-forward tokenizer: 6.4k vocab compresses English bytes-per-token; compare models using BPB, not raw PPL, as the README warns.
- Maintenance burden: You own data hygiene, evals, and safety when you train from scratch — there is no vendor abuse filter.
Security and licensing
Code is Apache 2.0. Dataset cards cite mixed community sources; read each jsonl’s provenance before publishing a derivative model commercially.
Who should use MiniMind this week
| Reader | Action |
|---|---|
| Student / career switcher | Run mini pretrain + SFT once, plot loss |
| Agent engineer | Study train_agent.py tool loops, compare to SWE agent benchmarks |
| Infra engineer | Skip training; use repo to sanity-check how small models stress inference stacks |
Comparison to other “tiny LLM” projects
Unlike leaderboard-focused distills, MiniMind optimizes for pedagogy and reproducibility. If you only need a small instruct model for prod, downloading Qwen or Phi weights is faster. If you need to explain logits, MoE aux loss, and DPO objectives to a teammate, MiniMind is the better weekend project.
MoE path: minimind-3-moe economics
The minimind-3-moe line advertises 198M total parameters with about 64M active per token using four routed experts, top-1 (README changelog removed a shared-expert design to track Qwen3-MoE more closely). Training time rises to roughly 3.23 hours and ~4.2 RMB on a 3090 for the mini jsonl path — about 50% slower than dense at the same expert sweet spot, which matches the author’s warning that naive MoE without fused kernels gets expensive.
If your goal is learning router auxiliary loss and expert load imbalance, start dense, then flip to MoE once pretrain loss is sane. Jumping straight to 256-expert configs without Triton or DeepSpeed-MoE support will teach you about kernel launch overhead, not robotics.
Serving: OpenAI-compatible API and Ollama
After SFT, MiniMind ships serve_openai_api.py so you can point Open WebUI or FastGPT at a local port with reasoning_content, tool_calls, and open_thinking flags. README also documents ollama run jingyaogong/minimind-3 and vLLM serve once you export weights.
That matters for agent hackers comparing latency of a 64M local model against prefix-cached long-horizon agents: your baseline will be wrong if you only benchmark frontier APIs.
Evaluation scripts
eval_llm.py supports quick sanity prompts; the README lists C-Eval, C-MMLU, OpenBookQA integrations for third-party scoring. Expect modest scores — the educational win is correlating benchmark deltas with SFT data mixes, not chasing GPT-4 numbers.
Tokenizer design trade-off
With only 6,400 tokens, embedding tables stay tiny relative to width. The downside is higher bytes per token for English prose, which inflates apparent context usage versus Qwen’s 150k vocab. When you log training, track tokens/sec and BPB together.
Classroom lab outline (one weekend)
- Friday night — Clone repo, run mini pretrain, watch loss for NaNs.
- Saturday — Full SFT, sample chats, break with intentional bad data row.
- Sunday — Optional DPO on a hundred pairs; write a one-page report comparing pretrain-only vs SFT answers.
Teams onboarding to enterprise agent workshops can use the same outline before touching production Claude accounts.
Relationship to Chinese open-model ecosystem
MiniMind’s changelog explicitly aligns with Qwen3 blocks while remaining independent of Alibaba’s official trainers. Readers following Tencent HY4 or Sarvam Token Factory can treat MiniMind as the opposite scale: proof that pedagogy repos still trend when frontier labs dominate headlines.
What developers on Hacker News said
The story thread tied to minimind had low engagement when we checked (single-digit points). Treat social proof here as GitHub stars and issue velocity, not a heated HN debate.
Troubleshooting
| Symptom | Check |
|---|---|
| CUDA false in torch | Reinstall torch build matching driver |
| OOM on 24 GB | Lower max_seq_len in trainer args; use mini jsonl |
| Garbled chat | Confirm eval_llm.py --weight matches last SFT checkpoint |
| Slow MoE | README notes 4 experts top-1 is the practical MoE sweet spot |
Related reading
- Qwen Code open terminal agent
- Baseten efficient frontier inference
- Together AI Ollama prefix cache
- StepFun multi-teacher distillation
- DeepSeek free chat limits
- SWE-Cowork agent benchmark
- Echelon Muon MoE training research
- Sarvam India open-weight inference
Common failure modes on first run
Renters often forget to pin CUDA-compatible torch builds; the README reminds you to print torch.cuda.is_available() before overnight jobs. Another frequent mistake is training on full jsonl with mini max_seq_len, which truncates most documents and yields gibberish chat — match seq length to the dataset readme hints.
Packaging for Hugging Face upload
After you train, README documents merging LoRA into base weights via scripts/convert_model.py before pushing to Hugging Face. That step matters if you want spaces demos or to share a class project without handing teammates a folder of .pth files.
Primary sources: jingyaogong/minimind on GitHub and ModelScope gongjy/minimind-3, accessed November 17, 2026.
