explainx.ai350K+ learn from us
TrendingAI News TodayPathwaysSkills
Pricing
explainx.ai

Upskill in AI — 16 free pathways, live workshops & bootcamps, and 50+ courses from practitioners. Plus the skills, tools, and MCP servers to practice on.

follow us

follow on google

Add explainx.ai as a preferred source

corporate training

support@explainx.ai

get started

Find your pathTake Free Evaluation

community

Join the community

learn

mind: share how you thinkpathways — start freeworkshopsbootcampscoursescompare Explainxcertificationsmock testsexplainx universitycorporate traininglearn skills & mcp

discover

skillsmcp serversexplainx mcpestacktoolsmdx readeragentsllmsdesignsdictionarypeopleagi trackerfelony benchranks

company

aboutvisionmissionteaminstructorsteach on explainxpartnershipscommunityhackathonscareers

content

daily AI newsstate of AI — live resultsblogreleasespromptsgeneratorsresource libraryfor LLMsexplainx.ai kids

solutions

all solutionsdeveloper upskillingmarketing upskillingproduct manager upskillingleadership upskilling

newsletter · weekly

Get AI news, tools, and insights in your inbox.

supportcontactprivacytermsdata rightshow we create contentsubmission guidelines

© 2026 AISOLO Technologies Pvt Ltd

explainx.ai

On this page

  • TL;DR
  • Why this repo exists
  • Model lineup
  • Training pipeline (what each script does)
  • Quick start (mini Zero chat)
  • Data files you actually need
  • How minimind-3 differs from minimind2
  • Cost and time claims (what is verified)
  • Extensions: vision and Omni
  • Limitations honest builders should expect
  • Security and licensing
  • Who should use MiniMind this week
  • Comparison to other “tiny LLM” projects
  • MoE path: minimind-3-moe economics
  • Serving: OpenAI-compatible API and Ollama
  • Evaluation scripts
  • Tokenizer design trade-off
  • Classroom lab outline (one weekend)
  • Relationship to Chinese open-model ecosystem
  • What developers on Hacker News said
  • Troubleshooting
  • Related reading
  • Common failure modes on first run
  • Packaging for Hugging Face upload
← Back to blog

explainx / blog

MiniMind: Train a 64M LLM From Scratch on a Single GPU

Open Source, LLM Training, GitHub, Education, PyTorch

Part of AI Tools and Apps

jingyaogong/minimind teaches pretrain, SFT, DPO, and agentic RL with pure PyTorch. How to try MiniMind-3 on a 3090 and what the repo includes.

Nov 17, 2026·8 min read·Yash Thakker
add explainx.ai
go deep
MiniMind: Train a 64M LLM From Scratch on a Single GPU

MiniMind is a trending open-source project that trains a 64M-parameter chat model from scratch in roughly two hours on a single RTX 3090, according to the jingyaogong/minimind README. The repo had on the order of 65,000 GitHub stars when candidates were ranked on November 17, 2026.

If you have only used LoRA on a 70B instruct model, MiniMind is the opposite end of the curriculum: tokenizer, pretrain, SFT, preference optimization, and agent RL with native PyTorch loops instead of one-line Trainer calls. explainx.ai covers adjacent builder paths in Baseten’s frontier inference guide, Qwen Code’s harness, and how labs distill small models.

TL;DR

table · 2 cols
Itemminimind-3 (current line)
Dense params~64M
MoE variant~198M total, ~64M active
StructureAligned with Qwen3 / Qwen3-MoE
Tokenizer vocab6,400 (custom minimind_tokenizer)
LicenseApache 2.0
Fast path datapretrain_t2t_mini.jsonl + sft_t2t_mini.jsonl
Claimed 3090 cost~3 RMB rent for mini path (author estimate)

Why this repo exists

The README argues most “learn LLMs” content stops at inference or LoRA, not full-stage training. MiniMind keeps model code, tokenizer training, datasets, and RL scripts in one tree so you can see how attention, MoE routing, and chat templates connect.

That educational goal matters for agent builders. When you later debug why a 32B tool-calling model drops JSON, it helps to have once watched loss curves on a 64M SFT run you controlled end to end.

Model lineup

Published checkpoints on ModelScope and Hugging Face include:

  • minimind-3 — 64M dense (April 2026 release date in README table)
  • minimind-3-moe — sparse MoE with top-1 on four experts
  • Older minimind2 and minimind-v1 lines for comparison

The minimind-3 config in README uses 8 layers, 768 hidden size, 8 query heads / 4 KV heads, and 32K max positions with RoPE theta 1e6 — deliberately small so embedding and lm_head do not dominate parameters the way a 150k vocab would on a 26M width model.

Training pipeline (what each script does)

All trainers live under ./trainer/:

  1. train_pretrain.py — Next-token prediction on large text jsonl; output checkpoints like pretrain_768.pth.
  2. train_full_sft.py — Supervised chat tuning on conversational jsonl with tool-call examples mixed in.
  3. train_dpo.py — Preference pairs from dpo.jsonl.
  4. train_ppo.py / GRPO variants — RLAIF on rlaif.jsonl and agent trajectories.
  5. train_agent.py — Multi-turn tool-use RL (GRPO / CISPO) on agent_rl.jsonl.

The README also documents distillation, YaRN long-context extension, and experimental diffusion LM plus linear attention branches — optional reading after the mini path works.

Checkpoint resume

Every script accepts --from_resume 1 to pick up ./checkpoints/<name>_resume.pth, including wandb/swanlab run continuity — useful on rented GPUs that preempt.

Quick start (mini Zero chat)

From the README’s “Step 0” flow:

bash
git clone --depth 1 https://github.com/jingyaogong/minimind
cd minimind && pip install -r requirements.txt

Download weights or train:

bash
modelscope download --model gongjy/minimind-3 --local-dir ./minimind-3
# or train:
cd trainer && python train_pretrain.py
cd trainer && python train_full_sft.py
python eval_llm.py --weight full_sft

Multi-GPU uses torchrun --nproc_per_node N train_xxx.py without DeepSpeed required, though DeepSpeed is supported.

For a Streamlit WebUI, the README points to scripts/web_demo.py after copying transformers-format weights into ./scripts/.

Data files you actually need

The dataset README lists multi-gigabyte pretrain_t2t.jsonl and sft_t2t.jsonl for full fidelity, but the mini pair (~1.2 GB + ~1.6 GB) is the author’s recommended fast replicate.

Formats are plain jsonl:

  • Pretrain: { "text": "..." }
  • SFT: { "conversations": [ { "role": "user", "content": "..." }, ... ] } including tool_call turns

Tool-call capability is baked into default SFT data for minimind-3, so you do not run a separate tool SFT stage unless you customize.

How minimind-3 differs from minimind2

The April 2026 minimind-3 update (per changelog) aligns blocks with Qwen3, removes a shared-expert MoE pattern, adds agentic RL scripts, merges reasoning tags via <think> instead of a standalone train_reason.py, and ships OpenAI-compatible serve_openai_api.py with reasoning_content and tool_calls fields.

Weights convert to transformers / llama.cpp / vllm / ollama per README — relevant if you want to compare your 64M run against hosted APIs in the same harness as OpenCode.

Cost and time claims (what is verified)

The README’s table lists ~2.31 h and ~3 RMB for minimind-3 on a single 3090 for one epoch each on mini pretrain and SFT. Those numbers are author benchmarks, not independent audits. Your cost depends on cloud vendor, epoch count, and whether you train MoE.

The “3 块钱” marketing line is rent only — it excludes your electricity, time, and failed runs.

Extensions: vision and Omni

The same organization points to MiniMind-V and MiniMind-O in GitHub Discussions for multimodal lines. This post focuses on the text minimind-3 path trending this week.

Limitations honest builders should expect

  • Quality ceiling: A 64M Zero model will not compete with frontier APIs on reasoning benchmarks.
  • Chinese-forward tokenizer: 6.4k vocab compresses English bytes-per-token; compare models using BPB, not raw PPL, as the README warns.
  • Maintenance burden: You own data hygiene, evals, and safety when you train from scratch — there is no vendor abuse filter.

Security and licensing

Code is Apache 2.0. Dataset cards cite mixed community sources; read each jsonl’s provenance before publishing a derivative model commercially.

Who should use MiniMind this week

table · 2 cols
ReaderAction
Student / career switcherRun mini pretrain + SFT once, plot loss
Agent engineerStudy train_agent.py tool loops, compare to SWE agent benchmarks
Infra engineerSkip training; use repo to sanity-check how small models stress inference stacks

Comparison to other “tiny LLM” projects

Unlike leaderboard-focused distills, MiniMind optimizes for pedagogy and reproducibility. If you only need a small instruct model for prod, downloading Qwen or Phi weights is faster. If you need to explain logits, MoE aux loss, and DPO objectives to a teammate, MiniMind is the better weekend project.

MoE path: minimind-3-moe economics

The minimind-3-moe line advertises 198M total parameters with about 64M active per token using four routed experts, top-1 (README changelog removed a shared-expert design to track Qwen3-MoE more closely). Training time rises to roughly 3.23 hours and ~4.2 RMB on a 3090 for the mini jsonl path — about 50% slower than dense at the same expert sweet spot, which matches the author’s warning that naive MoE without fused kernels gets expensive.

If your goal is learning router auxiliary loss and expert load imbalance, start dense, then flip to MoE once pretrain loss is sane. Jumping straight to 256-expert configs without Triton or DeepSpeed-MoE support will teach you about kernel launch overhead, not robotics.

Serving: OpenAI-compatible API and Ollama

After SFT, MiniMind ships serve_openai_api.py so you can point Open WebUI or FastGPT at a local port with reasoning_content, tool_calls, and open_thinking flags. README also documents ollama run jingyaogong/minimind-3 and vLLM serve once you export weights.

That matters for agent hackers comparing latency of a 64M local model against prefix-cached long-horizon agents: your baseline will be wrong if you only benchmark frontier APIs.

Evaluation scripts

eval_llm.py supports quick sanity prompts; the README lists C-Eval, C-MMLU, OpenBookQA integrations for third-party scoring. Expect modest scores — the educational win is correlating benchmark deltas with SFT data mixes, not chasing GPT-4 numbers.

Tokenizer design trade-off

With only 6,400 tokens, embedding tables stay tiny relative to width. The downside is higher bytes per token for English prose, which inflates apparent context usage versus Qwen’s 150k vocab. When you log training, track tokens/sec and BPB together.

Classroom lab outline (one weekend)

  1. Friday night — Clone repo, run mini pretrain, watch loss for NaNs.
  2. Saturday — Full SFT, sample chats, break with intentional bad data row.
  3. Sunday — Optional DPO on a hundred pairs; write a one-page report comparing pretrain-only vs SFT answers.

Teams onboarding to enterprise agent workshops can use the same outline before touching production Claude accounts.

Relationship to Chinese open-model ecosystem

MiniMind’s changelog explicitly aligns with Qwen3 blocks while remaining independent of Alibaba’s official trainers. Readers following Tencent HY4 or Sarvam Token Factory can treat MiniMind as the opposite scale: proof that pedagogy repos still trend when frontier labs dominate headlines.

What developers on Hacker News said

The story thread tied to minimind had low engagement when we checked (single-digit points). Treat social proof here as GitHub stars and issue velocity, not a heated HN debate.

Troubleshooting

table · 2 cols
SymptomCheck
CUDA false in torchReinstall torch build matching driver
OOM on 24 GBLower max_seq_len in trainer args; use mini jsonl
Garbled chatConfirm eval_llm.py --weight matches last SFT checkpoint
Slow MoEREADME notes 4 experts top-1 is the practical MoE sweet spot
Daily briefFrom Yash · 350K+ students taught

Catch up on AI

One AI story explained in plain English, plus one thing to try. Every day, free. Unsubscribe anytime.

Related reading

  • Qwen Code open terminal agent
  • Baseten efficient frontier inference
  • Together AI Ollama prefix cache
  • StepFun multi-teacher distillation
  • DeepSeek free chat limits
  • SWE-Cowork agent benchmark
  • Echelon Muon MoE training research
  • Sarvam India open-weight inference

Common failure modes on first run

Renters often forget to pin CUDA-compatible torch builds; the README reminds you to print torch.cuda.is_available() before overnight jobs. Another frequent mistake is training on full jsonl with mini max_seq_len, which truncates most documents and yields gibberish chat — match seq length to the dataset readme hints.

Packaging for Hugging Face upload

After you train, README documents merging LoRA into base weights via scripts/convert_model.py before pushing to Hugging Face. That step matters if you want spaces demos or to share a class project without handing teammates a folder of .pth files.

Primary sources: jingyaogong/minimind on GitHub and ModelScope gongjy/minimind-3, accessed November 17, 2026.

Spotted something out of date? Let us know.
Daily briefFrom Yash · 350K+ students taught

Catch up on AI

One AI story explained in plain English, plus one thing to try. Every day, free. Unsubscribe anytime.

  • ✓ Free
  • ✓ One email a day, no spam
  • ✓ Unsubscribe anytime
Yash Thakker

Written by

Yash Thakker

Yash is an AI expert with over 300K learners. Join his workshops →

View Yash Thakker in People in AI →

Related posts

Nov 10, 2026

Jane Street Open-Sources agent_skills With a Socratic Tutor SKILL.md

The janestreet/agent_skills repository gained more than 2,100 GitHub stars within days of its November 2026 debut. It ships one production skill, socratic-tutor, written in the same SKILL.md format AWS Quick and Claude Code use. Here is what Jane Street published and how developers are reacting.

Nov 17, 2026

PhoneBox: Old Android Phone as a Local AI Coding Dashboard

PhoneBox is an MIT-licensed GitHub project that repurposes spare Android hardware as a touch panel for system monitoring, AI coding session limits, and desktop automation without a cloud backend.

Nov 17, 2026

Sapper v2: Open Autonomous Pentest Legion on the Nüwa Agent Framework

GitHub trending sapper-v2 packages an evolvable pentest legion: a commander that dispatches scouts and soldiers in tmux, a scripted watchdog for crashes, and competition results showing 80.18 points on 1 billion tokens.