PAN 2 is an open 29-billion-parameter world model MBZUAI's Institute of Foundation Models (IFM) published on Hugging Face at IFM/PAN2. It generates up to 20 seconds of video from text actions plus an optional starting image, letting agents practice simulative reasoning: imagine several plans, render each outcome, then act on the best simulation.
The release follows other world-model research threads explainx.ai tracks, from embodied AI exposure benchmarks to VibePrevisBench 4D scenes. PAN 2 is unusual because it ships Apache 2.0 weights, inference scripts, and guardrail guidance in one card.
Architecture: GLP plus dual backbones
Wireframe cube over a grid floor, illustrating latent world-model prediction before agent action
PAN 2 uses Generative Latent Prediction (GLP) from the team's Critique of World Models paper on Hugging Face:
| Component | Role |
|---|---|
| Conceptual reasoner (Qwen3.5-9B) | Reads observation plus actions; writes structured, time-grounded world-state captions |
| 29B MMDiT diffusion backbone | Predicts video latents conditioned on captions and optional first-frame VAE latent |
| Wan 2.2 VAE decoder | Renders latents into 704×1248 video at 24 fps |
The language model handles symbolic planning; the diffusion stack handles pixels. MBZUAI did not fine-tune the frozen text encoder or VAE; it trained the perceptual backbone on hundreds of millions of videos with shared structured annotations.
How to run PAN 2 locally
The model card documents a conda environment on Python 3.14, PyTorch 2.12.1, and flash-attn 4 for CUDA 13.2. Steps:
hf download IFM/PAN2for transformer weights, thenscripts/convert_checkpoint.py to-native.- Download Qwen/Qwen3.5-9B and Wan-AI/Wan2.2-TI2V-5B-Diffusers into
HF_HOME. - Launch
bash inference/inference.sh <CKPT_PATH> <OUTPUT_DIR>with one worker per GPU.
Defaults use 50 steps, CFG 2.5, 241 frames (~10 s) unless prompts specify up to 481 frames (20 s). Outputs land in t2v or i2v folders with JSON configs; reruns skip finished clips.
For free-text prompts, MBZUAI ships a PAN 2 Conceptual Reasoner agent skill plus prompts_rewritter.py so free language becomes structured captions the diffusion model expects.
Benchmarks MBZUAI reports
The card lists single-action tests (300 cases) and multi-step plans (100 tasks, 456 steps) with Kimi K3 as the default conceptual reasoner unless noted.
Public leaderboard entries include Physics-IQ Verified, PhyGround, and RBench, plus human Elo comparisons across 1,400 pairwise judgments. Closed API models ran once through fal in some tables; open PAN 2 runs used multiple seeds.
Verification discipline: MBZUAI admits long plans fail often (~two in three contain a missed step), appearance drifts across clips without memory, and models may shortcut physics (pasta appearing before draining). Those limitations matter for robotics teams comparing PAN 2 to Unitree or Figure demos.
Blocks inside a wireframe scene cube, illustrating simulative spatial reasoning before robots act
Safety: pan2-guardrail is mandatory in docs
Filmstrip frames with one highlighted cell, illustrating short generated video plans agents compare before acting
MBZUAI insists on pan2-guardrail with:
- Qwen3Guard-Gen-4B prompt screening
- Q16 per-frame classification after generation
Internal stats claim harmful T2V drops from 33.9% to 1.5%, with 11.5% false positives on benign T2V prompts. Out-of-scope uses include deepfakes, misinformation as real footage, and unsupervised safety-critical control.
Builders shipping simulators to users should treat guardrails like production middleware, similar to how we advise CSAM classifiers in OpenAI ChatGPT warning posts.
When to pick PAN 2 versus other video stacks
| Use case | PAN 2 fit |
|---|---|
| Agent plan scoring | Strong: simulates multiple text plans cheaply |
| Marketing clips | Weak: structured captions required; guardrails block misuse |
| Game cinematics | Compare against mihoyo Gamer-VLA integrations |
| Scientific video | PAN 2 is entertainment-grade physics; do not confuse with lab instruments |
Pair PAN 2 research with NVIDIA Nemotron for Science if you need verifiable numeric benchmarks instead of generative video alone.
Prompting pipeline in detail
MBZUAI splits generation into two stages on purpose:
- Conceptual reasoner converts messy user text into structured world-state captions with timed actions, actors, and camera motion. The PAN 2 Conceptual Reasoner skill enforces token budgets (soft 1,480 / hard 1,536 tokens) so captions stay inside encoder limits.
- Perceptual backbone renders video conditioned on those captions plus optional first-frame latents.
The card ranks reasoners tested for full-plan completion: Claude Opus 5.5, GPT-6 Astra (medium), Kimi K3, Qwen 3.8 Flash. Any instruction-following LLM can substitute, but quality swings wildly; budget for prompt iteration like you would for diffusion CFG sweeps.
Negative prompts ship as structured fields in the example JSON (inference/prompts/examples.json). Copy that schema before hacking your own agent harness.
Training and customization hooks
Configs config/pretrain_29B_720p.yaml and config/sft_29B_720p.yaml expose architecture hyperparameters. You point data paths at your own latents if you plan continued pretraining. That matters for robotics labs with proprietary teleoperation logs who want domain-specific physics without waiting for MBZUAI releases.
Fine-tunes must still respect Wan and Qwen upstream licenses named in THIRD_PARTY_LICENSES.
Limitations MBZUAI admits openly
The Hugging Face card lists failure modes worth quoting in proposals:
- Long-horizon consistency drifts because each clip starts from at most one frame.
- Text actions cannot specify dexterous finger placement.
- Shortcut physics can advance narratives while violating conservation laws.
- Synthetic realism can fool viewers unless you disclose AI generation.
Those limits separate research simulators from production digital twins. Pair PAN 2 with measurement-heavy stacks such as Hubego skilled-labor video training when you need real contact physics.
Competitive landscape November 2026
| System | What it optimizes | License posture |
|---|---|---|
| PAN 2 | Short agent planning via video | Apache backbone |
| Sora-class APIs | Creator workflows | Closed API |
| Kling / Blender agents | 4D scene benchmarks | Mixed |
| NVIDIA Cosmos predict | Industrial digital twins | Vendor specific |
PAN 2's niche is open simulative reasoning with transparent weights. It will not beat closed APIs on aesthetic polish today, but you can fork it inside air-gapped labs.
Deployment tips for agent builders
- Shard prompts across GPUs with
inference.sh; each worker grabs every Nth prompt. - Set
EXTRA_ARGSfor CFG windows when structured captions overspecify motion. - Run pan2-guardrail as a sidecar container, not an optional script.
- Log structured captions alongside MP4 outputs so you can audit failure cases.
- Compare plans with embedding distance before trusting human preference on video alone.
Multi-step planning numbers from the model card
MBZUAI reports 100 egocentric everyday tasks expressed as image-to-video and text-to-video prompts (200 prompts, 456 steps). Metrics include step completion rate and full-plan completion (share of simulations where every step finishes). Mean and standard deviation cover three seeds; closed API rows use a single seed dagger in tables.
Interpreting those figures:
- High single-step accuracy does not imply reliable tool use in physical labs.
- Egocentric household tasks differ from factory floors or surgical suites.
- Kimi K3 as default reasoner means your stack must budget API cost for caption rewriting unless you self-host Qwen3.5.
Human evaluation and Elo methodology
Blind pairwise tests used 140 prompts (90 T2V, 50 I2V) with 13 reviewers across 1,400 comparisons. MBZUAI fit a Bradley–Terry model centered at 1500 Elo with 95% confidence intervals. That is rigorous for research papers, but 13 reviewers cannot represent global safety norms. Run your own red-team if outputs face the public internet.
Integration ideas for agent frameworks
- Wrap PAN 2 behind a plan scoring service in Nasiko-style gateways so agents pay per simulation minute.
- Feed best plans into robotics VLAs only after human approval.
- Store simulations in object storage with content credentials if regulations require provenance.
- Compare against cheaper still-image world models when video fidelity is overkill.
Downloads and traction
Hugging Face shows 374 downloads last month on the card we fetched November 17, 2026, plus 208 likes in the candidate feed. Stars are not the bar for top-lab releases; reproducible weights are.
Environment setup pitfalls
The model card pins Python 3.14, CUDA 13.2 wheels, and flash-attn 4 beta builds. Common failures:
- Mixing CUDA driver versions below what PyTorch 2.12.1 expects on the host.
- Skipping
convert_checkpoint.pysogenerate.pycannot read Diffusers layouts. - Underestimating VRAM when multiple workers each load full 29B weights.
- Forgetting to export
HF_HOMEbefore pulling Qwen and Wan dependencies.
Budget a full day for first successful clip unless you already run Wan 2.2 elsewhere, as in other video agent benchmarks.
Research lineage and citation
If you publish results, cite the PAN 2 card BibTeX entry (pan2_2026) and note which conceptual reasoner and guardrail revision you used. MBZUAI requests misuse reports at pan@ifm.ai.
FAQ-style quick answers for searchers
Can PAN 2 run on one consumer GPU? Not realistically at full 29B BF16; plan datacenter cards or heavy quantization experiments you validate yourself.
Does PAN 2 replace Isaac Sim? No. NVIDIA sim stacks target robotics PD control; PAN 2 targets language-conditioned video imagination.
Is PAN 2 safe for newsrooms? Only with pan2-guardrail and human review; misinformation risk is explicit in the card.
How does it relate to DeepMind world models? Same research lane, different license and encoder stack; compare benchmarks before picking a fork.
Where is the technical report? The Hugging Face papers link 2507.05169 hosts the Critique of World Models write-up that formalizes GLP.
