explainx.ai350K+ learn from us
TrendingAI News TodayPathwaysSkills
Pricing
explainx.ai

Upskill in AI — 16 free pathways, live workshops & bootcamps, and 50+ courses from practitioners. Plus the skills, tools, and MCP servers to practice on.

follow us

follow on google

Add explainx.ai as a preferred source

corporate training

support@explainx.ai

get started

Find your pathTake Free Evaluation

community

Join the community

learn

mind: share how you thinkpathways — start freeworkshopsbootcampscoursescompare Explainxcertificationsmock testsexplainx universitycorporate traininglearn skills & mcp

discover

skillsmcp serversexplainx mcpestacktoolsmdx readeragentsllmsdesignsdictionarypeopleagi trackerfelony benchranks

company

aboutvisionmissionteaminstructorsteach on explainxpartnershipscommunityhackathonscareers

content

daily AI newsstate of AI — live resultsblogreleasespromptsgeneratorsresource libraryfor LLMsexplainx.ai kids

solutions

all solutionsdeveloper upskillingmarketing upskillingproduct manager upskillingleadership upskilling

newsletter · weekly

Get AI news, tools, and insights in your inbox.

supportcontactprivacytermsdata rightshow we create contentsubmission guidelines

© 2026 AISOLO Technologies Pvt Ltd

explainx.ai

On this page

  • Architecture: GLP plus dual backbones
  • How to run PAN 2 locally
  • Benchmarks MBZUAI reports
  • Safety: pan2-guardrail is mandatory in docs
  • When to pick PAN 2 versus other video stacks
  • Prompting pipeline in detail
  • Training and customization hooks
  • Limitations MBZUAI admits openly
  • Competitive landscape November 2026
  • Deployment tips for agent builders
  • Multi-step planning numbers from the model card
  • Human evaluation and Elo methodology
  • Integration ideas for agent frameworks
  • Downloads and traction
  • Environment setup pitfalls
  • Research lineage and citation
  • FAQ-style quick answers for searchers
  • Related reading
← Back to blog

explainx / blog

MBZUAI PAN 2: Open 29B World Model for Simulative Agent Reasoning

Open Models, World Models, MBZUAI, Robotics, Video AI

Part of Robotics and Physical AI

IFM at MBZUAI released PAN 2 on Hugging Face: a 29B image-text-to-video world model for up to 20s simulative reasoning, Apache licensed with guardrails.

Nov 17, 2026·8 min read·Yash Thakker
add explainx.ai
go deep
MBZUAI PAN 2: Open 29B World Model for Simulative Agent Reasoning

PAN 2 is an open 29-billion-parameter world model MBZUAI's Institute of Foundation Models (IFM) published on Hugging Face at IFM/PAN2. It generates up to 20 seconds of video from text actions plus an optional starting image, letting agents practice simulative reasoning: imagine several plans, render each outcome, then act on the best simulation.

The release follows other world-model research threads explainx.ai tracks, from embodied AI exposure benchmarks to VibePrevisBench 4D scenes. PAN 2 is unusual because it ships Apache 2.0 weights, inference scripts, and guardrail guidance in one card.

Daily briefFrom Yash · 350K+ students taught

Catch up on AI

One AI story explained in plain English, plus one thing to try. Every day, free. Unsubscribe anytime.

Architecture: GLP plus dual backbones

Wireframe cube over a grid floor, illustrating latent world-model prediction before agent actionWireframe cube over a grid floor, illustrating latent world-model prediction before agent action

PAN 2 uses Generative Latent Prediction (GLP) from the team's Critique of World Models paper on Hugging Face:

table · 2 cols
ComponentRole
Conceptual reasoner (Qwen3.5-9B)Reads observation plus actions; writes structured, time-grounded world-state captions
29B MMDiT diffusion backbonePredicts video latents conditioned on captions and optional first-frame VAE latent
Wan 2.2 VAE decoderRenders latents into 704×1248 video at 24 fps

The language model handles symbolic planning; the diffusion stack handles pixels. MBZUAI did not fine-tune the frozen text encoder or VAE; it trained the perceptual backbone on hundreds of millions of videos with shared structured annotations.

How to run PAN 2 locally

The model card documents a conda environment on Python 3.14, PyTorch 2.12.1, and flash-attn 4 for CUDA 13.2. Steps:

  1. hf download IFM/PAN2 for transformer weights, then scripts/convert_checkpoint.py to-native.
  2. Download Qwen/Qwen3.5-9B and Wan-AI/Wan2.2-TI2V-5B-Diffusers into HF_HOME.
  3. Launch bash inference/inference.sh <CKPT_PATH> <OUTPUT_DIR> with one worker per GPU.

Defaults use 50 steps, CFG 2.5, 241 frames (~10 s) unless prompts specify up to 481 frames (20 s). Outputs land in t2v or i2v folders with JSON configs; reruns skip finished clips.

For free-text prompts, MBZUAI ships a PAN 2 Conceptual Reasoner agent skill plus prompts_rewritter.py so free language becomes structured captions the diffusion model expects.

Benchmarks MBZUAI reports

The card lists single-action tests (300 cases) and multi-step plans (100 tasks, 456 steps) with Kimi K3 as the default conceptual reasoner unless noted.

Public leaderboard entries include Physics-IQ Verified, PhyGround, and RBench, plus human Elo comparisons across 1,400 pairwise judgments. Closed API models ran once through fal in some tables; open PAN 2 runs used multiple seeds.

Verification discipline: MBZUAI admits long plans fail often (~two in three contain a missed step), appearance drifts across clips without memory, and models may shortcut physics (pasta appearing before draining). Those limitations matter for robotics teams comparing PAN 2 to Unitree or Figure demos.

Blocks inside a wireframe scene cube, illustrating simulative spatial reasoning before robots actBlocks inside a wireframe scene cube, illustrating simulative spatial reasoning before robots act

Safety: pan2-guardrail is mandatory in docs

Filmstrip frames with one highlighted cell, illustrating short generated video plans agents compare before actingFilmstrip frames with one highlighted cell, illustrating short generated video plans agents compare before acting

MBZUAI insists on pan2-guardrail with:

  • Qwen3Guard-Gen-4B prompt screening
  • Q16 per-frame classification after generation

Internal stats claim harmful T2V drops from 33.9% to 1.5%, with 11.5% false positives on benign T2V prompts. Out-of-scope uses include deepfakes, misinformation as real footage, and unsupervised safety-critical control.

Builders shipping simulators to users should treat guardrails like production middleware, similar to how we advise CSAM classifiers in OpenAI ChatGPT warning posts.

When to pick PAN 2 versus other video stacks

table · 2 cols
Use casePAN 2 fit
Agent plan scoringStrong: simulates multiple text plans cheaply
Marketing clipsWeak: structured captions required; guardrails block misuse
Game cinematicsCompare against mihoyo Gamer-VLA integrations
Scientific videoPAN 2 is entertainment-grade physics; do not confuse with lab instruments

Pair PAN 2 research with NVIDIA Nemotron for Science if you need verifiable numeric benchmarks instead of generative video alone.

Prompting pipeline in detail

MBZUAI splits generation into two stages on purpose:

  1. Conceptual reasoner converts messy user text into structured world-state captions with timed actions, actors, and camera motion. The PAN 2 Conceptual Reasoner skill enforces token budgets (soft 1,480 / hard 1,536 tokens) so captions stay inside encoder limits.
  2. Perceptual backbone renders video conditioned on those captions plus optional first-frame latents.

The card ranks reasoners tested for full-plan completion: Claude Opus 5.5, GPT-6 Astra (medium), Kimi K3, Qwen 3.8 Flash. Any instruction-following LLM can substitute, but quality swings wildly; budget for prompt iteration like you would for diffusion CFG sweeps.

Negative prompts ship as structured fields in the example JSON (inference/prompts/examples.json). Copy that schema before hacking your own agent harness.

Training and customization hooks

Configs config/pretrain_29B_720p.yaml and config/sft_29B_720p.yaml expose architecture hyperparameters. You point data paths at your own latents if you plan continued pretraining. That matters for robotics labs with proprietary teleoperation logs who want domain-specific physics without waiting for MBZUAI releases.

Fine-tunes must still respect Wan and Qwen upstream licenses named in THIRD_PARTY_LICENSES.

Limitations MBZUAI admits openly

The Hugging Face card lists failure modes worth quoting in proposals:

  • Long-horizon consistency drifts because each clip starts from at most one frame.
  • Text actions cannot specify dexterous finger placement.
  • Shortcut physics can advance narratives while violating conservation laws.
  • Synthetic realism can fool viewers unless you disclose AI generation.

Those limits separate research simulators from production digital twins. Pair PAN 2 with measurement-heavy stacks such as Hubego skilled-labor video training when you need real contact physics.

Competitive landscape November 2026

table · 3 cols
SystemWhat it optimizesLicense posture
PAN 2Short agent planning via videoApache backbone
Sora-class APIsCreator workflowsClosed API
Kling / Blender agents4D scene benchmarksMixed
NVIDIA Cosmos predictIndustrial digital twinsVendor specific

PAN 2's niche is open simulative reasoning with transparent weights. It will not beat closed APIs on aesthetic polish today, but you can fork it inside air-gapped labs.

Deployment tips for agent builders

  • Shard prompts across GPUs with inference.sh; each worker grabs every Nth prompt.
  • Set EXTRA_ARGS for CFG windows when structured captions overspecify motion.
  • Run pan2-guardrail as a sidecar container, not an optional script.
  • Log structured captions alongside MP4 outputs so you can audit failure cases.
  • Compare plans with embedding distance before trusting human preference on video alone.

Multi-step planning numbers from the model card

MBZUAI reports 100 egocentric everyday tasks expressed as image-to-video and text-to-video prompts (200 prompts, 456 steps). Metrics include step completion rate and full-plan completion (share of simulations where every step finishes). Mean and standard deviation cover three seeds; closed API rows use a single seed dagger in tables.

Interpreting those figures:

  • High single-step accuracy does not imply reliable tool use in physical labs.
  • Egocentric household tasks differ from factory floors or surgical suites.
  • Kimi K3 as default reasoner means your stack must budget API cost for caption rewriting unless you self-host Qwen3.5.

Human evaluation and Elo methodology

Blind pairwise tests used 140 prompts (90 T2V, 50 I2V) with 13 reviewers across 1,400 comparisons. MBZUAI fit a Bradley–Terry model centered at 1500 Elo with 95% confidence intervals. That is rigorous for research papers, but 13 reviewers cannot represent global safety norms. Run your own red-team if outputs face the public internet.

Integration ideas for agent frameworks

  • Wrap PAN 2 behind a plan scoring service in Nasiko-style gateways so agents pay per simulation minute.
  • Feed best plans into robotics VLAs only after human approval.
  • Store simulations in object storage with content credentials if regulations require provenance.
  • Compare against cheaper still-image world models when video fidelity is overkill.

Downloads and traction

Hugging Face shows 374 downloads last month on the card we fetched November 17, 2026, plus 208 likes in the candidate feed. Stars are not the bar for top-lab releases; reproducible weights are.

Environment setup pitfalls

The model card pins Python 3.14, CUDA 13.2 wheels, and flash-attn 4 beta builds. Common failures:

  • Mixing CUDA driver versions below what PyTorch 2.12.1 expects on the host.
  • Skipping convert_checkpoint.py so generate.py cannot read Diffusers layouts.
  • Underestimating VRAM when multiple workers each load full 29B weights.
  • Forgetting to export HF_HOME before pulling Qwen and Wan dependencies.

Budget a full day for first successful clip unless you already run Wan 2.2 elsewhere, as in other video agent benchmarks.

Research lineage and citation

If you publish results, cite the PAN 2 card BibTeX entry (pan2_2026) and note which conceptual reasoner and guardrail revision you used. MBZUAI requests misuse reports at pan@ifm.ai.

FAQ-style quick answers for searchers

Can PAN 2 run on one consumer GPU? Not realistically at full 29B BF16; plan datacenter cards or heavy quantization experiments you validate yourself.

Does PAN 2 replace Isaac Sim? No. NVIDIA sim stacks target robotics PD control; PAN 2 targets language-conditioned video imagination.

Is PAN 2 safe for newsrooms? Only with pan2-guardrail and human review; misinformation risk is explicit in the card.

How does it relate to DeepMind world models? Same research lane, different license and encoder stack; compare benchmarks before picking a fork.

Where is the technical report? The Hugging Face papers link 2507.05169 hosts the Critique of World Models write-up that formalizes GLP.

Related reading

  • Embodied AI exposure benchmark
  • VibePrevisBench 4D agent benchmark
  • Agentic robotics roadmap survey
  • Nemotron for Science open agents
  • Mihoyo Gamer-VLA game agent model
Spotted something out of date? Let us know.
Daily briefFrom Yash · 350K+ students taught

Catch up on AI

One AI story explained in plain English, plus one thing to try. Every day, free. Unsubscribe anytime.

  • ✓ Free
  • ✓ One email a day, no spam
  • ✓ Unsubscribe anytime
Yash Thakker

Written by

Yash Thakker

Yash is an AI expert with over 300K learners. Join his workshops →

View Yash Thakker in People in AI →

Related posts

Nov 17, 2026

ScaleVLA: 552B VLA Backbone Trains Faster on Fewer Demos

Researchers introduced ScaleVLA, pairing a 552 billion-parameter vision-language backbone with a trainable action diffusion transformer and showing better few-shot robot success than 2B VLAs on LIBERO-PRO and hardware tests.

Nov 15, 2026

LeCun at CoRL 2026: Do Not Work on LLMs If You Want Human-Level AI

Conference on Robot Learning 2026 brought Yann LeCun back to an embodied-AI audience with a familiar punchline: large language models are already staggeringly useful, but if human-level intelligence is the target, he argues researchers should bet on JEPA-style world models instead of scaling text predictors.

Nov 13, 2026

NVIDIA BirdDreams: Joint BEV and Camera Driving World Model

NVIDIA researchers published BirdDreams on November 10, 2026 — an autoregressive driving world model that predicts perspective video and schematic BEV video from a single camera frame, with optional ego trajectories in meters and reactive surrounding traffic.