VibePrevisBench tests whether coding agents can turn director briefs into 4D Blender scenes and graybox proxy videos that controllable video models paint over—242 tasks, dual scoring, and public case replays at vibeprevisbench.github.io. The arXiv paper (2611.15789) comes from researchers affiliated with Kling AI, Tianjin University, BUPT, and other institutions. It fills a gap between SWE-bench-style code fixes and Seedance-class video generation: the intermediate previsualization layer.
If your team ships AI video pipelines, the benchmark answers a practical question: can your agent block a multi-shot scene well enough that Seedance 2.5 or similar models inherit correct camera and timing? The authors argue errors in the proxy propagate to final pixels—a point illustrated on the site with GPT-6 Astra grayboxes fed to Seedance 2.5.
TL;DR
| Question | Answer |
|---|---|
| Tasks | 242 across script, photo, continuation, reference video |
| Delivery | Blender .blend + rendered graybox proxy video |
| Gate 1 | Scene Check — deterministic facts in 9 dimensions |
| Gate 2 | Screen Check — VLM judge on visibility and story |
| Agents tested | 11 frontier coding agents (site leaderboard) |
| Demo stack | Grayboxes by GPT-6 Astra; paint pass by Seedance 2.5 |
| Primary site | vibeprevisbench.github.io |
Where this sits in the video stack
Modern controllable video workflows separate layout from appearance:
- A coding or layout agent builds geometry, character blocking, and camera cuts in Blender.
- The agent renders a graybox proxy (depth-like or flat-shaded preview).
- A video diffusion model conditions on that proxy to output photoreal frames.
explainx.ai has tracked the consumer side in Seedance 2.5 pricing claims and Fish Audio's controllable drama audio. VibePrevisBench scores step 1–2—the part studios historically paid previz artists to do in Maya or Blender before dailies.
The paper's Table 1 (reproduced on arXiv) contrasts prior benchmarks: many cover text-to-video pixels or short VQA, but few jointly require multi-shot staging, timed actions, 4D scene state, and per-fact ground truth. VibePrevisBench marks every column yes where competitors are partial or no.
Video storyboard panels illustrating VibePrevisBench graybox proxy videos before photorealistic generation
Task design: one brief, four input modes
Each task gives an agent:
- A natural-language brief (and sometimes images or reference clips).
- A Blender environment where it must write and execute Python.
Success means an importable scene file plus a proxy video that respects duration, fps, resolution, and codec constraints. Tasks span long, multi-shot setups—not single-table coding puzzles.
The site groups cases by scene archetype (market, classroom, street chase, etc.). Viewers can re-rank agents per scene, highlighting that no single agent wins everywhere—useful if you route models like Tencent Hy4's Blender computer-use claims versus lighter harnesses.
Three delivery gates before scoring
The benchmark zeroes tasks that fail infrastructure checks:
- Code runs — valid syntax, standalone execution, tagging toolkit untouched.
- Renders complete — full frame range decodes at specified resolution and fps.
- Scene opens —
.blendreloads with correct timeline settings.
Only then do Scene Check and Screen Check run. This design punishes agents that narrate success in chat without shipping binaries—a failure mode common in SWE-GAME-style game builds.
Scene Check (rule-based)
Ground-truth facts become measurable predicates on exported scene state: subject positions, routes, turn timing, camera paths, and cut points. Checks use registered tolerances and partial credit. Example on the site: start-point error 0.46 m with subscores for route vs corner placement.
This is closer to simulation regression testing than to BLEU on captions—aligned with envisioning agentic robotics surveys that separate world models from policy chatter.
Screen Check (VLM judge)
A Gemini 4 judge (with cross-checks against GPT-6.1 Sol and Qwen3.8 Omni Flash) answers structured questions about the proxy video: recognizable subjects, readable actions, visible key moments, and blind story recap order. Judges score only facts Scene Check already passed, preventing "looks pretty" passes when geometry is wrong.
Authors report Spearman 0.92–0.96 rank correlation between judges across 2,628 shared runs and 0.90–0.94 correlation with 242 human pairwise preferences—strong for leaderboard stability, though absolute VLM scores differ by ~14 points.
Agent failure modes diagram for coding agents that pass syntax gates but fail VibePrevisBench spatial checks
Findings: multimodal space beats eloquence
Across 11 agents, the paper highlights:
- Multimodal understanding separates leaders—especially 3D space and camera motion from flat text briefs.
- Basic staging (standing subjects, simple pans) is often solved; hard constraints (precise corners, multi-beat timing) break weaker agents.
- Self-inspection loops exist—agents render previews—but error correction still fails, mirroring Claude Opus TouchSynth PCB stories where vision without verification wastes iterations.
The public site opens two typical cases per scene type with all 11 agents' proxy videos and downloadable interactive 3D exports—far richer than a single aggregate accuracy number.
Seedance, Kling, and builder takeaways
Kling AI's video products compete in the same director-controlled generation space as ByteDance's Seedance line. VibePrevisBench makes the division of labor explicit: agents block; diffusion paints. Teams optimizing inference cost on Seedance 2.5 Flash should still budget agent GPU time for Blender iterations—cheap video per second does not fix bad previz.
Practical checklist:
| Step | Action |
|---|---|
| Baseline | Pick one scene type on the site and rerun your agent harness against published briefs |
| Metrics | Track Scene Check dimension failures, not only final MP4 aesthetics |
| Harness | Compare official vendor coding agents vs generic Terminus-style wrappers |
| Product | Gate user-facing "script-to-video" on proxy pass rate, like Cinecode gates film skills on ffmpeg outputs |
What builders ask after reading the leaderboard
Teams comparing MCP or skill-based Blender control (Google Stitch skills) against monolithic agent loops want to know whether skills reduce spatial errors. VibePrevisBench suggests item-by-item verification against the brief beats a single "render and hope" pass, because Screen Check only awards points when Scene Check facts already hold.
Judge agreement and why ranks stabilize
The site publishes a correlation matrix between VLM judges and humans that matters for anyone borrowing Screen Check as a CI gate. Across 2,628 runs scored by Gemini 4, GPT-6.1 Sol, and Qwen3.8 Omni Flash, Spearman ρ between judges runs 0.94–0.96, and Pearson r on agent means reaches 0.97–0.99. That means a cheaper judge model can substitute for leaderboard ordering even when absolute scores shift.
Against humans, the paper reports 242 pairwise duels—one pair of proxy videos per task—with annotators picking the clip that is easier to understand. Spearman ρ between human win rates and each judge's mean scores lands 0.90–0.94. Practically, if your agent beats another by 10+ points on Scene Check, humans usually agree on the winner; if agents tie within noise on geometry, do not trust Screen Check alone to break the tie.
Authors note GPT-6.1 Sol declined 20 runs under its content policy and that 14 runs failed rendering gates—those exclusions are why explainx.ai treats the public leaderboard as research-grade, not a vendor SLA certification. Replicate the gates locally before you paste VibePrevis numbers into a pitch deck.
Interactive cases and what to watch in replays
The flagship UX on vibeprevisbench.github.io is not the aggregate bar chart. It is the per-scene case browser: two representative tasks per scene type, each showing inputs, eleven proxy videos, and downloadable interactive 3D scene files. When you scrub timelines, look for recurring failure families:
- Camera drift — agents nail subject placement in Scene Check but VLM Screen Check marks partial because heads clip at frame edges during close-ups.
- Route errors — pedestrian paths pass coarse distance checks yet miss turning maneuvers called out in briefs; the site quotes 0.46 m start-point error as an exemplar.
- Multi-shot continuity — agents stage shot one correctly then reset world state between cuts, failing hidden continuity facts even when each isolated render looks fine.
Those patterns mirror DeltaTau bench findings that agents handle single-tool success but stumble when environment state must persist across turns—except here the environment is literally a timeline.
For studios already using Claude Code skills for film pipelines, VibePrevisBench suggests adding a Blender previz skill whose acceptance tests are Scene Check dimensions, not subjective director notes. Pair that with Seedance 2.5 Flash only after proxies pass, so you do not waste diffusion budget beautifying wrong blocking.
How this compares to other agent benchmarks
| Benchmark family | What it measures | VibePrevisBench difference |
|---|---|---|
| SWE-bench / Terminal-Bench | Patch repos, run tests | Measures 3D scene + video deliverables |
| VBench / video quality suites | Final pixel fidelity | Scores graybox layout upstream of diffusion |
| Robotics sim evals | Physics and control | Focuses on director brief → Blender code |
| SWE-GAME | Game builds in Godot | Same long-horizon vibe, different engine |
None of those replace VibePrevisBench; they explain where to plug it in a CI matrix. Use Terminal-Bench for tooling, VibePrevisBench for previz, and Seedance for final pixels—three gates, three budgets.
Limitations
- November 2026 release — confirm task JSON and harness drops on arXiv updates before baking into CI.
- Judge dependence — Screen Check inherits VLM blind spots; human agreement is high on ranks, not every edge frame.
- Kling affiliation — authors include Kling AI; treat leaderboard ordering as research, not a vendor certification.
- Compute — full 242-task sweeps across 11 agents are not laptop-friendly; sample by scene type first.
Related reading
- Seedance 2.5 4K video release
- Seedance 2.5 Flash price claims
- Tencent Hy4 Blender computer-use positioning
- Cinecode: Claude Code skill for films from code
- Google Stitch agent skills for coding tools
- SWE-GAME: agents building games
- Agentic robotics survey roadmap
Paper version, site copy, and model names are accurate as of arXiv 2611.15789 and vibeprevisbench.github.io accessed November 16, 2026.
