LongBanana: Expert Benchmark for Long-Context, Multi-Reference Image Synthesis

LongBanana is a new expert-verified benchmark for long-context, multi-reference image synthesis — the pattern where an agent must read dozens of reference images, ignore distractors, and produce one output that satisfies dense instructions. The project site says it contains 90 tasks, about 32 reference images per case, and 4,670 atomic evaluation items, with code and data public. The paper is arXiv 2611.06856, listed on Hugging Face Papers, and marked accepted to NeurIPS 2026 on the homepage.

That matters because image models already look good on short prompts. The harder product shape is reference-grounded composition: merchandising boards, animation pipelines, interior layouts, and agent loops that pick visual evidence before calling a generator. LongBanana is built to score that loop, not just aesthetic quality.

What problem LongBanana targets

Text-to-image leaderboards reward pretty pixels. They rarely check whether the model used the correct reference object, kept relationships straight, or met count and layout constraints when 30 similar thumbnails compete for attention.

The LongBanana authors frame three skills:

1. Retrieve relevant objects from a large, heterogeneous reference set and reject hard negatives.
2. Bind identity, appearance, and state to the right entities.
3. Compose one image that satisfies global constraints — counts, spatial layout, temporal state — not only local texture match.

The benchmark is explicitly aimed at agent-driven image generation, where a controller plans steps, calls tools, and may revise output. That aligns with the agentic editing direction in research such as Adobe EditAgent and with broader multimodal context budgeting questions when vision tokens stack up.

Benchmark shape: 90 cases, three suites, nine categories

LongBanana benchmark taxonomy: three suites and nine capability categories Image: LongBanana

The site organizes tasks in a 3×3 taxonomy across three suites of 30 cases each:

Each case ships with an entity–relation graph. Experts turn graph nodes, edges, and whole-graph obligations into checklist items scored as fail (0), uncertain (1), or pass (2). Correct omission when an entity must be absent also earns pass.

Scale highlights from the homepage and paper summary:

• 90 expert-curated cases in the reported benchmark set.
• ~32 references per case.
• ~20 constraints and ~50 evaluation items per task.
• 4,670 expert-specified atomic items in total.
• At least 30 cases include two or more hard-negative references.

Developers can browse simulated targets and full reference boards in the gallery before downloading anything.

Graph-grounded evaluation: why checklists beat one score

LongBanana task overview: references, generation, and checklist evaluation pipeline Image: LongBanana

LongBanana-Eval scores three layers:

• Content fidelity (node level) — required entities present with correct identity and appearance.
• Relational binding (edge level) — attributes and relationships attach to the right entities.
• Global constraint satisfaction (graph level) — counts, layout, and composition hold for the whole scene.

Each checklist item can carry an evidence trace: which references support the judgment, optional OCR or counting tools, and critic outputs before aggregation. The authors report 7,005 double-labeled judgments on a subset with 89.4% exact agreement and 0.83 quadratic weighted kappa — unusually high for dense vision rubrics, though still human-labeled rather than independent third-party audit.

For teams used to RULER-style effective-length tests on text, the analogy is: LongBanana asks where visual context stops being usable, not where a single needle fact can be retrieved.

LongBanana-Gen: agent harness and RPR workflow

LongBanana-Gen agent workflow with retrieve, plan, execute, and reflect Image: LongBanana

The paper introduces LongBanana-Gen, a harness for long-horizon generation. Experiments compare:

• Direct generation from a single model call.
• Minimal agent harnesses with lightweight tool loops.
• ReAct–Plan–Exec–Reflect (RPR) workflows that plan, execute, and repair reference-grounded errors.

The public leaderboard (November 2026 snapshot on the site) highlights that controller choice dominates:

• Strongest reported checklist average in the table: 78.4 / 100 for an OpenClaw + RPR stack with Nano Banana 2.
• Highest task pass rate: 47.8%, shared by OpenClaw + RPR and OpenCode in the manuscript tables.

The site notes a subtle result: RPR raises checklist averages across harnesses, but task pass rates can fall for some controllers if reasoning steps break binding. Better planning is not free if the final generator loses track of references.

That is a lesson for anyone wiring image MCP tools or ByteDance Seedream-style multi-round editors: the orchestration layer needs its own eval, not only the base diffusion model.

How to try LongBanana this week

1. Read the task gallery on long-banana.com — pick one suite that matches your product (retrieval-heavy retail vs layout-heavy interiors).
2. Clone the GitHub repo linked from the paper page and inspect annotation format before you fine-tune.
3. Submit or reproduce leaderboard runs using the same 90-case split; do not mix development QC cases into reported scores.
4. Pair with agent evals your team already runs — coding agents are not substitutes, but agent benchmark hygiene transfers.

Hardware needs depend on your generator stack; the benchmark itself is evaluation data and harness code, not a single shipped foundation model.

How LongBanana compares to nearby benchmarks

It does not replace human preference evals or product A/B tests. It gives a regression suite when you change retrieval, reranking, or tool policies in an image agent.

Why teams should care in November 2026

Image APIs now advertise editing rounds, reference slots, and board-style inputs. Without a benchmark like LongBanana, teams optimize for the first frame that looks plausible, then fail in production when the wrong SKU, character, or layout variant slips through.

The Kolors / KlingAI authorship also signals that video and image labs treating reference banks as first-class inputs are investing in public measurement, not only demo reels. If you build on open weights from Seedream or closed APIs, the harness ideas still apply.

Scoring protocol: what the leaderboard numbers mean

The LongBanana site publishes two headline metrics in its November 2026 tables: checklist-average score on a 0–100 scale and task pass rate at a difficulty-weighted threshold (reported as P at 70% in the UI copy). Checklist averages reward incremental correctness across thousands of atomic items. Pass rate asks whether an entire case clears a stricter bar. A system can improve the average while still failing whole tasks when one global constraint breaks — exactly the failure mode product teams see when a render looks polished but violates a count or layout rule.

The human-audited pool described on the site covers 900 generated outputs across 10 generation conditions and 46,700 reviewed checklist labels for those runs. That scale is expensive, which is why the authors emphasize graph-grounded automation: critics can call OCR, counting, layout, and identity tools before a human adjudicates borderline items. Teams borrowing the benchmark should not confuse the public leaderboard snapshot with a continuously refreshed production harness; it is a frozen manuscript table unless the repo publishes a live submission API.

Inter-rater stats matter for legal and safety buyers, not only researchers. The 89.4% exact agreement and 0.83 quadratic weighted kappa on double-labeled items beat many vision rubrics, but kappa still leaves room for disagreement on edge cases. If you fine-tune on LongBanana scores, hold out entire suites — not random items — so binding and retrieval skills do not leak across categories.

Implementation notes for teams running evals

Reference inventory design. Each case ships with on the order of 32 images plus distractors. Your agent must declare which references it attended to if you want debugging parity with LongBanana-Eval traces. Logging retrieval indices and similarity scores helps explain checklist failures without rerunning generation.

Controller ablations. The RPR result is instructive: adding reflection repaired some binding errors but hurt pass rate for certain open controllers. When you A/B harnesses, report both per-item and per-task metrics. A coding-style lesson from agent harness studies applies here — extra turns buy quality only when the tool surface preserves state.

Compute budgeting. Thirty-two references per step can dominate vision-token cost before diffusion runs. Pair LongBanana with token budget guides so you do not confuse benchmark scores with economical production configs.

NeurIPS acceptance signals peer review of the methodology, not endorsement of any single commercial generator on the leaderboard. Reproduce from the GitHub assets linked on long-banana.com before you cite rank numbers in customer decks.

Related reading

• Multimodal vision token budget guide
• EditAgent and verification records
• AI benchmarks complete guide
• How to read a Hugging Face model card
• vLLM vs SGLang for multimodal serving
• Google StreamVibe proactive video agent benchmark

Sources

• LongBanana project, homepage, leaderboard, and dataset links, accessed November 9, 2026.
• Cao, H. et al., LongBanana: An Expert-Verified Benchmark for Long-Context Multi-Reference Image Synthesis, arXiv:2611.06856.