Are AI Labs "Pelicanmaxxing"? A 1,008-SVG Study Says Probably Not

TL;DR: Simon Willison's "SVG of a pelican riding a bicycle" prompt has become AI's most famous informal benchmark — and the obvious suspicion is that labs quietly train on it. Dylan Castillo tested that directly: 1,008 SVGs across 8 animals × 6 vehicles × 7 models, scored by an LLM judge, analyzed with a difficulty-adjusted fixed-effects regression. The result: no statistically significant pelican-specific or bicycle-specific boost at any lab. The more plausible explanation, per the study, is general SVG-generation improvement ("SVGmaxxing") — which several labs, including Google/DeepMind, have openly discussed pursuing.

Why this study exists

Simon Willison has tested nearly every major LLM release with the same prompt for years: "Generate an SVG of a pelican riding a bicycle." Results frequently rank among the most-upvoted comments on Hacker News threads about new releases, and the benchmark's fame has raised an obvious question: with billions of dollars riding on perceived model quality, wouldn't a lab be tempted to specifically optimize for this one famous, highly visible prompt?

Willison himself has engaged with the skepticism directly, saying his "absolute dream situation" would be catching a lab demonstrably better at pelicans-on-bicycles than other animal-vehicle combinations — which would be "really funny" and would validate the concern.

Castillo built a small, rigorous experiment specifically to test the hypothesis, rather than eyeballing individual images the way most informal commentary does.

TL;DR — the study design and result

How the difficulty adjustment works

The naive test — just ranking how good each lab's pelican-on-bicycle image looks — can't distinguish "labs trained specifically on this combination" from "pelicans and bicycles are just inherently harder or easier to draw than other animals and vehicles." Castillo's regression solves this by including animal × vehicle interaction terms that absorb each combination's baseline difficulty, then adding per-lab interaction terms specifically for pelican, bicycle, and the pelican-bicycle cell — measuring each lab's boost relative to the average lab, after controlling for how hard that particular animal or vehicle is to draw in general.

This matters because the raw rankings actually pointed the opposite direction from what pelicanmaxxing would predict: pooled across all seven models, pelicans ranked 6th of 8 animals (behind cat, whale, raccoon, heron, and antelope), and bicycles ranked 2nd-to-last of 6 vehicles. If labs were quietly training on the famous benchmark, you'd expect pelicans and bicycles to rank near the top, not the bottom half — though, as Castillo notes, that alone doesn't prove anything, since a pelican or bicycle could simply be harder to draw than a cat or skateboard regardless of any training effort.

The regression is what actually adjusts for that difficulty gap. Across all 21 statistical tests (pelican effect, bicycle effect, and combined-cell effect, for each of 7 labs), only one cleared conventional significance (p < 0.05): Gemini 3.5 Flash's bicycle effect. With 21 tests run at p < 0.05, chance alone predicts roughly one false positive (21 × 0.05 ≈ 1.05) — which is exactly what turned up. The result also fails a Bonferroni correction for multiple comparisons (threshold ≈ 0.002 across 21 tests; the observed p-value was 0.022).

The direction evidence — pelicans all face right, but so does everything else

Some prior critiques of the pelican benchmark have pointed to recurring compositional patterns — the pelican consistently facing the same direction, recurring props like a sun or scarf — as evidence of a memorized, templated response rather than genuine generation. Castillo tested this directly using an image feature-extraction pass.

Direction: All 21 pelican-bicycle images, across all seven labs, faced right — the only combination in the 48-cell grid with unanimous agreement. But 60% of all 1,008 images faced right overall, and several other combinations came close to unanimous too: antelope-on-scooter and pelican-on-scooter landed at 20 of 21, heron-on-bicycle at 19 of 21. Castillo's conclusion: pelicans and bicycles are both naturally easiest to draw from the side (hard to render either facing the viewer), so consistent right-facing composition reflects a shared drawing convention across side-on subjects generally, not evidence specific to the pelican-bicycle combination.

Recurring scene elements: Some combinations do show striking element consistency — every single flamingo-on-boat image includes a sun, otters-on-planes wear scarves 38% of the time, cats-on-bicycles get a basket 38% of the time. But pelican-on-bicycle didn't stand out from this general pattern of recurring elements that shows up across many combinations, not specifically that one.

What the study can't rule out: SVGmaxxing

Castillo is explicit that this experiment has a real blind spot. A lab that trained on SVG generation as a general capability — rather than any specific animal-vehicle pairing — would improve on every cell of the grid simultaneously, which is statistically indistinguishable from "the lab is just genuinely better at this." Google/DeepMind has openly discussed doing exactly this kind of general SVG optimization, which Willison himself has referenced separately.

This reframes the useful question: not "did a lab pelicanmaxx," but "did a lab SVGmaxx, and does that matter?" Several Hacker News commenters argued this reframing actually resolves the controversy in AI's favor — general SVG-generation skill is a legitimate, useful capability (icons, diagrams, charts, generative art), not a narrow benchmark trick. As one commenter put it: "If a model can improve at drawing 'X doing Y' and that prompt wasn't in the training set then it means it has improved its internal mapping from text-to-spatial-to-text." Whether that improvement generalizes to genuinely useful SVG output — versus just scoring well on animal-riding-vehicle prompts specifically — is a separate, harder question this study wasn't designed to answer.

Limitations worth flagging

Castillo lists several honestly in the original writeup:

• Single LLM judge (GPT-5.6 Luna), with no cross-validation against other judges or human raters, and no measured self-consistency on repeat scoring.
• Judge-model family overlap — the judge is from the same family as one contestant (GPT-5.6 Terra) — though since every lab is scored on the same 48-cell grid, a judge that happens to favor one lab's style lifts that lab's entire grid uniformly, which shouldn't bias the within-lab difficulty-adjusted comparisons this study actually relies on.
• Small budget (~$80) capped the study at 3 samples per cell and a single judge pass — wider confidence intervals (roughly ±0.6 judge points) mean smaller true effects than that could exist undetected.
• SVGmaxxing is fundamentally undetectable by this study's design, as discussed above.

Why this matters beyond one silly benchmark

The pelican benchmark is a useful microcosm of a much bigger problem in AI evaluation: any sufficiently famous benchmark creates an incentive to game it, and distinguishing "genuine capability improvement" from "narrow benchmark optimization" requires exactly this kind of controlled, difficulty-adjusted experimental design — not eyeballing a single output. Explainx.ai's own AI benchmarks complete guide covers this dynamic (often called benchmaxxing) across more consequential evals like SWE-bench and MMLU, where the stakes of gaming a benchmark are considerably higher than looking good at drawing pelicans.

The methodology here — a full factorial grid, an LLM judge, and a regression that isolates a specific effect from general capability — is a reusable template for testing benchmark-gaming suspicions on any narrow, famous eval, not just this one.

Related on explainx.ai

• AI benchmarks — complete guide
• Kimi K3, and what we can still learn from the pelican benchmark
• Cursor reward hacking and SWE-bench eval contamination
• Senior SWE-bench — Snorkel AI coding agents benchmark

Primary sources: dylancastillo.co — full writeup and code · Simon Willison's pelican benchmark posts · GitHub repo with full pipeline

Study methodology, figures, and quotes reflect Dylan Castillo's July 18, 2026 writeup (updated July 22, 2026) and associated Hacker News discussion as of publication. The underlying code and data are open source — see the original post for reproduction instructions.