explainx / blog
Dylan Castillo tested 7 models across 48 animal-vehicle SVG combos to check for pelican-benchmark training. No significant effect found.
Jul 13, 2026
Public SWE-Bench scores won't protect your alpha. After Nadella's Reverse Information Paradox, explainx.ai maps how enterprises define "good," build private golden tasks, and hill-climb without leaking intelligence exhaust.
Jul 22, 2026
Fireworks AI's benchmark makes the case that picking one frontier model is already the wrong question — the real gains come from routing tasks to whichever model is cheapest for that specific job. Here's what the 1,030-task study actually measured, and why "oracle routing" isn't the same as a real router.
Jul 21, 2026
Google's July 21 Flash refresh promises 17% fewer output tokens and lower prices, but ships with zero frontier comparisons and a Gemini 3.5 Pro that's still not GA. Here's what the numbers, the model card, and developer reaction actually show.
TL;DR: Simon Willison's "SVG of a pelican riding a bicycle" prompt has become AI's most famous informal benchmark — and the obvious suspicion is that labs quietly train on it. Dylan Castillo tested that directly: 1,008 SVGs across 8 animals × 6 vehicles × 7 models, scored by an LLM judge, analyzed with a difficulty-adjusted fixed-effects regression. The result: no statistically significant pelican-specific or bicycle-specific boost at any lab. The more plausible explanation, per the study, is general SVG-generation improvement ("SVGmaxxing") — which several labs, including Google/DeepMind, have openly discussed pursuing.
Simon Willison has tested nearly every major LLM release with the same prompt for years: "Generate an SVG of a pelican riding a bicycle." Results frequently rank among the most-upvoted comments on Hacker News threads about new releases, and the benchmark's fame has raised an obvious question: with billions of dollars riding on perceived model quality, wouldn't a lab be tempted to specifically optimize for this one famous, highly visible prompt?
Willison himself has engaged with the skepticism directly, saying his "absolute dream situation" would be catching a lab demonstrably better at pelicans-on-bicycles than other animal-vehicle combinations — which would be "really funny" and would validate the concern.
Castillo built a small, rigorous experiment specifically to test the hypothesis, rather than eyeballing individual images the way most informal commentary does.
| Question | Answer |
|---|---|
| Grid size? | 8 animals × 6 vehicles = 48 prompts |
| Models tested? | 7, via OpenRouter |
| Samples per prompt? | 3, temperature 1.0 |
| Total SVGs generated? | 1,008 |
| Scoring method? | LLM judge (GPT-5.6 Luna), 1-5 on animal/vehicle/coherence |
| Statistical method? | Fixed-effects regression with robust standard errors |
| Pelican-specific effect found? | No — every lab's interval crosses zero |
| Bicycle-specific effect found? | One lab significant (Gemini 3.5 Flash), doesn't survive correction |
| Pelican-bicycle cell effect? | No significant effect for any lab |
| Total cost? | ~$80 in API credits |
The naive test — just ranking how good each lab's pelican-on-bicycle image looks — can't distinguish "labs trained specifically on this combination" from "pelicans and bicycles are just inherently harder or easier to draw than other animals and vehicles." Castillo's regression solves this by including animal × vehicle interaction terms that absorb each combination's baseline difficulty, then adding per-lab interaction terms specifically for pelican, bicycle, and the pelican-bicycle cell — measuring each lab's boost relative to the average lab, after controlling for how hard that particular animal or vehicle is to draw in general.
This matters because the raw rankings actually pointed the opposite direction from what pelicanmaxxing would predict: pooled across all seven models, pelicans ranked 6th of 8 animals (behind cat, whale, raccoon, heron, and antelope), and bicycles ranked 2nd-to-last of 6 vehicles. If labs were quietly training on the famous benchmark, you'd expect pelicans and bicycles to rank near the top, not the bottom half — though, as Castillo notes, that alone doesn't prove anything, since a pelican or bicycle could simply be harder to draw than a cat or skateboard regardless of any training effort.
The regression is what actually adjusts for that difficulty gap. Across all 21 statistical tests (pelican effect, bicycle effect, and combined-cell effect, for each of 7 labs), only one cleared conventional significance (p < 0.05): Gemini 3.5 Flash's bicycle effect. With 21 tests run at p < 0.05, chance alone predicts roughly one false positive (21 × 0.05 ≈ 1.05) — which is exactly what turned up. The result also fails a Bonferroni correction for multiple comparisons (threshold ≈ 0.002 across 21 tests; the observed p-value was 0.022).
Some prior critiques of the pelican benchmark have pointed to recurring compositional patterns — the pelican consistently facing the same direction, recurring props like a sun or scarf — as evidence of a memorized, templated response rather than genuine generation. Castillo tested this directly using an image feature-extraction pass.
Direction: All 21 pelican-bicycle images, across all seven labs, faced right — the only combination in the 48-cell grid with unanimous agreement. But 60% of all 1,008 images faced right overall, and several other combinations came close to unanimous too: antelope-on-scooter and pelican-on-scooter landed at 20 of 21, heron-on-bicycle at 19 of 21. Castillo's conclusion: pelicans and bicycles are both naturally easiest to draw from the side (hard to render either facing the viewer), so consistent right-facing composition reflects a shared drawing convention across side-on subjects generally, not evidence specific to the pelican-bicycle combination.
Recurring scene elements: Some combinations do show striking element consistency — every single flamingo-on-boat image includes a sun, otters-on-planes wear scarves 38% of the time, cats-on-bicycles get a basket 38% of the time. But pelican-on-bicycle didn't stand out from this general pattern of recurring elements that shows up across many combinations, not specifically that one.
Castillo is explicit that this experiment has a real blind spot. A lab that trained on SVG generation as a general capability — rather than any specific animal-vehicle pairing — would improve on every cell of the grid simultaneously, which is statistically indistinguishable from "the lab is just genuinely better at this." Google/DeepMind has openly discussed doing exactly this kind of general SVG optimization, which Willison himself has referenced separately.
This reframes the useful question: not "did a lab pelicanmaxx," but "did a lab SVGmaxx, and does that matter?" Several Hacker News commenters argued this reframing actually resolves the controversy in AI's favor — general SVG-generation skill is a legitimate, useful capability (icons, diagrams, charts, generative art), not a narrow benchmark trick. As one commenter put it: "If a model can improve at drawing 'X doing Y' and that prompt wasn't in the training set then it means it has improved its internal mapping from text-to-spatial-to-text." Whether that improvement generalizes to genuinely useful SVG output — versus just scoring well on animal-riding-vehicle prompts specifically — is a separate, harder question this study wasn't designed to answer.
Castillo lists several honestly in the original writeup:
The pelican benchmark is a useful microcosm of a much bigger problem in AI evaluation: any sufficiently famous benchmark creates an incentive to game it, and distinguishing "genuine capability improvement" from "narrow benchmark optimization" requires exactly this kind of controlled, difficulty-adjusted experimental design — not eyeballing a single output. Explainx.ai's own AI benchmarks complete guide covers this dynamic (often called benchmaxxing) across more consequential evals like SWE-bench and MMLU, where the stakes of gaming a benchmark are considerably higher than looking good at drawing pelicans.
The methodology here — a full factorial grid, an LLM judge, and a regression that isolates a specific effect from general capability — is a reusable template for testing benchmark-gaming suspicions on any narrow, famous eval, not just this one.
Primary sources: dylancastillo.co — full writeup and code · Simon Willison's pelican benchmark posts · GitHub repo with full pipeline
Study methodology, figures, and quotes reflect Dylan Castillo's July 18, 2026 writeup (updated July 22, 2026) and associated Hacker News discussion as of publication. The underlying code and data are open source — see the original post for reproduction instructions.
