Roboflow Benchmark: GPT-5.6 Sol Is OpenAI's Best Vision Model — Gemini Still Wins

Roboflow ML growth engineer Piotr Skalski posted a vision-language-model (VLM) benchmark on July 16, 2026, comparing GPT-5.6 Sol, Terra, and Luna against Gemini 3.5 Flash and others on object detection, counting, OCR, and data extraction. It resurfaced on Hacker News and hit #1 again roughly 13 hours before this writing, with 303 points and 154 comments — Skalski answered questions in the thread throughout, and one of his replies changes the post's own conclusion.

The headline is real: GPT-5.6 Sol is OpenAI's best-ever vision model by a wide margin. The catch, visible in Roboflow's own numbers, is that "OpenAI's best" and "the best" are different claims — Gemini 3.5 Flash still beats Sol on the two tasks that matter most for real computer-vision work, at roughly a third of the cost.

TL;DR

The benchmark numbers, straight from Roboflow

Roboflow ran its upcoming VLM benchmark — covering object detection, counting, OCR, and text extraction — across OpenAI's GPT-5.6 family and competitors including Gemini 3.5 Flash.

Object detection (mAP@50):

Roboflow's framing: detection moved from "a major weakness to a practical capability" across OpenAI's whole GPT-5.6 lineup. That's a genuine leap, not a rounding error.

Object counting:

OCR (mean similarity):

Sol is flat-to-slightly-worse than GPT-5.5 on raw OCR — the improvement is concentrated in detection and counting, not text reading.

Text extraction (a distinct, harder task than full-page OCR: pulling one specific requested field out of a document rather than transcribing everything):

Sol actually regressed here versus GPT-5.5 — a detail easy to miss if you only read the detection headline.

Cost and speed:

Sol was the second most expensive model in Roboflow's whole test set, behind only Claude Fable 5. And on the two tasks buyers actually care about — detection and counting — Gemini 3.5 Flash beat it outright at less than a third of the price. HN's top-voted comment, from user HarHarVeryFunny, put it bluntly: Roboflow's own data shows Gemini winning nearly every metric at a third the cost, and called the post's "clear limits" framing of Sol understated.

Gemini 3.7 Flash already makes this stale — per Roboflow's own author

Here's the part that matters most if you're picking a model this week, and it isn't in the original post — it's in Skalski's own HN replies. Roboflow's benchmark was compiled about four weeks before this resurgence, and the VLM landscape has moved past it already:

"I wrote it about 4 weeks ago, and the VLM world is moving so fast that it's already kinda outdated. I think Gemini 3.7 Flash might be a better choice now, especially when you factor in the price... Gemini 3.7 Flash is now 50% off on OpenRouter."

That's a direct update from the person who ran the benchmark, not outside speculation — Gemini 3.7 Flash launched August 14, 2026 and already supersedes 3.5 Flash on Google's own charts, at half the price of the prior Flash tier. If Gemini 3.5 Flash was already beating Sol on cost-adjusted vision accuracy, 3.7 Flash widens that gap further.

Skalski also flagged open-weight options in the thread: Qwen3.8-Max is "totally insane" among Chinese/open models, though it's API-only through Alibaba Cloud, while Qwen3.8-27B is a genuinely good self-hostable pick if you need to keep inference in-house.

Two bugs that will silently tank your accuracy

Two practitioner-level details in Roboflow's writeup are worth more than the headline score, because they're the kind of thing that costs you accuracy points without any error message telling you why.

Coordinate format matters as much as the model. GPT-5.6 Sol expects absolute XYXY pixel coordinates for bounding boxes. Gemini 3.5 Flash expects normalized YXYX coordinates on a 0-1000 scale. Send Sol the wrong format and Roboflow measured roughly a 15 mAP-point drop — not because the model is worse, but because you fed it data in the format built for a different vendor's API. If you're switching between vision providers, check this before you touch anything else.

Large images destabilize Sol's bounding boxes. On images around 2000x2000 pixels or larger — especially at lower reasoning-effort settings — Sol starts producing misplaced or rotated bounding boxes. OpenAI confirmed this directly to Roboflow. The workaround is mundane but effective: resize or crop large images before sending them to the API. This is exactly the kind of gotcha that doesn't show up in a benchmark table but will show up in your production logs.

If you're building detection pipelines and want a deeper primer on how mAP, coordinate formats, and detection benchmarks actually work under the hood, Roboflow's own RF-DETR — their real-time detection transformer — is a useful companion read; it's the same team's purpose-built detector rather than a general VLM repurposed for the task.

Should you actually run a VLM in production for this?

A meaningful chunk of the HN thread wasn't about which model wins — it was about whether a general-purpose VLM is the right tool at all. Commenters mv4, iamniels, and bearjaws independently questioned using something like GPT-5.6 Sol for tasks like counting pills in an image, when OpenCV template matching or a small fine-tuned detector already solves that reliably and cheaply.

Skalski and others (lebek, throwup238) pushed back with the practical framing that actually matters: the real value of a frontier VLM here usually isn't running it in production at inference scale — it's using it to auto-generate or auto-label training data for a smaller, cheaper, specialized model that then runs in production. You pay the VLM's per-image cost once, during dataset creation, instead of on every inference call.

That framing also explains why a 73% counting accuracy score isn't automatically a production-ready number. Commenter rhplus pointed out that 70-80% accuracy is "terrible" for something like actual pharmacy pill counting — a benchmark score and a production-grade guarantee are not the same thing, especially in domains where an error has real consequences. If you need production-grade detection or counting at scale, the Roboflow thread's consensus is: use the VLM to label, then fine-tune or deploy a purpose-built detector like RF-DETR — don't run the general VLM as your inference engine.

How to read this benchmark honestly

The gap between "OpenAI's best-ever vision model" and "the best vision model" is exactly the kind of framing distinction worth training yourself to catch in vendor and third-party benchmark posts generally — see our guide to reading AI benchmarks without getting fooled for more on separating a real internal improvement from a competitive win. Roboflow's post is honest about the numbers; the resurfaced HN discussion is what forced the comparative framing — Gemini beats Sol — into the foreground where the original post's structure (about OpenAI's improvement) had left it more implicit.

Related reading

• RF-DETR: Roboflow's Real-Time Detection Transformer, Explained
• How to Read an AI Benchmark and Not Get Fooled
• AI Benchmarks in 2026: The Complete Guide
• Gemini 3.7 Flash vs Grok 4.6 vs Sonnet 5 vs GPT-5.6: The Real Numbers
• Gemini 3.7 Flash Pricing Leak
• GPT-5.6 Sol Now Runs All of ChatGPT
• Qwen3.8-Max Open Weights Are Live
• Qwen3.8-27B Is Live

Official source: Roboflow's VLM benchmark post and the Hacker News discussion thread.

Benchmark numbers, pricing, and model names reflect Roboflow's published data and the Hacker News discussion as of August 18, 2026. VLM pricing and rankings change fast — verify current numbers before making a production decision.