OpenAI July 8, 2026: agent + human audit finds ~30% of SWE-Bench Pro tasks broken — overly strict tests, underspecified prompts, low coverage. Retracts prior Pro endorsement. explainx.ai maps what builders should eval instead.
OpenAI told everyone to use SWE-Bench Pro. Then it audited the benchmark and walked that back.
On July 8, 2026, OpenAI published Separating signal from noise in coding evaluations — a datapoint-by-datapoint audit of SWE-Bench Pro's 731-task public split. Verdict: ~30% of tasks are broken. Frontier pass rates climbed from 23.3% to 80.3% in eight months — but a third of the yardstick may not measure what it claims.
For teams using GPT-5.6 Sol vs Fable 5 leaderboard numbers in procurement decks, this is the paper that says stop treating SWE-Bench Pro as ground truth.
OpenAI audit: ~30% broken → retracts Pro endorsement
explainx.ai read: The benchmark stack is correcting in public — first Verified, now Pro. Leaderboard marketing did not wait for the audit.
What OpenAI found — issue taxonomy
Share of 731 public tasks flagged by issue type (agent pipeline + human campaign):
Issue
Agent pipeline
Human reviewers (most common label)
Overly strict tests¹
14.4%
High
Underspecified prompts²
7.5%
High
Low-coverage tests
4.1%
9.4% (top human pick)
Misleading prompt
1.9%
Present
Miscellaneous
6.3%
Overlapping labels common
¹ Previously called narrow tests. ² Previously wide tests.
Four failure modes (plain language)
Mode
What goes wrong
Effect on scores
Overly strict tests
Hidden tests require specific implementation details not in prompt
False negatives — correct solutions fail
Underspecified prompts
Prompt omits requirements tests enforce
Models guess wrong architecture
Low-coverage tests
Tests don't exercise requested feature fully
False positives — incomplete fixes pass
Misleading prompt
Prompt points wrong direction vs tests
Wastes agent tokens on dead ends
Humans flagged more broken tasks than agents (34.1% vs 27.4%) and often assigned multiple labels — conservative agent pipeline, broader human judgment.
" | Chapter 1 | 1" ← one leading space
"** | Chapter 1 | 1"
" | Just title | "
Hidden tests require:
snippet
" | Chapter 1 | 1" ← TWO leading spaces
"** | Chapter 1 | 1"
" | Just title | "
A model that follows the prompt fails. A model that guesses the hidden spacing convention passes. That is not software engineering capability — it is benchmark archaeology.
Why this matters for 2026: OpenAI explicitly argues improved models make large-scale benchmark QA practical — inspect prompts, tests, patches, and edge cases at depth previously too costly.
Same week Tibo shipped Codex quota fixes — the tooling to run investigator-style audits is becoming product, not research-only.
Overlap: 74% category agreement when agents flagged issues; humans more often saw compound failures.
Why OSS pull requests make bad eval shards
OpenAI's structural diagnosis:
Issues and pull requests from open-source repositories were originally created for human collaboration… problem descriptions, merged code, and unit tests do not always line up.
PR tests are written to validate a specific patch, not define implementation-agnostic success. Mining them into benchmarks imports:
Maintainer-specific formatting quirks
Hidden assumptions from thread context
Tests that encode author's diff, not user intent
DeepSWE attacked the same root cause from a startup benchmark — original tasks, behavior-focused verifiers, shallow git state.
If a third of tasks are broken, ranking models by single Pro number is Goodhart bait — optimize for hidden test quirks, not shipped code.
Agnes 2.5 Pro internal chart cited 61.8% Pro vs Opus 69.2% (Singapore post) — discount all vendor internal Pro numbers until independent QA reruns on cleaned splits.
Examine Pro results carefully — do not treat as deployment-grade signal alone
Build benchmarks with experienced developers for evaluation — not scraped PR archaeology
Use agents for scalable data quality checks — QA pipelines as standard practice
Prioritize evals that inform Preparedness decisions — valid, informative, hard to game
OpenAI's newer direction: domain-specific research evals like GeneBench-Pro and LifeSciBench — judgment-heavy, constructed problems — vs repo-mined coding shards.
What engineering teams should do
Stop
Basing model procurement only on SWE-Bench Pro pass rate
Quoting 80.3% vs 69.2% without benchmark-caveat slides
Assuming Verified → Pro migration fixed eval trust (both flawed, different ways)
A task is valid only if:
- Prompt fully specifies behavior tests check
- Tests fail known-bad implementations
- Tests pass ≥2 independent correct implementations
- Gold patch not reachable via git history in container
Converging evidence — DeepSWE + OpenAI
Source
Finding
Date
Datacurve DeepSWE
8.5% false pass · 24% false fail · git leakage
May 2026
OpenAI audit
~30% broken tasks · retracts Pro push
Jul 8, 2026
OpenAI Verified
Contamination + design flaws · deprecated
Earlier 2026
Two independent paths, same conclusion: SWE-Bench Pro scores are useful for hype, risky for decisions.
Audit statistics reflect OpenAI's July 8, 2026 publication. Benchmark curators may patch tasks — re-check issue trackers before citing pass rates in production decisions.
