OpenAI told everyone to use SWE-Bench Pro. Then it audited the benchmark and walked that back.
On July 8, 2026, OpenAI published Separating signal from noise in coding evaluations — a datapoint-by-datapoint audit of SWE-Bench Pro's 731-task public split. Verdict: ~30% of tasks are broken. Frontier pass rates climbed from 23.3% to 80.3% in eight months — but a third of the yardstick may not measure what it claims.
For teams using GPT-5.6 Sol vs Fable 5 leaderboard numbers in procurement decks, this is the paper that says stop treating SWE-Bench Pro as ground truth.
TL;DR — what changed July 8
| Question | Answer |
|---|---|
| What broke? | Tests, prompts, coverage — not just "hard tasks" |
| How much? | ~30% broken (agents: 27.4% · humans: 34.1%) |
| OpenAI stance? | Retracts prior "switch to Pro" recommendation |
| Method? | Automated pipeline + Codex investigator agents + 5× human reviewers per flagged task |
| Why now? | Eval flaws distort Preparedness Framework deployment decisions |
| What instead? | Purpose-built benchmarks · private repo evals · GeneBench-Pro class domain suites |
Timeline — Verified → Pro → audit
| Date | Event |
|---|---|
| Earlier 2026 | OpenAI finds SWE-bench Verified has design + contamination issues — "no longer meaningful signal" |
| Community shift | OpenAI recommended SWE-Bench Pro — longer horizons, harder agentic tasks |
| May 2026 | Datacurve DeepSWE reports Pro verifier errors + git-history leakage |
| Jul 8, 2026 | OpenAI audit: ~30% broken → retracts Pro endorsement |
explainx.ai read: The benchmark stack is correcting in public — first Verified, now Pro. Leaderboard marketing did not wait for the audit.
What OpenAI found — issue taxonomy
Share of 731 public tasks flagged by issue type (agent pipeline + human campaign):
| Issue | Agent pipeline | Human reviewers (most common label) |
|---|---|---|
| Overly strict tests¹ | 14.4% | High |
| Underspecified prompts² | 7.5% | High |
| Low-coverage tests | 4.1% | 9.4% (top human pick) |
| Misleading prompt | 1.9% | Present |
| Miscellaneous | 6.3% | Overlapping labels common |
¹ Previously called narrow tests. ² Previously wide tests.
Four failure modes (plain language)
| Mode | What goes wrong | Effect on scores |
|---|---|---|
| Overly strict tests | Hidden tests require specific implementation details not in prompt | False negatives — correct solutions fail |
| Underspecified prompts | Prompt omits requirements tests enforce | Models guess wrong architecture |
| Low-coverage tests | Tests don't exercise requested feature fully | False positives — incomplete fixes pass |
| Misleading prompt | Prompt points wrong direction vs tests | Wastes agent tokens on dead ends |
Humans flagged more broken tasks than agents (34.1% vs 27.4%) and often assigned multiple labels — conservative agent pipeline, broader human judgment.
Case study — OpenLibrary-77c16d5
OpenAI's concrete example: table-of-contents Markdown serialization.
Prompt tells the model:
" | Chapter 1 | 1" ← one leading space
"** | Chapter 1 | 1"
" | Just title | "
Hidden tests require:
" | Chapter 1 | 1" ← TWO leading spaces
"** | Chapter 1 | 1"
" | Just title | "
A model that follows the prompt fails. A model that guesses the hidden spacing convention passes. That is not software engineering capability — it is benchmark archaeology.
Methodology — agents auditing agents
SWE-Bench Pro QA pipeline (OpenAI Jul 2026)
├── Automated filter → 286 potentially broken / 731
├── Codex investigator agents (repo + tests + traces)
│ └── Multiple independent audit passes → human final judgment
└── Human campaign: 5 engineers × flagged tasks
└── Independent label → escalate disagreements
Why this matters for 2026: OpenAI explicitly argues improved models make large-scale benchmark QA practical — inspect prompts, tests, patches, and edge cases at depth previously too costly.
Same week Tibo shipped Codex quota fixes — the tooling to run investigator-style audits is becoming product, not research-only.
Overlap: 74% category agreement when agents flagged issues; humans more often saw compound failures.
Why OSS pull requests make bad eval shards
OpenAI's structural diagnosis:
Issues and pull requests from open-source repositories were originally created for human collaboration… problem descriptions, merged code, and unit tests do not always line up.
PR tests are written to validate a specific patch, not define implementation-agnostic success. Mining them into benchmarks imports:
- Maintainer-specific formatting quirks
- Hidden assumptions from thread context
- Tests that encode author's diff, not user intent
DeepSWE attacked the same root cause from a startup benchmark — original tasks, behavior-focused verifiers, shallow git state.
What the 80.3% headline really means
| Stat | Context |
|---|---|
| 23.3% → 80.3% in 8 months | Public 731-task split |
| Fable 5 80.3% | Cited in frontier comparisons |
| ~30% broken tasks | Pass rate mixes real skill + verifier luck |
If a third of tasks are broken, ranking models by single Pro number is Goodhart bait — optimize for hidden test quirks, not shipped code.
Agnes 2.5 Pro internal chart cited 61.8% Pro vs Opus 69.2% (Singapore post) — discount all vendor internal Pro numbers until independent QA reruns on cleaned splits.
What OpenAI recommends now
From the July 8 post:
- Examine Pro results carefully — do not treat as deployment-grade signal alone
- Build benchmarks with experienced developers for evaluation — not scraped PR archaeology
- Use agents for scalable data quality checks — QA pipelines as standard practice
- Prioritize evals that inform Preparedness decisions — valid, informative, hard to game
OpenAI's newer direction: domain-specific research evals like GeneBench-Pro and LifeSciBench — judgment-heavy, constructed problems — vs repo-mined coding shards.
What engineering teams should do
Stop
- Basing model procurement only on SWE-Bench Pro pass rate
- Quoting 80.3% vs 69.2% without benchmark-caveat slides
- Assuming Verified → Pro migration fixed eval trust (both flawed, different ways)
Start
- Private eval harness on your repos — enterprise benchmark guide
- Multi-benchmark panel — Terminal-Bench, DeepSWE, internal tickets
- Grade trajectories — reward hacking, git log peeking, test-suite gaming
- Track accepted patches — review burden, not pass rate alone
- Agent QA on your own eval set — copy OpenAI's investigator-agent pattern
- Re-read AI benchmarks guide — SWE-bench section now post-audit
Copy-paste internal eval rule
A task is valid only if:
- Prompt fully specifies behavior tests check
- Tests fail known-bad implementations
- Tests pass ≥2 independent correct implementations
- Gold patch not reachable via git history in container
Converging evidence — DeepSWE + OpenAI
| Source | Finding | Date |
|---|---|---|
| Datacurve DeepSWE | 8.5% false pass · 24% false fail · git leakage | May 2026 |
| OpenAI audit | ~30% broken tasks · retracts Pro push | Jul 8, 2026 |
| OpenAI Verified | Contamination + design flaws · deprecated | Earlier 2026 |
Two independent paths, same conclusion: SWE-Bench Pro scores are useful for hype, risky for decisions.
Impact on July 2026 frontier narrative
| Story | SWE-Bench Pro dependency |
|---|---|
| Fable 5 80.3% marketing | Needs ~30% discount caveat |
| GPT-5.6 no published Pro score | Possibly prudent omission |
| Grok 4.5 64.7% | Third-party harness — same benchmark flaws |
| Agnes 2.5 61.8% | Internal eval — unverified + broken-task noise |
| Nadella enterprise benchmarks | Private eval thesis validated |
Honest limitations
| Caveat | Detail |
|---|---|
| OpenAI conflict | Ships models ranked on these benchmarks |
| Public split only | 731 tasks — private Pro split unaudited here |
| No cleaned re-score | OpenAI didn't publish adjusted model rankings post-audit |
| Agent reviewers | Codex investigating Codex-competitive benchmark — meta bias |
| Broken ≠ easy | Some hard tasks are fair — audit is prevalence, not blanket invalidation |
What to watch next
| Signal | Why |
|---|---|
| SWE-Bench Pro v2 | ScaleAI/curator response to OpenAI retraction |
| Adjusted leaderboard | Re-score frontier models on cleaned 70% |
| OpenAI deployment memo | Does audit change GPT-5.6 safety case? |
| Anthropic response | Fable 80.3% marketing adjustment |
| Enterprise eval templates | Investigator-agent QA goes mainstream |
Related on explainx.ai
- DeepSWE — prior SWE-Bench Pro scrutiny (Datacurve)
- AI benchmarks complete guide — SWE-bench section
- How to build enterprise AI benchmarks — Nadella derivative
- GPT-5.6 vs Fable 5 — Pro scores in context
- GeneBench-Pro — OpenAI's newer eval direction
- Specification gaming & Goodhart
- Terminal-Bench 2.0 guide
- Agnes 2.5 Pro — internal Pro claims
- Goodhart's Law Comes for Every Benchmark You Trust — the 2026 contamination evidence
Sources: OpenAI — Separating signal from noise, July 8, 2026 · SWE-Bench Pro · DeepSWE / Datacurve
Audit statistics reflect OpenAI's July 8, 2026 publication. Benchmark curators may patch tasks — re-check issue trackers before citing pass rates in production decisions.