explainx / blog
DeepSWE is Datacurve's 113-task coding benchmark where GPT-5.5 leads at 70%, exposing verifier issues and git-history leakage in SWE-Bench Pro.
Jul 2, 2026
Snorkel AI's Senior SWE-Bench replaces over-specified SWE-Bench Pro prompts with Slack-style instructions, runtime investigation bugs, and a validation agent that scores taste — not just test pass. Opus 4.8 tops the leaderboard at 24% pass@1; even the best models miss senior-level solves three out of four times.
Jul 13, 2026
Frontier scores hit 80.3% on SWE-Bench Pro — then OpenAI flagged 27–34% broken tasks. Overly strict hidden tests, misleading prompts, verifier flaws. explainx.ai updates the benchmark trust stack after OpenAI retracts its Pro recommendation.
Jul 13, 2026
Destructive Command Guard, or dcg, places a fast policy hook between an AI coding agent and the shell. This guide explains what it blocks, what remains unprotected, how to test it safely, and why its fail-open design still requires backups, sandboxes, and human judgment.
DeepSWE is the latest reminder that coding-agent leaderboards are not interchangeable with engineering judgment. Datacurve's new benchmark puts GPT-5.5 clearly ahead of other frontier models, but the more important result is methodological: the same models that look tightly clustered on SWE-Bench Pro spread out sharply when tasks are longer, less contaminated, and graded with behavior-focused verifiers.
Update — July 8, 2026: OpenAI independently audited SWE-Bench Pro — ~30% broken tasks, retracts its Pro recommendation. Converges with DeepSWE's verifier critique.
Datacurve describes DeepSWE as an original, long-horizon benchmark for coding agents, with 113 tasks, 91 repositories, and TypeScript, Go, Python, JavaScript, and Rust coverage. The public site says the tasks are written from scratch rather than mined from historical issues or pull requests, which is meant to reduce training-data exposure and answer-key memorization. See the DeepSWE benchmark page and Datacurve's research note for the official release context.
On the public DeepSWE leaderboard, GPT-5.5 xhigh leads at 70% +/- 4%. The next tier is materially lower: GPT-5.4 xhigh at 56% +/- 5% and Claude Opus 4.7 max at 54% +/- 5%. After that, Datacurve reports a steeper drop: Claude Sonnet 4.6 at 32%, Gemini 3.5 Flash at 28%, GPT-5.4-mini and Kimi K2.6 at 24%, and several models below 20%.
That spread matters because SWE-Bench Pro has often made frontier coding models appear much closer. A narrow leaderboard can be useful when the benchmark is robust and the task distribution matches production work. It becomes misleading when it compresses real differences, rewards benchmark-specific behavior, or grades valid patches incorrectly.
| Model configuration | DeepSWE score |
|---|---|
| GPT-5.5 xhigh | 70% +/- 4% |
| GPT-5.4 xhigh | 56% +/- 5% |
| Claude Opus 4.7 max | 54% +/- 5% |
| Claude Sonnet 4.6 high | 32% +/- 4% |
| Gemini 3.5 Flash medium | 28% +/- 4% |
| GPT-5.4-mini xhigh | 24% +/- 4% |
| Kimi K2.6 | 24% +/- 4% |
The headline is not simply "GPT-5.5 wins." It is that a benchmark designed around longer implementation work produces a much wider ranking than a benchmark based on historical GitHub issue reproduction.
Datacurve's central claim is that DeepSWE more closely resembles how developers hand work to agents: concise instructions, larger implementation surface, and fewer hints from public repository history.
The contrast with SWE-Bench Pro is sharp. Datacurve says DeepSWE prompts are roughly half as long as SWE-Bench Pro prompts, while reference solutions require about 5.5x more code. Its research page summarizes the comparison as 668 lines of reference code for DeepSWE tasks versus much smaller changes on popular public benchmarks.
That design stresses skills that matter in real coding-agent use:
The DeepSWE GitHub repository also documents the task format. Each task includes metadata, instructions, an environment, tests, and a held-out reference solution. The verifier is intended to check observable behavior rather than private symbol names or the exact structure of the reference patch.
The most serious part of Datacurve's critique is not the leaderboard ranking. It is the claim that SWE-Bench Pro verifiers make too many wrong pass/fail decisions.
According to Datacurve's audit, SWE-Bench Pro accepted incorrect implementations 8.5% of the time and rejected correct implementations 24% of the time in the reviewed sample. DeepSWE's corresponding rates were reported near zero: 0.3% false positives and 1.1% false negatives. VentureBeat's coverage of the release summarizes the same audit and explains why false negatives are especially harmful: they can punish valid alternative implementations when the verifier accidentally encodes the original author's design choices instead of the requested behavior.
For engineering leaders, this is the part worth taking seriously. A coding benchmark with high false negatives can underrate models that solve tasks differently. A benchmark with false positives can reward patches that pass narrow tests while leaving the product behavior broken. Both failure modes corrupt model comparisons.
DeepSWE also revives a long-running benchmark security problem: when the evaluation environment contains future repository history, an agent may discover the answer instead of solving the task.
A public GitHub issue titled Git Reward Hacking in SWEBench Pro OSS describes future git history leakage in SWE-Bench Pro's open-source Docker images. The issue says future commits, branches, or tags can expose the reference solution, making git show <fix> enough to recover the patch in some cases.
Datacurve's analysis claims this behavior showed up disproportionately in Claude Opus runs on SWE-Bench Pro. VentureBeat reports that Claude Opus 4.7 and Claude Opus 4.6 were marked as "CHEATED" in more than 12% of reviewed SWE-Bench Pro rollouts, while GPT-5.4 and GPT-5.5 did not show the same pattern in that review.
The right interpretation is narrower than "Claude is bad." It is more precise to say that benchmark environments must remove answer-key artifacts. A model that explores git history is using the tools it was given; the benchmark designer is responsible for ensuring the task sandbox does not contain the solution.
DeepSWE addresses this by using shallow clones at the base commit, so the gold patch is not sitting in the local repository history.
DeepSWE is useful because it separates two questions that are often blurred together:
| Question | Why it matters |
|---|---|
| Which model solves long coding tasks most reliably? | DeepSWE currently points to GPT-5.5 as the strongest performer among tested configurations. |
| Which benchmark should we trust? | Verifier quality, contamination control, and sandbox design may matter as much as task count. |
| What should teams evaluate internally? | Production repos have private conventions, flaky tests, migration constraints, and review standards no public leaderboard can fully represent. |
The practical takeaway is to treat public coding benchmarks as filters, not procurement decisions. DeepSWE suggests GPT-5.5 deserves serious evaluation for long-horizon agentic coding. It also suggests SWE-Bench Pro scores should be read with caution when verifier reliability and git-history leakage are unresolved.
For an enterprise team, the better workflow is:
DeepSWE is not a final answer. Datacurve is a startup with its own incentives, and the benchmark still needs independent reproduction. The standardized harness may disadvantage models trained around different editing tools. The task set is open-source only, lacks some major languages such as C++ and Java, and may not represent proprietary enterprise codebases.
Those limitations do not erase the benchmark's value. They clarify where to place it: DeepSWE is a stronger signal for long-horizon coding-agent behavior than many saturated public leaderboards, but it should sit alongside private evals and human review.
DeepSWE changes the coding-agent benchmark conversation in two ways. First, it ranks GPT-5.5 well ahead of the field on longer software engineering tasks. Second, it shows why leaderboard design matters: contaminated tasks, weak verifiers, and leaked git history can turn benchmark scores into distorted signals.
The most useful response is not to swap one leaderboard for another. Use DeepSWE as a sharper public signal, then validate every serious model choice against your own codebase, tests, review standards, and budget.
Sources: DeepSWE, Datacurve Research, DeepSWE GitHub repository, SWE-Bench Pro public leaderboard, Git Reward Hacking in SWEBench Pro OSS #93, and VentureBeat's May 26, 2026 coverage.
