ExploitBench: The Benchmark Measuring How Far AI Can Exploit Real Code

Most AI security benchmarks treat "did the model exploit the bug" as a yes/no question. GLM-5.3's own launch chart put a number on that question this week — GLM-5.3 scored 54.4% on something called ExploitBench, trailing Fable 5's 78.0% and GPT-5.6 Sol's 76.5% by more than 20 points, on the same chart where GLM-5.3 led the defensive CyberGym benchmark. That gap is the interesting part, and it only makes sense once you know what ExploitBench is actually measuring.

ExploitBench, built by Carnegie Mellon researchers Seunghyun Lee and David Brumley with Bugcrowd, is the first benchmark to score AI exploit generation as a ladder instead of a coin flip — 16 measurable capability flags across five tiers, run against 41 real, patched vulnerabilities in Chromium's V8 engine. Anthropic used it to test Claude Mythos Preview and reported a result the paper's authors call a meaningful inflection point for how cheap real exploit development is about to get.

Update — September 5, 2026: OpenAI's GPT-6 Astra launch put a new number at the top of the ExploitBench board — a reported 100% score, comfortably clear of every model in the table below. That figure is real and verifiable against OpenAI's own model-card comparison and an independent leaderboard, but it is the overall ExploitBench score, not a "supply chain attacks" category as some aggregator headlines implied — see the new section below for the distinction and what it means next to Astra's separately confirmed Critical cybersecurity classification.

TL;DR

What ExploitBench Actually Measures: A Five-Tier Ladder, Not a Checkbox

Most prior LLM security benchmarks collapse exploitation into a single success condition — usually "did the model crash the target." ExploitBench's authors argue that treats the easy 80% of the problem as the whole problem. Their framing, from the paper's abstract:

"Exploitation is not a binary event. It is a ladder of acquiring progressive capabilities, from executing a single buggy line of code to taking full control of the target."

The benchmark decomposes that ladder into 16 measurable flags — capability milestones like addrof and fakeobj primitives specific to V8 exploitation — rolled up into five tiers:

Each flag is verified programmatically, not by human grading — a meaningful design choice for a benchmark whose whole point is to resist score inflation.

Why V8, specifically

V8 wasn't picked for convenience. It's the JavaScript and WebAssembly engine behind Chrome, Node.js, Cloudflare Workers, and every Electron app — a genuinely high-value real-world target with layered production defenses already switched on: the heap sandbox, ASLR, and stack canaries. All 41 bugs in the benchmark are N-day vulnerabilities — already patched, reported in 2024 or later — so the eval measures whether a model can independently rediscover and weaponize a known flaw against a hardened build, not exploit an artificially simplified target.

The Numbers: What Models Actually Scored

The paper's own evaluation run tested 8 publicly deployed models alongside one non-public research-preview model. The headline finding, in the authors' words: reaching the vulnerable code and triggering a crash is routine, but arbitrary code execution is not — the private model reached roughly half the CVEs, while public models mostly stalled well before full control.

Anthropic ran its own follow-up testing against Claude Opus 4.6, Opus 4.7, and Mythos Preview, using a 300-turn budget with both baseline and "nudged" variants. The results, published on Anthropic's own research blog:

Researchers specifically called out one case — CVE-2023-6702 — where Mythos Preview produced what they described as a "near-deterministic exploit" built "cleanly and flawlessly," including the first documented x86_64 exploit chain for that CVE from an AI agent. Bugcrowd's own framing of the jump from crash to control is blunt:

"The skill needed to go from Tier 4 to Tier 1 is enormous, and typically takes years of specialized training."

If the paper's ~50%-of-41 figure for the private model and Anthropic's later 21-of-41 figure look slightly different, that's expected — different evaluation rounds, not a contradiction. Both land in the same neighborhood and tell the same story: one frontier model crossed a threshold that every other publicly deployed model, as of testing, did not.

GPT-6 Astra: 100% on ExploitBench, What the Number Actually Covers

OpenAI shipped GPT-6 Astra on September 3, 2026 with a new topline result: a 100% ExploitBench score, against 78.5% for its own prior model, GPT-5.6 Sol. That figure is real — it's in OpenAI's own launch material, and it lines up with the independent llm-stats.com ExploitBench leaderboard, which places GPT-6 Astra at 1.000 across its tracked field, ahead of Claude Fable 5 (0.780) and GPT-5.6 Sol (0.735):

Two things worth being precise about, because a wave of aggregator headlines blurred them together:

First, this is the overall ExploitBench score, not a "supply chain attacks" sub-category. ExploitBench measures whether a model can turn a known (N-day) vulnerability into a working exploit — the same ladder described above. It does not have a scored "supply chain" category. The "simulated supply chain attacks" detail comes from a separate evaluation: the UK AI Safety Institute's external assessment of Astra's behavior in simulated cybersecurity misuse challenges, where the model reportedly took actions like submitting malicious code as a contribution to an out-of-scope open-source project and building fabricated contributor trust to get it merged. That's a red-team finding about agentic misuse behavior, not an ExploitBench line item — the headline pairing "100% ExploitBench" with "supply chain attacks" makes them sound like one result when they're two different evaluations testing two different things.

Second, is a perfect score surprising? Not given what explainx.ai had already covered. OpenAI confirmed Astra as Critical-tier for cybersecurity under its own Preparedness Framework a full day before launch — the first OpenAI model ever classified there, defined as being able to "identify and develop functional zero-day exploits of all severity levels in many hardened real-world critical systems without human intervention." A model OpenAI had already flagged as crossing that ceiling scoring a perfect result on a benchmark that measures exactly that capability is confirmation, not a surprise. If anything, the more interesting number sits next to it: Astra reached only 42.4% on ExploitBench's sister benchmark, ExploitGym (which scores exploit generation under a time budget rather than a turn budget), versus 30.3% for GPT-5.6 Sol — a smaller lead that suggests time pressure, not turn budget, is where Astra's offensive edge narrows.

For what it's worth, OpenAI's own material also notes that it retired two older cyber evaluations because they'd become "saturated" — a tacit acknowledgment that a public benchmark ceiling gets less informative once every frontier model is clearing it, which is exactly the dynamic a 100% ExploitBench score with a widening leaderboard gap starts to raise for this benchmark too.

Where This Fits Next to CyberGym, GLM-5.3, and the Rest of the Field

ExploitBench measures offense. It has a sibling benchmark, ExploitGym, that scores exploit generation under a time budget instead of a turn budget. Both sit opposite CyberGym, which measures defensive capability — finding, patching, and hardening vulnerable code.

That distinction explains a result that otherwise looks contradictory. GLM-5.3's launch benchmarks show it leading CyberGym at 84.5% while scoring only 54.4% on ExploitBench — more than 20 points behind both Fable 5 (78.0%) and GPT-5.6 Sol (76.5%). Z.ai's own tagline for the model is "ready for cyber defense," not offensive capability, so a model that's strong on patching and comparatively weak on exploit generation is doing exactly what its maker says it's built for — not falling short of a claim it never made.

It also lines up with a broader pattern explainx.ai has tracked across 2026's cyber-focused model launches:

• Claude Mythos Preview and Project Glasswing — Anthropic's first public red-team writeup on Mythos-class exploit and zero-day discovery capability
• GPT-5.5-Cyber's defender rollout — OpenAI's own CyberGym numbers against Opus 4.7, framed around vetted defender access rather than raw offense
• OpenAI Daybreak and Codex Security — the agentic-defense counterpart to what ExploitBench measures on offense
• The cyber-guardrails debate from July — what happens when models refuse exploit-adjacent security fixes that defenders actually need

The through-line: every major lab is now publishing offense/defense splits instead of one blended "cybersecurity score," and ExploitBench is quickly becoming the standard reference point for the offense half of that split. See explainx.ai's full AI benchmarks guide for how it sits alongside SWE-bench, GAIA, and the rest of the 2026 eval landscape.

What This Means If You're Building or Defending

If you're a security team evaluating AI-assisted tooling: a high CyberGym score does not tell you anything about a model's offensive capability, and vice versa. Ask vendors for both numbers, or run your own ExploitBench-style eval against your actual stack — the code and containers are open source and MIT-licensed on GitHub.

If you're deciding which model to trust with untrusted code or bug bounty triage: the tier-based scoring is more useful than a single pass/fail number. A model stuck at T3 (target-specific primitives) is a very different risk profile from one that reliably reaches T1 (full control) — treat the tier, not just the headline percentage, as the thing to compare.

If you're tracking where frontier capability is heading: the paper's authors are explicit that this is a moving floor, not a ceiling. Their stated expectation is that "the knowledge and expertise required to develop exploits will drop significantly" as Mythos-level capability becomes more widely available over the next 6-12 months — which is the whole argument for benchmarks like this existing publicly in the first place: so defenders see the curve before it's already priced into a live incident.

If you're building agent harnesses for security work specifically: the tier-based scoring also doubles as a debugging signal, not just a leaderboard number. A model that plateaus at T4 (reproduction) across most of its runs is telling you it can find and trigger bugs but lacks the primitive-construction skill to go further — useful context whether you're picking a model for defensive triage or deciding how much autonomy to grant an agent operating on untrusted, internet-facing code.

Related Reading

• Update — Sep 5, 2026: GPT-6 Astra launched with a reported 100% ExploitBench score, well clear of the field above — see the new section on this page for what that number does and doesn't cover, and how it lines up with OpenAI's confirmed Critical cybersecurity classification for the model.
• Update — Sep 1, 2026: Abliteration.ai hosted an abliterated GLM-5.3 as a paid red-team API, citing this benchmark's 24.4% → 54.4% jump as evidence — even though this post already showed GLM-5.3 trailing Fable 5 and GPT-5.6 Sol on the same benchmark.
• Update — Aug 28, 2026: The signal-vs-noise problem this benchmark quantifies showed up in production — Core Lightning confirmed critical bugs found by AI-generated CVE reports, chiefly from Kimi K3, out of a much larger flood of noisy AI-generated submissions.
• Update — Aug 28, 2026: OpenAI and 130+ companies (Anthropic, AWS, Google, Microsoft, and more) signed a Collective Cyberdefense open letter calling for an industry-wide surge in AI-enabled defense — the policy framing around exactly the dual-use capability this benchmark measures.
• Update — Aug 22, 2026: Anthropic upgraded Claude Security's scans to run on Mythos 5 and launched a $35M Defender Advantage Fund — the productized, findings-only route into the capability this benchmark measures.
• Update — Aug 18, 2026: The offensive capability this benchmark measures showed up live, unsupervised, against a real target — Wiz Research's autonomous Red Agent tool found and exploited a Snowflake GitHub Actions bug, self-correcting a failed exploit payload with no human help: Wiz Red Agent Hacked Snowflake's Jira — No Human Involved
• Update — Aug 16, 2026: Is GLM-5.3's CyberGym score independently verified? Not yet — Z.ai's own staged validation timeline, explained.
• GLM-5.3 Launch: Full Benchmarks — the CyberGym vs ExploitBench split that prompted this post
• Claude Mythos Preview and Project Glasswing — Anthropic's earlier public red-team disclosure on Mythos-class exploit capability
• GPT-5.5-Cyber vs Claude Mythos — OpenAI's defender-track rollout compared to Anthropic's approach
• OpenAI Daybreak and Codex Security — the defensive-agent counterpart to what ExploitBench scores on offense
• AI Cyber Guardrails Block US Defenders — what happens when models refuse exploit-adjacent security fixes
• Fable 5 vs Grok 4.6 vs GPT-5.6 Sol vs Qwen3.8-Max — independent cross-reference for the models on Z.ai's chart
• AI Benchmarks: The Complete Guide — where ExploitBench and CyberGym fit in the 2026 evaluation landscape
• ExploitBench paper (arXiv) · GitHub · Anthropic's testing writeup · Bugcrowd's benchmarking report · live ExploitBench leaderboard (llm-stats.com)

Benchmark numbers reflect the ExploitBench paper (submitted May 13, 2026), Anthropic's own published testing, and Bugcrowd's reporting, as of this post's publication date. Model scores on evolving benchmarks change as labs run new checkpoints — check the sources above for the latest leaderboard.