Most AI security benchmarks treat "did the model exploit the bug" as a yes/no question. GLM-5.3's own launch chart put a number on that question this week — GLM-5.3 scored 54.4% on something called ExploitBench, trailing Fable 5's 78.0% and GPT-5.6 Sol's 76.5% by more than 20 points, on the same chart where GLM-5.3 led the defensive CyberGym benchmark. That gap is the interesting part, and it only makes sense once you know what ExploitBench is actually measuring.
ExploitBench, built by Carnegie Mellon researchers Seunghyun Lee and David Brumley with Bugcrowd, is the first benchmark to score AI exploit generation as a ladder instead of a coin flip — 16 measurable capability flags across five tiers, run against 41 real, patched vulnerabilities in Chromium's V8 engine. Anthropic used it to test Claude Mythos Preview and reported a result the paper's authors call a meaningful inflection point for how cheap real exploit development is about to get.
TL;DR
| Question | Answer |
|---|---|
| What is it? | A benchmark scoring AI exploit-generation capability across a five-tier ladder, not a binary pass/fail |
| Who built it? | Seunghyun Lee and David Brumley (Carnegie Mellon University), with Bugcrowd |
| What's the target? | 41 real, patched (N-day) vulnerabilities in Chromium's V8 JavaScript/WASM engine |
| What's measured? | 16 capability flags — coverage, crash, sandbox primitives, arbitrary read/write, control-flow hijack, code execution |
| Best public-model result | ~2 of 41 CVEs reaching arbitrary code execution (GPT-5.5, per Bugcrowd's write-up) |
| Best private-model result | Mythos Preview: 21 of 41 CVEs reaching arbitrary code execution (Anthropic's own testing) |
| Is it open source? | Yes — MIT-licensed, code and eval containers on GitHub, transcripts on Hugging Face |
| Where does GLM-5.3 fit? | 54.4% — leads CyberGym (defense) but trails Fable 5 and GPT-5.6 Sol on ExploitBench (offense) |
What ExploitBench Actually Measures: A Five-Tier Ladder, Not a Checkbox
Most prior LLM security benchmarks collapse exploitation into a single success condition — usually "did the model crash the target." ExploitBench's authors argue that treats the easy 80% of the problem as the whole problem. Their framing, from the paper's abstract:
"Exploitation is not a binary event. It is a ladder of acquiring progressive capabilities, from executing a single buggy line of code to taking full control of the target."
The benchmark decomposes that ladder into 16 measurable flags — capability milestones like addrof and fakeobj primitives specific to V8 exploitation — rolled up into five tiers:
| Tier | Name | What it means |
|---|---|---|
| T5 | Coverage | The vulnerable code path is reachable at all |
| T4 | Reproduction | Triggers a crash or sanitizer violation (denial of service) |
| T3 | Target-specific primitives | Manipulates V8 objects, constructs fake objects |
| T2 | Generic primitives | Arbitrary memory read/write |
| T1 | Full control | Arbitrary code execution |
Each flag is verified programmatically, not by human grading — a meaningful design choice for a benchmark whose whole point is to resist score inflation.
Why V8, specifically
V8 wasn't picked for convenience. It's the JavaScript and WebAssembly engine behind Chrome, Node.js, Cloudflare Workers, and every Electron app — a genuinely high-value real-world target with layered production defenses already switched on: the heap sandbox, ASLR, and stack canaries. All 41 bugs in the benchmark are N-day vulnerabilities — already patched, reported in 2024 or later — so the eval measures whether a model can independently rediscover and weaponize a known flaw against a hardened build, not exploit an artificially simplified target.
The Numbers: What Models Actually Scored
The paper's own evaluation run tested 8 publicly deployed models alongside one non-public research-preview model. The headline finding, in the authors' words: reaching the vulnerable code and triggering a crash is routine, but arbitrary code execution is not — the private model reached roughly half the CVEs, while public models mostly stalled well before full control.
Anthropic ran its own follow-up testing against Claude Opus 4.6, Opus 4.7, and Mythos Preview, using a 300-turn budget with both baseline and "nudged" variants. The results, published on Anthropic's own research blog:
| Model | Result |
|---|---|
| Mythos Preview | Arbitrary code execution on 21 of 41 CVEs |
| Best public competitor (GPT-5.5) | Arbitrary code execution on 2 of 41 CVEs, per Bugcrowd |
| Claude Opus 4.6 / 4.7 | Progress into sandbox primitives; did not reliably escape the sandbox |
| Gemini and other Claude tiers | In-sandbox primitives only, per Bugcrowd's reporting |
Researchers specifically called out one case — CVE-2023-6702 — where Mythos Preview produced what they described as a "near-deterministic exploit" built "cleanly and flawlessly," including the first documented x86_64 exploit chain for that CVE from an AI agent. Bugcrowd's own framing of the jump from crash to control is blunt:
"The skill needed to go from Tier 4 to Tier 1 is enormous, and typically takes years of specialized training."
If the paper's ~50%-of-41 figure for the private model and Anthropic's later 21-of-41 figure look slightly different, that's expected — different evaluation rounds, not a contradiction. Both land in the same neighborhood and tell the same story: one frontier model crossed a threshold that every other publicly deployed model, as of testing, did not.
Where This Fits Next to CyberGym, GLM-5.3, and the Rest of the Field
ExploitBench measures offense. It has a sibling benchmark, ExploitGym, that scores exploit generation under a time budget instead of a turn budget. Both sit opposite CyberGym, which measures defensive capability — finding, patching, and hardening vulnerable code.
That distinction explains a result that otherwise looks contradictory. GLM-5.3's launch benchmarks show it leading CyberGym at 84.5% while scoring only 54.4% on ExploitBench — more than 20 points behind both Fable 5 (78.0%) and GPT-5.6 Sol (76.5%). Z.ai's own tagline for the model is "ready for cyber defense," not offensive capability, so a model that's strong on patching and comparatively weak on exploit generation is doing exactly what its maker says it's built for — not falling short of a claim it never made.
It also lines up with a broader pattern explainx.ai has tracked across 2026's cyber-focused model launches:
- Claude Mythos Preview and Project Glasswing — Anthropic's first public red-team writeup on Mythos-class exploit and zero-day discovery capability
- GPT-5.5-Cyber's defender rollout — OpenAI's own CyberGym numbers against Opus 4.7, framed around vetted defender access rather than raw offense
- OpenAI Daybreak and Codex Security — the agentic-defense counterpart to what ExploitBench measures on offense
- The cyber-guardrails debate from July — what happens when models refuse exploit-adjacent security fixes that defenders actually need
The through-line: every major lab is now publishing offense/defense splits instead of one blended "cybersecurity score," and ExploitBench is quickly becoming the standard reference point for the offense half of that split. See explainx.ai's full AI benchmarks guide for how it sits alongside SWE-bench, GAIA, and the rest of the 2026 eval landscape.
What This Means If You're Building or Defending
If you're a security team evaluating AI-assisted tooling: a high CyberGym score does not tell you anything about a model's offensive capability, and vice versa. Ask vendors for both numbers, or run your own ExploitBench-style eval against your actual stack — the code and containers are open source and MIT-licensed on GitHub.
If you're deciding which model to trust with untrusted code or bug bounty triage: the tier-based scoring is more useful than a single pass/fail number. A model stuck at T3 (target-specific primitives) is a very different risk profile from one that reliably reaches T1 (full control) — treat the tier, not just the headline percentage, as the thing to compare.
If you're tracking where frontier capability is heading: the paper's authors are explicit that this is a moving floor, not a ceiling. Their stated expectation is that "the knowledge and expertise required to develop exploits will drop significantly" as Mythos-level capability becomes more widely available over the next 6-12 months — which is the whole argument for benchmarks like this existing publicly in the first place: so defenders see the curve before it's already priced into a live incident.
If you're building agent harnesses for security work specifically: the tier-based scoring also doubles as a debugging signal, not just a leaderboard number. A model that plateaus at T4 (reproduction) across most of its runs is telling you it can find and trigger bugs but lacks the primitive-construction skill to go further — useful context whether you're picking a model for defensive triage or deciding how much autonomy to grant an agent operating on untrusted, internet-facing code.
Related Reading
- GLM-5.3 Launch: Full Benchmarks — the CyberGym vs ExploitBench split that prompted this post
- Claude Mythos Preview and Project Glasswing — Anthropic's earlier public red-team disclosure on Mythos-class exploit capability
- GPT-5.5-Cyber vs Claude Mythos — OpenAI's defender-track rollout compared to Anthropic's approach
- OpenAI Daybreak and Codex Security — the defensive-agent counterpart to what ExploitBench scores on offense
- AI Cyber Guardrails Block US Defenders — what happens when models refuse exploit-adjacent security fixes
- Fable 5 vs Grok 4.6 vs GPT-5.6 Sol vs Qwen3.8-Max — independent cross-reference for the models on Z.ai's chart
- AI Benchmarks: The Complete Guide — where ExploitBench and CyberGym fit in the 2026 evaluation landscape
- ExploitBench paper (arXiv) · GitHub · Anthropic's testing writeup · Bugcrowd's benchmarking report
Benchmark numbers reflect the ExploitBench paper (submitted May 13, 2026), Anthropic's own published testing, and Bugcrowd's reporting, as of this post's publication date. Model scores on evolving benchmarks change as labs run new checkpoints — check the sources above for the latest leaderboard.