Last Secure Code Benchmark: Even Claude Fable 5 Writes Secure, Working Code Only 25.1% of the Time

AI coding agents write much of the code that ships, but a new benchmark suggests most of what they produce by default is not secure. The Last Secure Code Benchmark (LSCB), posted as a Hugging Face paper on October 14, 2026, reports that six frontier agents pass between 73.6% and 91.3% of tasks functionally, yet even the best one, Claude Fable 5, produces code that is both working and secure on only 25.1% of tasks.

That gap, between "it runs" and "it is safe," is the practical question behind every vibe-coding session. This post explains how the benchmark is built, what the numbers do and do not show, how it relates to earlier work, and what to change in your workflow. The arXiv preprint was submitted on October 10. We read the abstract and paper page; we found no peer review or independent replication, so treat the figures as the authors' reported results.

Quick reference

How the benchmark works

The design choice that matters most is the prompt. Every task gives a specification that asks only for functionality and never mentions security. That mirrors how most people actually prompt an agent: "add an upload endpoint," not "add an upload endpoint that resists path traversal." Hidden tests then check two things separately: does the code do what was asked, and does it avoid the specific weakness the original vulnerability represented.

The tasks come from 150 real vulnerabilities, expanded to 450 by testing three vibe-coding scenarios for each: completing a function, writing a whole file, and generating a repository's source tree from scratch. The benchmark spans seven widely used languages and 24 weakness categories. The authors say that is more CWE coverage than any benchmark they compare against, including 21 of the 2025 CWE Top 25 list and 8 of the Top 10 known-exploited-vulnerability weaknesses. Using real, known vulnerabilities rather than synthetic bugs makes the test closer to production risk, and the hidden-test approach reduces the chance of models gaming a visible checker.

What the results show

Functional performance is strong: between 73.6% and 91.3% of tasks pass. If you only measured whether code works, these agents look excellent. The security-adjusted view is bleak. The strongest agent, Claude Fable 5, yields code that is both functional and secure for 25.1% of tasks, and the other five land between 16.4% and 24.7%. The authors add that the best model passes only 12% of the vulnerability scenarios across all vibe-coding settings, which they describe as a strong concern for the security of vibe-coded software.

Put simply, roughly three out of four tasks, even for the leader, end with code that either does not work or contains the weakness the benchmark checks for. Because functional pass rates are so much higher, most of that failure is insecurity, not breakage. The code compiles, the feature demo works, and the vulnerability ships.

How does this fit with earlier research?

This is not the first study to find a wide gap between working and secure agent code. Related work includes SusVibes, the "Is Vibe Coding Safe?" benchmark of repository-level tasks built from past vulnerability fixes, and SecureVibeBench for C and C++. A search of those papers shows the same direction: SusVibes reports that most functionally passing agent solutions were insecure, and the best reported result in its materials was around 11.8% correct and secure. LSCB's numbers are higher for the top agent, which is plausible because it tests newer models, but different benchmarks use different tasks and metrics, so the percentages are not directly comparable across papers.

On explainx.ai, we have followed the practical side of this problem: vibe-coding nightmares and how to avoid them, the Claude Code security guidance plugin, and the argument that AI coding agents produce more code, not more software, where review becomes the bottleneck. LSCB adds a number to the claim that review cannot be skipped.

Caveats before you quote the numbers

• Default prompts, not best effort. Specifications deliberately omit security. Agents given explicit security instructions, a secure-coding skill or a review loop may do much better. The benchmark measures defaults.
• Hidden tests define "secure." A pass means the code avoided the specific weakness tested. It does not prove the rest of the code is safe, and a fail may include cases where the tested weakness is arguably out of scope for the stated function.
• Six agents, one snapshot. Agent harnesses change monthly. A score for a model plus scaffold today may not hold after the next update.
• Not peer reviewed here. The paper is an arXiv preprint (submitted October 10, 2026) that we read as an abstract only. The paper page links a project page and a GitHub repository, but we did not review the harness or the exact agent versions tested.
• Which Fable 5? The abstract names Claude Fable 5; Anthropic has since shipped later versions, and we cannot tell from the page which snapshot was tested. For the newer line, see our coverage of Fable 5.1 and Mythos 5.1.

Reading the three scenarios

The three vibe-coding settings probe different failure modes, and the paper page reports that the best model passes only 12% of vulnerability scenarios across all of them. Filling in a function is the narrowest task: the agent sees surrounding code and can often infer the safe pattern, so insecurity here usually means a missed check. Writing a whole file removes that context, and the agent must decide on its own how to handle inputs, paths and errors. Building a repository's source tree from scratch is the hardest and the closest to pure vibe coding, because there is no existing code to imitate and every design decision, from how sessions are stored to how queries are built, is the agent's.

Why would a model that writes working code still miss the secure version? Mostly because insecure code and secure code both satisfy the functional spec. A string-concatenated SQL query returns the same rows as a parameterized one in every normal test; only an attacker's input tells them apart. Training data is full of both styles, and without a signal in the prompt, the shorter, more common pattern wins. That also explains why explicit instructions and security-focused skills are likely to help, though the paper page we read does not measure that.

A fair way to use the benchmark is as a regression suite for your own setup: take a handful of its task types, run your agent with and without your security instructions, and compare. If your instructions do not move the secure-pass rate, they are not doing what you think.

What to do differently tomorrow

1. Say it in the prompt. Add security requirements to your task specs and agent instruction files: input validation, parameterized queries, no hardcoded secrets, safe file paths. The benchmark shows the default does not do this for you.
2. Run scanners in the loop. Static analysis, secret scanning and dependency checks should run on every agent change, not only at release. Anthropic's free OSS scanner for maintainers shows the same technology applied defensively.
3. Write security tests first. Hidden security tests are what exposed the gap. Your equivalent is a short set of abuse-case tests for each endpoint the agent adds.
4. Review the risky 10%. Authentication, authorization, file handling, deserialization and anything touching money or personal data deserve human review even when the rest is auto-merged.
5. Constrain what agents can do. Agents that write and run code need guardrails at runtime. AgentBeam, from the explainx.ai team, stops AI agents before they take dangerous actions, which complements, but does not replace, secure code generation.

Why it matters

Vulnerability discovery is getting automated on the attack side; our coverage of VulnCheck's work on AI-found bugs and exploit rates shows how quickly flaws get weaponized. If the code feeding those scanners is increasingly written by agents that are secure about a quarter of the time by default, the supply of exploitable bugs will keep growing. The benchmark's value is as a baseline: the next time a lab claims a model "writes secure code," the right question is whether it was measured without being told to.

Related reading

• Vibe-coding nightmares and how to avoid them
• Claude Code security guidance plugin
• AI coding agents: more code, not more software
• Anthropic's free OSS scanner for maintainers
• VulnCheck: AI-found bugs and exploit rates
• Claude Fable 5.1 and Mythos 5.1

Figures are the authors' reported results as shown on the Hugging Face paper page on October 14, 2026.