AI is finding bugs faster than people can safely fix them, and Trail of Bits has just published evidence about what goes wrong when agents help write the fixes. On October 21, 2026 the security firm released two plugins, patching-bugs and regression-hunt, in its skills repository, and disclosed that a review of its own upstream patches found 21 confirmed regressions among more than 130 merged pull requests. The write-up is on the Trail of Bits blog.
The post matters beyond security teams. It is a rare measured account of the failure modes of agent-written changes, with a design that other teams can copy: structured steps, evidence requirements and a project-specific threat model. This guide summarizes the numbers, explains each plugin, and lists limits. It builds on our coverage of OpenAI's Patch the Planet disclosure push and Claude's attack surface reduction work.
TL;DR: what Trail of Bits reported
| Question | Answer |
|---|---|
| What shipped? | Two plugins: patching-bugs and regression-hunt |
| Where did they come from? | Patch the Planet, Trail of Bits' joint initiative with OpenAI, plus thousands of Codex sessions |
| Regressions found in own patches | 21 among more than 130 merged PRs (16 functional, 1 performance, 4 build/test) |
| Did any break security? | No, none broke a security protection |
| Review precision | 21 confirmed among 58 candidates in one run, about 36% |
| regression-hunt vs baseline | 25% fewer false positives (95% CI 14% to 38%) than a Codex agent told to review the PR for regressions |
| Trade-off | Some known regressions only reached borderline status |
Why the review happened
Trail of Bits says a September post reported regressions introduced by some of its Patch the Planet patches. It then reviewed every patch one of its engineers authored and maintainers merged. For each patch an agent looked for potential regressions, confirmed each with a proof of concept run before and after the change, and passed results to an engineer. By the firm's definition, a regression is a new defect introduced by a change that breaks previously correct and supported behavior along a path a real user or attacker can reach.
The honest finding is that the review itself was noisy. Of 58 candidates in a single run, 21 were real, roughly 36% precision. And because agents take different approaches each run, repeated runs generate more candidates to sort. Two conclusions followed: write patches in a standard way with the same checks every time, and check for regressions with written rules about what counts.
regression-hunt: proof or it did not happen
The regression-hunt plugin enumerates candidate regressions by tracing the diff through callers to real interfaces, reproduces each candidate on a live deployment, replays the proof with a fresh-context agent, and then has an adjudication agent judge whether the evidence meets the bar. The burden of proof sits on the agent: it must show the finding is not a false positive, an incomplete fix, a latent bug or a pre-existing bug. Only findings that pass every gate are reported.
If a threat model exists, the orchestrator passes it into every step so each agent must consult it before concluding. Against the 21 confirmed regressions as a test set, regression-hunt produced 25% fewer false-positive findings than a baseline Codex agent prompted with "review the PR for regressions." The firm is open about the cost: strict evidence requirements left some known regressions classed as borderline, so fewer false positives came with fewer confidently confirmed true ones. Next steps include working out why those cases stalled and improving evidence gathering.
patching-bugs: lessons from thousands of sessions
The patching-bugs plugin distills practices from analyzing thousands of Codex sessions by Trail of Bits engineers. Beyond reproducing bugs on live deployments and independent adversarial review, it stresses three ideas:
- The three-arm proof. Run the same reproduction against three versions. The original code must fail, the patched code must pass, and a complete revert must fail again. Bringing the bug back by removing the patch is stronger evidence than a single passing test.
- Invariant-guided repair. Fix the violated security invariant, not just the reported symptom, so the patch also covers variants of the same bug class.
- Mandatory regression analysis. The plugin requires regression-hunt, so every patch goes through the same pipeline.
Four design principles you can copy
Trail of Bits lists four principles that shaped both plugins, and they apply to any agent workflow:
- Opinionated guidance in as few words as possible. Give the agent wide latitude and record only your judgment on what a good patch and a real regression look like.
- Modular steps. Skills can be run together or alone. Our primer on agent skills explains why this packaging works.
- Strict input and output schemas. Each step takes and returns a defined structure with the claim, code revision, required check, raw evidence and open questions, so each step verifies the previous one instead of trusting a summary.
- A live proof of concept for every claim. If a step cannot produce a proof against a running instance reached through user-facing interfaces, its result is inconclusive.
How to adopt it in your repository
The plugins improve when given project knowledge. Trail of Bits asks maintainers for two files:
- THREAT_MODEL.md describing assets whose compromise matters, trust boundaries, access controls, supported configurations, what is or is not a vulnerability, and the evidence required for a valid bug report.
- PATCHING_GUIDELINES.md or AGENTS.md describing how to build and test the project, how big a fix should be, where it should go, how to present changes, and what a regression test must show.
The firm says it wrote a threat model and a project-specific patching skill for GPAC, and that liboqs and validator.js have published threat models following its guidance. The post links the plugin repository and an example of each file. The stated workflow is: install both plugins, add the two files, and run them on your next bug report. If you let agents act on production systems while patching, AgentBeam is the agent security platform from the explainx.ai team that stops AI agents before they take dangerous actions.
A worked example of the three-arm proof
Imagine a bug report that says a parser crashes on a malformed header. A common agent workflow writes a fix, adds a test that passes, and stops. The three-arm proof asks for more. First, run the reproduction against the original code and confirm it fails, which proves the reproduction is real. Second, run it against the patched code and confirm it passes. Third, revert the patch completely and confirm the reproduction fails again.
The third arm is the clever one. If the test still passes after the revert, something else changed the outcome, such as a different build flag, a cached artifact or a flaky environment, and the patch gets no credit. This example is our illustration of the method described in the post, not a case from the firm's data.
What people are asking
Is this only for security patches? The plugins come from security work, but the ideas, namely schemas, live proofs and written definitions of a regression, apply to any bug-fix workflow.
Why not just ask the agent to review the diff? That is the baseline Trail of Bits compared against, and its version produced more false positives. Asking for a review without defined gates invites plausible-sounding but unproven findings.
Do I need Codex? The firm built the plugins from Codex sessions and tested against a Codex baseline. Whether they port cleanly to other agents is not stated.
Will this replace human reviewers? No. In the described process, engineers review every result, and the plugins are meant to give maintainers evidence and structure so review is faster and better informed.
How reliable is the 25% figure? It is a preliminary comparison on a small test set of 21 known regressions, and the confidence interval of 14% to 38% shows the uncertainty.
What this says about AI-assisted patching
Three takeaways stand out. First, 21 regressions in 130 patches is about 16%, but the firm stresses none broke a security protection, which is the distinction maintainers care about most. Second, the figures come from a single organization's reviewed patches and a preliminary evaluation, so treat 25% as an early signal with a wide interval. Third, the real lesson is process: written definitions, schemas and live proofs reduce agent noise more than prompt tweaks.
This matters now because disclosure is accelerating. The PHP2Shell flaws were found with AI help and patched within days, and Anthropic's free scanning for maintainers in our OSS scanner post adds to the queue. Maintainers will need quality gates for the fixes, not just the finds.
Limits and open questions
- The test set is the 21 regressions the firm itself found, which may bias evaluation toward what its own process can detect.
- Regression-hunt trades recall for precision; borderline cases need human judgment.
- The work uses Codex; results with other agents are not reported.
- The evaluation for patching-bugs itself is still being built, per the post.
- Live proof requires a deployable instance, which can be hard for libraries or hardware-dependent software.
Related reading
- OpenAI Patch the Planet and remote patching
- Claude attack surface reduction: 493 issues across 25 projects
- Anthropic's free scanner for open-source maintainers
- Claude Security continuous scanning
- PHP2Shell: what happened and what to do
- What are agent skills?
Figures are from the Trail of Bits post of October 21, 2026 and are preliminary.