Boris Cherny Used Opus 5.5 to Formally Verify the Claude Agent SDK

Boris Cherny, who leads Claude Code at Anthropic, posted a claim that's easy to skim past and genuinely worth stopping on: "I used Opus 5.5 to formally verify the Claude Agent SDK using Lean. A couple short prompts = 16 PRs fixing various bugs and race conditions." The headline number is real, but the more useful story is in the workflow underneath it — how a formal-methods project that would normally require deep expertise in two specialist languages got compressed into natural-language prompts, and what that actually surfaced.

TL;DR

What formal verification actually means here

Most bug-finding relies on running code and checking whether the observed behavior matches expectations — unit tests, integration tests, fuzzing. Formal verification is a different category entirely: instead of running the code, you build a mathematical model of what the system is supposed to do, then prove — not test, prove — that the model satisfies specific properties for every possible input and execution order, not just the ones a test suite happens to exercise. That's specifically why formal methods excel at catching race conditions and state-management bugs: a test can only catch a race condition if it happens to run at exactly the wrong moment, while a formal model can show a bad interleaving is possible even if no test has ever hit it.

Lean 4 is a proof assistant — you write a formal specification, then construct a proof of a property against it, and the Lean compiler mechanically checks every step of the proof is actually valid. TLA+ is a related tool specifically built for modeling concurrent and distributed systems, developed by Leslie Lamport, widely used at companies like Amazon to catch distributed-systems bugs before they ship. Cherny used both, sometimes combined on the same problem: "I sometimes combine Lean and TLA+ to look for issues around data flow, concurrency, and state mgmt."

The numbers, and what they actually show

The project's own summary graphic breaks the 16 PRs into three categories, each with a distinct non-test-line delta:

That "Simplify" row is worth noticing on its own: six of the sixteen PRs removed more code than they added, a net -195 lines. Formal verification doesn't just find bugs — surfacing an unnecessarily complex state machine or redundant code path is often a direct byproduct of building the formal model in the first place, since a simpler model is both easier to write and easier to verify. Cherny's own framing supports this: "This approach is super useful for formally modeling your code and finding bugs that a human probably wouldn't have spotted."

Test coverage moved substantially too — 94 test cases added against 9 removed, with the underlying test-line count growing by 3,928 lines added against 247 removed. That's a meaningfully larger addition than the fix PRs themselves, suggesting the formal-verification process generated regression coverage for exactly the edge cases the proofs uncovered, which is arguably the more durable output than the fixes alone: those tests keep catching the same class of bug if it's reintroduced later, long after this specific project is done.

Infographic showing the Claude Agent SDK Lean 4 formal verification results — 16 pull requests, 24 bugs fixed, 1,529 theorems across 6 Lean models with zero unproven gaps

The genuinely interesting part: an LLM doing formal methods without deep expertise

The claim that should get the most scrutiny, not the least, is Cherny's own: "I don't know either language well, but Claude is excellent at both." That's a real lowering of the barrier to entry for formal methods, which have historically required years of specialized training to use productively — Lean's own community has spent a decade building tooling specifically because the learning curve is steep even for experienced software engineers. If an agent can take a natural-language description of a system and produce a well-formed Lean model of it, checkable to zero sorry gaps, that changes who can access this class of bug-finding, not just how fast it happens for people who already could.

It's worth being precise about what "zero sorry" actually certifies and doesn't. A sorry in Lean is a placeholder that lets a proof compile without being complete — the compiler accepts it as a promise to fill in later, which means a proof riddled with sorry statements isn't actually verified at all, just typed out in the shape of a proof. Zero sorry across 1,529 theorems means every one of those claims is mechanically checked, with no gaps and no unverified assumptions in the formal proofs themselves. What it doesn't certify is that the formal model correctly captures the real system's actual behavior — that's a translation step between "what the code does" and "what the Lean model says the code does," and getting that translation wrong is the classic failure mode in formal methods generally, independent of who or what wrote the model. Nineteen of the 24 bugs being found directly by the proofs (rather than the remaining 5, presumably found through the broader review and testing process around the project) is a strong signal the translation was largely faithful — but it's a signal, not a guarantee.

Why this fits a broader September pattern

This isn't Anthropic's only formal-methods result this month. Claude separately produced a 13-million-line, machine-checked Lean 4 proof of Fermat's Last Theorem earlier in September 2026 — a project mathematicians had expected to take years, completed in 11 days with 29,500 supporting theorems formalized along the way. That result was pure mathematics; Cherny's is applied software verification on a real, shipping SDK. Together they point toward the same underlying capability showing up in two very different domains: an LLM's ability to construct large, mechanically-checkable formal artifacts from much shorter human-supplied direction than the artifact itself would suggest.

Boris Cherny's benchmarking work has appeared in explainx.ai's coverage before — his prompt injection resistance comparison across 15 models sparked its own debate earlier in September about whether an Anthropic safety researcher publicly grading competitor models by name was appropriate. This project is a different kind of contribution from the same team — not a comparative benchmark, but a direct demonstration of formal methods as a practical, AI-assisted engineering technique on Anthropic's own production SDK.

Honest limitations

• 11 of the 16 PRs remained open at the time of the original post, meaning a meaningful share of the claimed fixes had not yet been merged and reviewed by the wider team.
• This account comes from Anthropic's own Claude Code lead, testing Anthropic's own model on Anthropic's own SDK — there's no independent replication of these specific results by a third party as of publication.
• "Zero sorry gaps" verifies the formal proofs are internally complete, not that the Lean/TLA+ models perfectly match the real system's behavior — that translation step is the harder-to-audit part of any formal-verification claim, by AI or human.
• Cherny doesn't specify how many prompts or how much iteration went into each PR beyond "a couple short prompts" for the initial pass — it's unclear how much back-and-forth refinement followed that initial exchange before landing at 1,529 zero-gap theorems.

What this means for builders

If your team maintains a system with genuine concurrency or complex state-machine logic — the exact class of bug that unit tests are structurally bad at catching — this is a concrete data point that formal verification is now reachable without a formal-methods specialist on staff, at least as a starting experiment. The practical move isn't "learn Lean" so much as "try pointing Opus 5.5 or a comparable model at your trickiest concurrent module and see what a Lean or TLA+ model of it surfaces." The 6-simplify-PR result is worth remembering too: even if you don't trust the bug list on first pass, the process of building a formal model tends to expose accidental complexity in state machines that's worth removing regardless of whether it was ever going to cause a real bug.

Update — September 27, 2026: This Lean and TLA+ pass is a working example of the bet in José Valim on programming languages in the AI era: agents can write explicit types and proofs, so languages should optimize for guarantees a checker can use.

Related on explainx.ai

• Programming languages in the AI era: what to expose to agents — why explicit types and proofs are the surface to invest in once agents write the annotations
• Claude Wrote the First Machine-Checked Proof of Fermat's Last Theorem — the other major September 2026 Lean 4 result from Claude, in pure mathematics rather than applied SDK verification
• Boris Cherny Benchmarks GPT-6 Astra's Prompt Injection Resistance — the same researcher's earlier, more contested September 2026 project
• Claude Opus 5.5 Launch: Every Benchmark and Reaction — the model used throughout this verification project
• Claude Agent SDK Support for AGENTS.md — explainx.ai's coverage of the SDK ecosystem this project verified
• How to Read AI Benchmarks Without Getting Fooled — the skepticism framework applied to the "zero sorry gaps" claim in this post

Primary source: Boris Cherny on X, September 23, 2026.

This post reflects the project as posted by Boris Cherny as of September 23, 2026. PR merge status and any further results may have changed since publication.