Senior SWE-Bench: Snorkel AI's Benchmark for Under-Specified Tasks and Tasteful Code

SWE-Bench Pro tells an agent exactly which function to add, which file to edit, and which test must pass. Real senior engineers get a Slack message: "Google Books should be a fallback metadata source for BookWorm."

Senior SWE-Bench — from Snorkel AI, Princeton, and UW–Madison — closes that gap. Its tagline: "We treat agents like senior engineers, so why evaluate them like junior engineers?"

The first public leaderboard is sobering: Claude Opus 4.8 leads at 24.0% tasteful solve rate. Even the best frontier model fails more than three out of four senior-level tasks when correctness and code taste both count.

TL;DR

Three Pillars: Why Junior Benchmarks Lie

1. Under-specified features, not spec documents

Senior SWE-Bench feature tasks read like natural messages — median 639 characters, zero explicit code symbols in the prompt.

Compare to a typical SWE-Bench Pro instruction: ~6,008 characters, ~39 named code symbols, function signatures, file paths, and interface tables spelled out line by line.

Senior SWE-Bench median instruction length is 31% of SWE-Bench Pro — closer to how a staff engineer actually delegates.

Because solutions vary, Snorkel introduces a validation agent: expert-designed recipes generate behavioral tests that adapt to each submitted patch — not a single golden diff.

2. Bugs that need runtime investigation

Bug and performance tasks come from PRs that required real debugging — starting services, reading logs, profiling, reproducing flaky behavior from user reports.

This is the opposite of "apply this one-line fix from the issue description." Agents must investigate before they patch.

3. Taste scoring — ship the right code, not just passing code

A tasteful solve requires all of:

Verifiers can enforce load-bearing codebase practices never stated in the prompt — naming conventions, error-handling patterns, how sibling files structure imports.

The benchmark explicitly punishes agents that over-build, ignore local style, or ship 2× bloated diffs that technically pass tests.

Leaderboard: Opus 4.8 Leads, Everyone Else Struggles

Public results use Mini-SWE-Agent with model-specific effort settings. Tasteful solve rate (pass@1):

Takeaways:

• Opus 4.8 and Sonnet 5 dominate — Anthropic's stack owns the top two slots when taste gates apply.
• GPT-5.5 lands third at 16% — strong, but not the leader here (contrast DeepSWE where GPT-5.5 leads at 70%).
• GLM-5.2 at 12.5% beats Kimi K2.6 and Gemini 3.1 Pro — open-weight competition is real but still far from frontier Anthropic/OpenAI tiers.
• The scatter plot tracks solve rate vs. compute (output tokens + agent steps). More tokens does not automatically mean higher tasteful solve.

The site states plainly: the top-performing frontier models fail to complete tasks with senior-level correctness and taste over 75% of the time.

Task Design vs. Other Coding Benchmarks

Senior SWE-Bench compares itself to SWE-Bench Pro, DeepSWE, and internal FrontierCode metrics:

Tasks span libraries to multi-service applications, authored by engineers with hundreds of commits in each repo — not synthetic toy repos.

50 public tasks are released for community eval; 50 private tasks hold out contamination. Dataset lives on GitHub; runs go through Harbor (same framework as Terminal-Bench 2.0).

SWE-Bench Pro vs. Senior SWE-Bench: A Concrete Example

The Senior SWE-Bench site contrasts prompt styles side by side.

SWE-Bench Pro (excerpt — Open Library / BookWorm Google Books task):

• 40+ lines of Problem / Justify / Define Success / Proposal
• Named tuples (STAGED_SOURCES), exact URL templates, function signatures (stage_from_google_books, fetch_google_book, BaseLookupWorker)
• New public interfaces listed with inputs, outputs, and locations

Senior SWE-Bench equivalent:

#eng-platform

Engineer 10:42 AM
Google Books should be a fallback metadata source for BookWorm
for fallback/staging imports.

Same underlying feature. One prompt is a contract; the other is a message. Senior SWE-Bench argues agents that ace the contract may still fail the message — and production work looks like the message.

What the Validation Agent Does

Static golden tests break when valid implementations differ. Senior SWE-Bench's validation agent:

1. Takes the submitted solution
2. Applies expert-designed recipes per task type
3. Generates behavioral tests adapted to that specific patch
4. Scores rubric dimensions (correctness, bloat, practice fit, taste)

This mirrors how a senior reviewer would QA a PR — not just "CI green," but "would I merge this?"

Combined with verifiers (runtime tests) and taste metrics derived from observed codebase practices, the benchmark separates lucky passes from shippable work.

Implications for Teams Picking Coding Agents

Do not read 24% as "Opus is bad." Read it as "senior work is hard and our old benchmarks were measuring something easier."

Practical guidance:

1. Treat public SWE scores as necessary, not sufficient — especially SWE-Bench Pro-style over-specified tasks. See Cursor reward-hacking on SWE-Bench for contamination risks.
2. Add taste and review gates in your harness — Senior SWE-Bench's bloat and practice scores align with what staff engineers actually reject in code review.
3. Stress under-specified prompts in private evals — if your internal agent tickets read like Senior SWE-Bench, benchmark them that way. See how to build your own enterprise AI benchmark.
4. Budget for long horizons — Opus 4.8 averages 117K output tokens per feature task here. Token cost and latency matter as much as pass rate.
5. Use Harbor for reproducibility — same infra as Terminal-Bench; compare agent scaffolding, not just raw models.

For agent loop design — how to structure multi-step coding sessions — see loop engineering for Claude Code.

Limitations

• Early leaderboard — few models, single harness (Mini-SWE-Agent); other scaffolds may reorder ranks.
• 50 public tasks — small sample; high variance on pass@1.
• Taste metrics are codebase-specific — what counts as "good practice" in PostHog may not transfer to your monorepo.
• Private holdout — public leaderboards can still be gamed; private 50 tasks matter for official claims.
• Snorkel affiliation — Snorkel also co-created Terminal-Bench; evaluate independently where possible.

How to Run It

1. Visit senior-swe-bench.snorkel.ai
2. Clone the dataset from GitHub (linked on site)
3. Run agents via Harbor with Mini-SWE-Agent config
4. Read the blog post on validation agent, taste scoring, and QC (linked from overview)

Related Reading

• OpenAI SWE-Bench Pro audit — ~30% broken tasks (Jul 2026)
• How to Build Your Own Enterprise AI Benchmark (Nadella 2026)
• Nadella Reverse Information Paradox
• DeepSWE: GPT-5.5 Leads as SWE-Bench Pro Faces Scrutiny
• Terminal-Bench 2.0: The AI Agent Benchmark That Actually Matters
• Cursor Reward Hacking and SWE-Bench Contamination
• AI Benchmarks in 2026: Complete Guide
• Loop Engineering for Coding Agents
• Fable 5 After Relaunch — Developer Reaction

Sources: Senior SWE-Bench, Snorkel AI, Harbor framework. Leaderboard and task stats accurate as of July 2, 2026.