explainx.ai350K+ learn from us
TrendingAI News TodayPathwaysSkills
Pricing
explainx.ai

Upskill in AI — 16 free pathways, live workshops & bootcamps, and 50+ courses from practitioners. Plus the skills, tools, and MCP servers to practice on.

follow us

follow on google

Add explainx.ai as a preferred source

corporate training

support@explainx.ai

get started

Find your pathTake Free Evaluation

community

Join the community

learn

mind: share how you thinkpathways — start freeworkshopsbootcampscoursescompare Explainxcertificationsmock testsexplainx universitycorporate traininglearn skills & mcp

discover

skillsmcp serversexplainx mcpestacktoolsmdx readeragentsllmsdesignsdictionarypeopleagi trackerfelony benchranks

company

aboutvisionmissionteaminstructorsteach on explainxpartnershipscommunityhackathonscareers

content

daily AI newsstate of AI — live resultsblogreleasespromptsgeneratorsresource libraryfor LLMsexplainx.ai kids

solutions

all solutionsdeveloper upskillingmarketing upskillingproduct manager upskillingleadership upskilling

newsletter · weekly

Get AI news, tools, and insights in your inbox.

supportcontactprivacytermsdata rightshow we create contentsubmission guidelines

© 2026 AISOLO Technologies Pvt Ltd

explainx.ai

On this page

  • TL;DR
  • Why another coding benchmark matters in November 2026
  • Issue ticket vs organizational assignment
  • Leaderboard: tests vs full delivery
  • Requirement acquisition beats raw tool volume
  • Oracle specs expose an implementation ceiling
  • The four capabilities the authors emphasize
  • How to try SWE-CoWork
  • Five workplace information traps
  • Claude Opus 5.5 behavior profile (paper metrics)
  • Cost and runtime expectations
  • What SWE-CoWork does not measure
  • Building an internal version
  • Reading the leaderboard without marketing spin
  • Related on explainx.ai
← Back to blog

explainx / blog

SWE-CoWork: Coding Agents as Digital Employees Benchmark

Coding Agents, Benchmarks, Software Engineering, Agent Evals, Research

Part of AI Coding Tools

SWE-CoWork (arXiv 2611.20218) benchmarks 101 real repos with 26 simulated colleagues. Top agents pass most new tests yet complete under 8% of assignments.

Nov 17, 2026·8 min read·Yash Thakker
add explainx.ai
go deep
SWE-CoWork: Coding Agents as Digital Employees Benchmark

SWE-CoWork asks whether coding agents can act as digital employees — not only fixing an isolated GitHub issue, but finding owners, reconciling workplace records, and shipping verified work inside a software organization. The arXiv paper and project page describe 101 assignments across 101 real repositories, 11 languages, and 12 domains, each with 26 simulated colleagues.

The headline result is uncomfortable for leaderboard readers: Claude Opus 5.5 and GPT-6 Astra exceed 62% fail-to-pass (F2P) test success, yet full-task pass rates stay under 8%. Partial code changes look strong; accountable delivery does not.

TL;DR

table · 2 cols
QuestionAnswer
Tasks?101 real repos, 26 colleagues, private executable tests
Top F2P?Claude Opus 5.5 ~63.8%, GPT-6 Astra ~62.9%
Top full task?GPT-6 Astra 7.92%, Opus 6.93%
Key failure?Requirement tracking and verification, not tool count
Code?GitHub bench plus separate harness
Daily briefFrom Yash · 350K+ students taught

Catch up on AI

One AI story explained in plain English, plus one thing to try. Every day, free. Unsubscribe anytime.

Why another coding benchmark matters in November 2026

SWE-bench-style tasks reward agents that read an issue and patch a repo. Production engineering adds Slack threads, stale wikis, conflicting ticket comments, and owners on PTO. That is the gap Together AI agent evals hit on API documentation, and the gap Surge Dayjob tax assignments measured on professional knowledge work.

SWE-CoWork names the organizational slice explicitly. Each run begins with a short handoff, not a spec. Agents query a project board, versioned records, and colleagues with bounded knowledge, then must implement changes judged by F2P (new behavior) and P2P (regression) tests in the same run. A task completes only when every check passes — mirroring how loop engineering teams define done for agent PRs.

Issue ticket vs organizational assignment

SWE-CoWork compares issue-style coding tasks with organization-wide agent assignmentsSWE-CoWork compares issue-style coding tasks with organization-wide agent assignments Image: Tokenwave AI SWE-CoWork

Figure 1 in the release contrasts the same Click requirement presented as a classic issue (left) versus embedded in SWE-CoWork (right). On the right, knowledge is distributed: the agent must discover who owns the requirement, what changed in records, and when a colleague's answer supersedes an older doc.

That design punishes agents tuned for single-shot codegen. A model that writes plausible diffs without follow-through can still lift F2P on easy nodes while failing full-task scoring — similar to how OECD writing scores penalize fluent essays that ignore rubric constraints.

Leaderboard: tests vs full delivery

Engineering gear, laptop, and hard hat illustrating SWE-CoWork coding agents as digital employeesEngineering gear, laptop, and hard hat illustrating SWE-CoWork coding agents as digital employees

The public leaderboard (November 2026 release) ranks eleven models:

table · 4 cols
ModelF2PP2P (existing behavior)Full-task pass
Claude Opus 5.563.84%93.24%6.93%
GPT-6 Astra62.91%89.15%7.92%
Seed Evolving39.09%94.48%0.99%
Qwen3.8-Max35.70%91.80%0.00%

F2P and P2P are macro-averages of per-task test pass rates. Full-task pass is the percentage of assignments where all F2P and P2P nodes succeed together — the metric that maps to mergeable PRs in a real eng org.

If you only tweet F2P, you will overstate readiness. Pair leaderboard numbers with Qwen Code harness lessons on when agents stop early.

Requirement acquisition beats raw tool volume

SWE-CoWork chart linking coding agent F2P scores to requirements gathered from simulated colleaguesSWE-CoWork chart linking coding agent F2P scores to requirements gathered from simulated colleagues Image: Tokenwave AI SWE-CoWork

Figure 5 plots F2P against the share of colleague-held requirements obtained (Spearman ρ ≈ 0.94 across models) while median tool calls show ρ ≈ 0.08. More slashing at tools does not fix delivery; asking the right colleague does.

Claude Opus 5.5 trajectory analysis in the paper reports the model routes ~93% of first questions about queried cards to accountable owners, yet obtains only ~64% of colleague-held requirements. Deferred questions go unfollowed; 31% of requirements marked complete still fail tests.

That is the organizational version of Meta Rebalancer assignment failures — you can solve local optimization while missing the global constraint.

Oracle specs expose an implementation ceiling

Researchers also ran an oracle-spec setting with complete requirements upfront. GPT-6 Astra F2P jumps from 62.91% to 78.84%, but full-task pass stays 7.92%. Supplying the spec removes discovery pain yet implementation, verification, and handoff still break.

For builders, the lesson parallels DESIGN.md templates in agent UI work: better prompts raise ceiling on understanding, not automatically on shipping.

Practical eval tip from Together's sandbox judges: log requirement nodes separately from test nodes in your harness. If F2P climbs while full-task flatlines, you have a coordination bug, not a model bug.

The four capabilities the authors emphasize

The project page frames dependable digital engineers around:

  1. Continuity — reconcile boards, docs, chats, and code when evidence conflicts.
  2. Initiative — start from a handoff, find owners, keep scope disciplined.
  3. Concurrent progress — when one thread stalls, advance other actionable work.
  4. Persistence — track each requirement through clarify → implement → verify → report.

Those map directly to product features missing in many Claude Code command workflows today: thread memory across days, owner graphs, and explicit verification gates before a PR description claims done.

How to try SWE-CoWork

Official artifacts:

  • Paper PDF
  • Benchmark repository
  • Evaluation harness (task packages and credentials required separately per project docs)
  • Demo video on the project page

Expect heavy setup: real repos, workplace simulators, and private tests are not a single pip install like some GitHub trending agents. Budget time to mirror the colleague API in your own org graph if you want internal repeats.

Suggested pilot for a platform team:

  1. Run five tasks with your production agent harness and log colleague messages plus test outcomes per requirement ID.
  2. Compare F2P-only vs full-task metrics weekly — do not optimize solely on partial tests.
  3. Add a human reviewer step when the agent claims all requirements complete; SWE-CoWork shows false completion is common.

Five workplace information traps

The paper's Figure 4 names recurring failure modes in organizational data:

  • Late evidence — a correction lands after the agent already implemented against stale docs.
  • Superseded conclusions — two records disagree; only one is authoritative.
  • Ambiguous provenance — copied terminal output without run IDs.
  • Scope distractors — tempting tasks that are out of charter.
  • Fragmented ownership — multiple partial answers that never merge into one spec.

Those are exactly the failure classes Anthropic judgment-call research warns about when agents sound confident without traceable evidence. SWE-CoWork makes them unit-testable.

Claude Opus 5.5 behavior profile (paper metrics)

Deeper slices in the release focus on Claude Opus 5.5 because it leads F2P:

  • Delivery claims vs tests: a material share of requirements marked complete still fail executable checks — the organizational version of hallucinated done.
  • Deferred threads: when colleagues postpone answers, agents often never revisit the thread within budget.
  • Message timing: early messages skew toward easy owners; hard owners get contacted late or not at all.

None of that shows up in single-issue benchmarks. If your dashboard only tracks lines changed or CI green on main, you will miss the gap SWE-CoWork highlights.

Cost and runtime expectations

Figure 6 on the project page relates median tool calls and token usage to F2P. High performers are not necessarily cheap: organizational settings reward more messages when they convert to obtained requirements.

Before you blame the model, log tokens per obtained requirement in your internal replay. If that ratio climbs while F2P flatlines, your harness probably needs better colleague routing, not a larger context window.

What SWE-CoWork does not measure

The benchmark does not score security, license compliance, or cost per task — topics our MCP misbinding and token economics posts cover elsewhere. It also does not replace domain certifications like Sakana JFinBench for regulated knowledge.

Treat SWE-CoWork as a process benchmark: it tells you whether your agent can finish like a staff engineer, not whether it should be unsupervised on prod.

Building an internal version

Teams without Tokenwave's simulator can still steal the metric design:

table · 2 cols
SWE-CoWork ideaInternal hack
Colleagues with bounded knowledgeRotate on-call humans in eval scripts
Workplace recordsPoint agents at Notion plus Jira plus git history with intentional conflicts
Full-task passRequire CI green and ticket closure in one run
Requirement tracingUUID every acceptance criterion in prompts

That is more faithful than adding another pass@1 chart on isolated functions.

Reading the leaderboard without marketing spin

Vendors will cite 63% F2P in slide decks. SWE-CoWork's own authors emphasize under 8% full-task pass because enterprise buyers care whether an agent can close tickets, not whether it can touch tickets.

When you evaluate coding agent workshops or internal pilots, ask vendors for full-task or end-to-end metrics on workflows that include humans-in-the-loop. If they only provide isolated test pass rates, request a SWE-CoWork-style replay or refuse to expand the pilot.

HN and social threads will compare SWE-CoWork to SWE-bench Verified numbers. Keep the settings straight: SWE-CoWork uses private tests, colleague simulators, and multi-phase graphs. Scores are not portable without re-running the harness.

Related on explainx.ai

  • Together AI agent evals — product-specific AX testing
  • Surge Dayjob tax benchmark — professional task failure rates
  • Qwen Code trending agent — open harness patterns
  • Loop engineering guide — defining done for agents
  • Meta Rebalancer assignment solver — optimization vs global constraints
  • MCP tool misbinding — when agents call wrong tools after context loss

Primary sources: arXiv 2611.20218, SWE-CoWork project page, GitHub bench.

Leaderboard figures reflect the November 2026 public release as cited on the project page.

Spotted something out of date? Let us know.
Daily briefFrom Yash · 350K+ students taught

Catch up on AI

One AI story explained in plain English, plus one thing to try. Every day, free. Unsubscribe anytime.

  • ✓ Free
  • ✓ One email a day, no spam
  • ✓ Unsubscribe anytime
Yash Thakker

Written by

Yash Thakker

Yash is an AI expert with over 300K learners. Join his workshops →

View Yash Thakker in People in AI →

Related posts

Oct 30, 2026

Co-Design Bench: Best Coding Agent Fully Solves Only 26.4% of Real App Edits

Co-Design Bench tests whether coding agents can turn real users multimodal requests into working application revisions. The authors, from the Qwen team and collaborators, report a large gap between partial progress and finished tasks, and publish the code and dataset.

Oct 28, 2026

CodeCap-Bench: Can Executable Code Be a Better Video Caption Than Words?

A new benchmark asks models to watch a short 2D video and write code that renders it back. The best agent setup keeps about 73 out of 100 of the visual information, and motion is consistently harder than appearance.

Oct 9, 2026

SWE-Game: Can Coding Agents Build the Games We Want? What the New Godot Benchmark Shows

A new arXiv benchmark, SWE-Game, asks coding agents to build, repair and port real games in Godot and Unity. Opus5 tops every task type, but the best construction score is still below 60 out of 100. explainx.ai explains how it works and what to take from it.