SWE-CoWork asks whether coding agents can act as digital employees — not only fixing an isolated GitHub issue, but finding owners, reconciling workplace records, and shipping verified work inside a software organization. The arXiv paper and project page describe 101 assignments across 101 real repositories, 11 languages, and 12 domains, each with 26 simulated colleagues.
The headline result is uncomfortable for leaderboard readers: Claude Opus 5.5 and GPT-6 Astra exceed 62% fail-to-pass (F2P) test success, yet full-task pass rates stay under 8%. Partial code changes look strong; accountable delivery does not.
TL;DR
| Question | Answer |
|---|---|
| Tasks? | 101 real repos, 26 colleagues, private executable tests |
| Top F2P? | Claude Opus 5.5 ~63.8%, GPT-6 Astra ~62.9% |
| Top full task? | GPT-6 Astra 7.92%, Opus 6.93% |
| Key failure? | Requirement tracking and verification, not tool count |
| Code? | GitHub bench plus separate harness |
Why another coding benchmark matters in November 2026
SWE-bench-style tasks reward agents that read an issue and patch a repo. Production engineering adds Slack threads, stale wikis, conflicting ticket comments, and owners on PTO. That is the gap Together AI agent evals hit on API documentation, and the gap Surge Dayjob tax assignments measured on professional knowledge work.
SWE-CoWork names the organizational slice explicitly. Each run begins with a short handoff, not a spec. Agents query a project board, versioned records, and colleagues with bounded knowledge, then must implement changes judged by F2P (new behavior) and P2P (regression) tests in the same run. A task completes only when every check passes — mirroring how loop engineering teams define done for agent PRs.
Issue ticket vs organizational assignment
SWE-CoWork compares issue-style coding tasks with organization-wide agent assignments
Image: Tokenwave AI SWE-CoWork
Figure 1 in the release contrasts the same Click requirement presented as a classic issue (left) versus embedded in SWE-CoWork (right). On the right, knowledge is distributed: the agent must discover who owns the requirement, what changed in records, and when a colleague's answer supersedes an older doc.
That design punishes agents tuned for single-shot codegen. A model that writes plausible diffs without follow-through can still lift F2P on easy nodes while failing full-task scoring — similar to how OECD writing scores penalize fluent essays that ignore rubric constraints.
Leaderboard: tests vs full delivery
Engineering gear, laptop, and hard hat illustrating SWE-CoWork coding agents as digital employees
The public leaderboard (November 2026 release) ranks eleven models:
| Model | F2P | P2P (existing behavior) | Full-task pass |
|---|---|---|---|
| Claude Opus 5.5 | 63.84% | 93.24% | 6.93% |
| GPT-6 Astra | 62.91% | 89.15% | 7.92% |
| Seed Evolving | 39.09% | 94.48% | 0.99% |
| Qwen3.8-Max | 35.70% | 91.80% | 0.00% |
F2P and P2P are macro-averages of per-task test pass rates. Full-task pass is the percentage of assignments where all F2P and P2P nodes succeed together — the metric that maps to mergeable PRs in a real eng org.
If you only tweet F2P, you will overstate readiness. Pair leaderboard numbers with Qwen Code harness lessons on when agents stop early.
Requirement acquisition beats raw tool volume
SWE-CoWork chart linking coding agent F2P scores to requirements gathered from simulated colleagues
Image: Tokenwave AI SWE-CoWork
Figure 5 plots F2P against the share of colleague-held requirements obtained (Spearman ρ ≈ 0.94 across models) while median tool calls show ρ ≈ 0.08. More slashing at tools does not fix delivery; asking the right colleague does.
Claude Opus 5.5 trajectory analysis in the paper reports the model routes ~93% of first questions about queried cards to accountable owners, yet obtains only ~64% of colleague-held requirements. Deferred questions go unfollowed; 31% of requirements marked complete still fail tests.
That is the organizational version of Meta Rebalancer assignment failures — you can solve local optimization while missing the global constraint.
Oracle specs expose an implementation ceiling
Researchers also ran an oracle-spec setting with complete requirements upfront. GPT-6 Astra F2P jumps from 62.91% to 78.84%, but full-task pass stays 7.92%. Supplying the spec removes discovery pain yet implementation, verification, and handoff still break.
For builders, the lesson parallels DESIGN.md templates in agent UI work: better prompts raise ceiling on understanding, not automatically on shipping.
Practical eval tip from Together's sandbox judges: log requirement nodes separately from test nodes in your harness. If F2P climbs while full-task flatlines, you have a coordination bug, not a model bug.
The four capabilities the authors emphasize
The project page frames dependable digital engineers around:
- Continuity — reconcile boards, docs, chats, and code when evidence conflicts.
- Initiative — start from a handoff, find owners, keep scope disciplined.
- Concurrent progress — when one thread stalls, advance other actionable work.
- Persistence — track each requirement through clarify → implement → verify → report.
Those map directly to product features missing in many Claude Code command workflows today: thread memory across days, owner graphs, and explicit verification gates before a PR description claims done.
How to try SWE-CoWork
Official artifacts:
- Paper PDF
- Benchmark repository
- Evaluation harness (task packages and credentials required separately per project docs)
- Demo video on the project page
Expect heavy setup: real repos, workplace simulators, and private tests are not a single pip install like some GitHub trending agents. Budget time to mirror the colleague API in your own org graph if you want internal repeats.
Suggested pilot for a platform team:
- Run five tasks with your production agent harness and log colleague messages plus test outcomes per requirement ID.
- Compare F2P-only vs full-task metrics weekly — do not optimize solely on partial tests.
- Add a human reviewer step when the agent claims all requirements complete; SWE-CoWork shows false completion is common.
Five workplace information traps
The paper's Figure 4 names recurring failure modes in organizational data:
- Late evidence — a correction lands after the agent already implemented against stale docs.
- Superseded conclusions — two records disagree; only one is authoritative.
- Ambiguous provenance — copied terminal output without run IDs.
- Scope distractors — tempting tasks that are out of charter.
- Fragmented ownership — multiple partial answers that never merge into one spec.
Those are exactly the failure classes Anthropic judgment-call research warns about when agents sound confident without traceable evidence. SWE-CoWork makes them unit-testable.
Claude Opus 5.5 behavior profile (paper metrics)
Deeper slices in the release focus on Claude Opus 5.5 because it leads F2P:
- Delivery claims vs tests: a material share of requirements marked complete still fail executable checks — the organizational version of hallucinated done.
- Deferred threads: when colleagues postpone answers, agents often never revisit the thread within budget.
- Message timing: early messages skew toward easy owners; hard owners get contacted late or not at all.
None of that shows up in single-issue benchmarks. If your dashboard only tracks lines changed or CI green on main, you will miss the gap SWE-CoWork highlights.
Cost and runtime expectations
Figure 6 on the project page relates median tool calls and token usage to F2P. High performers are not necessarily cheap: organizational settings reward more messages when they convert to obtained requirements.
Before you blame the model, log tokens per obtained requirement in your internal replay. If that ratio climbs while F2P flatlines, your harness probably needs better colleague routing, not a larger context window.
What SWE-CoWork does not measure
The benchmark does not score security, license compliance, or cost per task — topics our MCP misbinding and token economics posts cover elsewhere. It also does not replace domain certifications like Sakana JFinBench for regulated knowledge.
Treat SWE-CoWork as a process benchmark: it tells you whether your agent can finish like a staff engineer, not whether it should be unsupervised on prod.
Building an internal version
Teams without Tokenwave's simulator can still steal the metric design:
| SWE-CoWork idea | Internal hack |
|---|---|
| Colleagues with bounded knowledge | Rotate on-call humans in eval scripts |
| Workplace records | Point agents at Notion plus Jira plus git history with intentional conflicts |
| Full-task pass | Require CI green and ticket closure in one run |
| Requirement tracing | UUID every acceptance criterion in prompts |
That is more faithful than adding another pass@1 chart on isolated functions.
Reading the leaderboard without marketing spin
Vendors will cite 63% F2P in slide decks. SWE-CoWork's own authors emphasize under 8% full-task pass because enterprise buyers care whether an agent can close tickets, not whether it can touch tickets.
When you evaluate coding agent workshops or internal pilots, ask vendors for full-task or end-to-end metrics on workflows that include humans-in-the-loop. If they only provide isolated test pass rates, request a SWE-CoWork-style replay or refuse to expand the pilot.
HN and social threads will compare SWE-CoWork to SWE-bench Verified numbers. Keep the settings straight: SWE-CoWork uses private tests, colleague simulators, and multi-phase graphs. Scores are not portable without re-running the harness.
Related on explainx.ai
- Together AI agent evals — product-specific AX testing
- Surge Dayjob tax benchmark — professional task failure rates
- Qwen Code trending agent — open harness patterns
- Loop engineering guide — defining done for agents
- Meta Rebalancer assignment solver — optimization vs global constraints
- MCP tool misbinding — when agents call wrong tools after context loss
Primary sources: arXiv 2611.20218, SWE-CoWork project page, GitHub bench.
Leaderboard figures reflect the November 2026 public release as cited on the project page.
