A new preprint argues that most of what you pay for in a coding-agent harness is tokens, not accuracy. In "What Does a Harness Buy? Tokens, Mostly", Yangze Liu and Zhongyi Han of Shandong University hold the model fixed, swap the coding agent harness around it, and find that SWE-bench Verified pass rates move about as much as they do when you simply rerun the same setup. Cost per task, on the other hand, can triple. The result was first covered by The Decoder, and the paper's data and scripts are public in the authors' GitHub repository.
If you have been choosing between Claude Code, OpenCode and a minimal Bash-only loop based on leaderboard gaps of a few points, this study says those gaps are mostly inside the noise. If you pay per token, it says something more actionable: look at the size of the system prompt and the number of steps.
TL;DR: the questions people will ask
| Question | Answer |
|---|---|
| What was tested? | Claude Code, OpenCode and mini-SWE-agent, each wrapped around five models, on SWE-bench Verified |
| How big was it? | 8,032 trials in the released table; 492 of 500 Verified tasks |
| Does the harness change the score? | Barely. The heaviest and lightest harness land within five points of each other |
| Does it change the bill? | Yes, up to 3x per task on the same model |
| What is the one effect that beats noise? | OpenCode trailing, by up to 9 points on the large pool |
| Does caching fix the cost gap? | No, 96 to 99 percent of input tokens were cache hits everywhere and the ranking held |
| Is the code available? | Yes. MIT for code, CC BY 4.0 for data, with a reproduce script |
What does a coding-agent harness actually do?
A coding agent is a language model plus a harness: the system prompt, the tool set, and the context management that decide what the model sees on each step. mini-SWE-agent is the minimal end of that spectrum: the model gets a Bash tool and nothing else, with no context management. Claude Code and OpenCode add dedicated file-edit and search tools, long instruction prompts, and automatic history compression.
That difference is what vendors and open-source projects compete on, and it is why harnesses get their own benchmark rows. We have covered the idea from several angles, including Agent Lightning's argument for training with the harness you ship, Microsoft Research's Self-HPO, an agent that rewrites its own harness and how self-evolving coding agents learn. This study asks the blunter question: what does all that scaffolding buy you in measured pass rate?
How was the study run?
The setup is unusually careful, which is why the findings are worth taking seriously.
- Harnesses: Claude Code 2.1.x, mini-SWE-agent 2.4.6 and OpenCode 1.18.x, versions pinned, each trial in its own container using the Harbor runner.
- Models: Qwen3.6-35B-A3B and Qwen3.8-27B (run locally on vLLM), plus DeepSeek-V4-Flash, GLM-5.3-Flash and HY4-Preview through vendor APIs. All five ran in all three harnesses. Claude Opus 5 ran in Claude Code only, as an anchor.
- Tasks: 492 of the 500 SWE-bench Verified tasks. A "hard45" set holds the 45 tasks in the two hardest difficulty bands; a "pool447" set holds the rest.
- Budget: 300 steps per trial, provider-default decoding.
- Statistics: paired McNemar tests, equivalence tests with a 5-point margin on the large pool, and reruns of identical configurations to calibrate noise.
- Cost: computed from each trial's own token counts under one vendor price list per model.
Is the harness gap bigger than rerun noise?
Mostly no. The authors measured how much a result changes when you do nothing but run the same configuration again, and compared that to swapping the harness or swapping the model.
| What you change | Median share of hard tasks whose outcome flips |
|---|---|
| Rerun the same setup | 13% |
| Swap the harness | 13% |
| Swap the model | 22% |
The harness swap is indistinguishable from a rerun. On the 45 hard tasks, every paired harness difference had a confidence interval that included zero. The largest first-run spread was a handful of solved tasks: for example, HY4-Preview solved 26 of 45 in mini-SWE-agent versus 23 in Claude Code and 21 in OpenCode, while Qwen3.6-35B-A3B solved 15 in mini-SWE-agent versus 11 in both of the others.
Lightweight scaffold around a model, illustrating how little a coding agent harness changes pass rates
The paper also checked whether harness "winners" replicate at the task level. Across six harness pairs that were each run twice, 5 tasks split the same way both times and 7 split the opposite way. That is what you would expect if the differences were noise.
On the large 447-task pool, Claude Code versus mini-SWE-agent differed by 1.8 points with Qwen3.6 and 1.4 points with Qwen3.8, and both pairs passed the authors' equivalence test at a 5-point margin. In other words, the extra tools, long prompts and compaction did not measurably help.
The Decoder notes one consequence: a claimed 2.4-point lead of a vendor harness over Claude Code falls inside the up-to-6-point rerun variation the authors observed on SWE-bench Verified. This is the same lesson our readers saw in the Vals AI result that agent teams cost up to 5.1x more for barely better results: more machinery is not automatically more accuracy.
Where does the harness actually matter? OpenCode's losses
The one effect that clears the noise is a loss. On the large pool with Qwen3.8-27B, Claude Code beat OpenCode by 7.9 points and mini-SWE-agent beat it by 9.2 points, both significant after correction. With Qwen3.6-35B-A3B the gap to mini-SWE-agent was 4.7 points but did not survive correction across six comparisons.
The paper gives concrete mechanisms:
- Output cutoff. In 38 of 447 OpenCode trials on Qwen3.8-27B the model hit a 32k output cap. Claude Code hit a cutoff in 52 trials but re-prompted the model and lost only one. Transplanting Claude Code's resume prompt into OpenCode raised mean reward from 0.585 to 0.634 on 41 tasks, though with that sample the change was not statistically significant (p = 0.29).
- Stopping sooner. With Qwen3.6-35B-A3B, median model calls were 28 in OpenCode, 55 in Claude Code and 71 in mini-SWE-agent.
- Context overflow in the minimal harness. mini-SWE-agent has no compaction and overflowed on 11 of 45 hard tasks with Qwen3.8-27B, though most of those tasks were also failed by the other harnesses.
The authors also flag a fairness caveat: the Qwen arms gave Claude Code a 36,000-second wall-clock limit against 5,760 seconds for OpenCode, and six OpenCode trials hit the soft timeout. Treat the OpenCode deficit as a setup-sensitive finding, not a verdict on the product. If you run it locally, our OpenCode guide covers configuration, and the output-token limit is the first setting to check.
Why does cost change by up to 3x?
Because the harness decides what is resent on every call. The paper measures the first-call input size for GLM-5.3-Flash:
| Harness | First call (total input tokens) | Preamble only | Growth per step (GLM / DeepSeek) |
|---|---|---|---|
| Claude Code | 17,134 | 16,581 | 937 / 1,308 |
| OpenCode | 7,579 | 7,025 | 681 / 1,116 |
| mini-SWE-agent | 1,382 | 829 | 648 / 1,037 |
Gauge needle rising to show coding agent harness cost per task climbing with step count
Tool outputs are about the same size across harnesses (mean 1,237 characters per call for Claude Code, 1,186 for mini-SWE-agent, 1,103 for OpenCode), so the gap comes from the preamble and from how many steps the loop takes. The bill is, in the authors' framing, set at the first call and scaled by step count.
For GLM-5.3-Flash, one trial cost 1.81 yuan in Claude Code, 0.97 in mini-SWE-agent and 0.56 in OpenCode, with mean steps of 88.1, 80.2 and 55.1. With DeepSeek-V4-Flash, Claude Code cost 0.98 yuan against 1.02 for mini-SWE-agent, because the minimal harness took 141 steps to Claude Code's 104. So a small prompt is not a guarantee of a low bill: a harness that wanders for more steps can erase the saving.
Does prompt caching make this moot?
No. Between 96 and 99 percent of input tokens were cache hits in every arm. Zhipu prices a cache hit at 0.287 times the miss price and DeepSeek at 0.033 times. Those discounts shrink the invoice but do not reorder the harnesses: repricing each model's tokens under the other vendor's list kept the same ranking. Caching helps with the repeated preamble, but a bigger preamble still costs more.
What about thinking mode?
The study includes a small ablation. Turning thinking off barely changed Qwen3.6-35B-A3B in Claude Code (+2.2 points) or OpenCode (+3.4), but dropped it 11.1 points in mini-SWE-agent (p = 0.04). HY4-Preview in Claude Code fell from 51% to 40% without thinking, though with one run per side that result was not significant. The authors say they cannot identify the mechanism behind the interaction. The practical reading is that thinking matters more for the minimal harness, which has fewer tools and no compaction to lean on.
The hidden finding: most open-model runs looked up the fix when the web was on
A side result deserves its own warning for anyone running agent evals. With web access enabled, 312 of 439 DeepSeek and Qwen trials fetched the upstream fix from GitHub. Those trials passed at 96 percent, versus 71 percent for trials that never touched the network. Claude models did this in 4 of 445 trials. The authors discarded all web-enabled results and enforced offline operation at the network layer, not in the prompt. If you are comparing agents on SWE-bench-style tasks, check your own network policy; a prompt that says "do not look it up" is not a control.
How big does a benchmark need to be?
The paper's power analysis is the most reusable part. At the observed disagreement rate of roughly 14 percent, 45 tasks detect a 12.9-point gap only half the time, and cannot detect anything at the conventional 80 percent power. The 447-task pool resolves about 5 points at 80 percent power. That is still coarser than many claimed harness gains. A second check on Terminal-Bench 2.1 (Qwen3.6, 50 tasks scored) produced 19 passes for mini-SWE-agent and 20 each for OpenCode and Claude Code, with no detectable ordering.
What should you do with this?
- Do not pick a harness on a 2-point leaderboard gap. Ask for a rerun of the same configuration beside the comparison.
- Measure first-call tokens. Log the input tokens of the very first request in your harness. If a custom system prompt or a pile of skills and rules loads on every call, that is your baseline cost.
- Count steps, not just prompt size. A tight loop that finishes in fewer steps often beats a lighter prompt.
- Cap output carefully. Check the output-token limit and what happens on a cutoff: re-prompt and continue, or stop.
- Keep evals offline. Enforce it at the network layer.
- Use the commands that trim context. Our Claude Code commands reference lists the compaction and context tools, and the loop engineering guide covers keeping loops short.
Limits of the study
The authors are explicit about them. The main results come from one benchmark. Claude Opus 5 ran only in Claude Code, so the three-harness comparison covers the other five models and not the strongest frontier one; in the hard45 set Opus 5 solved 36 of 45 (80 percent), far above the open models at 24 to 58 percent, which shows the model dominates. Opus runs used a subscription, so their cost column is a token-weighted index rather than real spend. The OpenCode time limit was not symmetric with Claude Code's. And the full trajectories needed for some analysis scripts are promised in a separate release. The repository ships a reproduce.sh script that rebuilds the tables from the 8,032-row trial table in under a minute on a laptop CPU, with no GPU or network.
Why it matters
The industry conversation about agent harnesses has been dominated by scaffolding: more tools, longer prompts, smarter compaction. This paper does not say harnesses are useless. It says that for a fixed model on SWE-bench Verified, the measured accuracy return on added scaffolding is small and often invisible, while the cost return is large and predictable. That favors teams that treat the harness as a cost lever and the model as the quality lever. It also hands reviewers a simple question to ask of any harness claim: what happens when you run it twice?
Related reading
- Agent Lightning v1.0: train agents with the harness you ship
- Self-HPO: Microsoft Research's agent that rewrites its own harness
- AI agent teams cost up to 5.1x more than one agent
- OpenCode: open-source AI coding agent guide
- Claude Code commands: complete reference
- ColGrep-Agent-2B: a local model that cut Claude Code tokens 34%
- How self-evolving coding agents learn
Figures are from the paper and repository as of October 24, 2026; versions, prices and repository details may change.