Hugging Face's harness-hub organization has published a third harness adapter for Qwen3.8-27B, this time for Nous Research's Hermes Agent. The model, harness-hub/Qwen3.8-27B-hermes-adapted, is an Apache-2.0 LoRA that lifts Terminal-Bench 2.1 from 44.7% to 58.8% under Hermes Agent with thinking off. It follows the Pi and OpenCode adapters released earlier this week.
The headline is not only the score. It is that a small adapter lets a 27B open-weight model match its own thinking mode while writing 3.5 to 4.4 times fewer tokens. The authors also publish error bars and the cases where the gain is weak, which makes this a rare agent-benchmark release you can reason about.
TL;DR: results, limits and setup
| Question | Answer from the model card |
|---|---|
| What is it? | LoRA on Qwen/Qwen3.8-27B, RL-trained inside Hermes Agent. |
| License | Apache-2.0. |
| SWE-bench Pro (hard 51) | 29.4% to 43.1% (+13.7 points). 10 tasks newly solved, 3 lost, p = 0.092. |
| Terminal-Bench 2.1 | 44.7% to 58.8% (+14.1 points). 19 newly solved, 7 lost, p = 0.029. |
| Versus base model thinking | 43.1% vs 45.1% on SWE-bench Pro, 58.8% vs 58.8% on Terminal-Bench 2.1. |
| Token use | 3.5 to 4.4 times fewer generated tokens per task. |
| Recommended setting | Keep thinking off. |
| Caveat | One attempt per task, 51 and 85 tasks, text only. |
What is a harness adapter?
An agent is a model plus a harness: the tool schemas, the compaction rules, the permission flow and the prompts that wrap the model. Different harnesses behave differently. A model that is good in one can stumble in another.
Harness-hub's bet is to keep the harness fixed and adapt the model. It trains a small LoRA per harness with reinforcement learning, then swaps the adapter at runtime. We covered the first release in our write-up of the Qwen3.8-27B OpenCode adapter, which reported 51% on SWE-bench Pro. The Hermes adapter is the third, after Pi and OpenCode, and it comes from the same training setup, which the org says uses async GRPO inside Harbor-style environments.
A rounded shape held inside a lightweight scaffold frame, showing how the Hermes Agent harness wraps the Qwen3.8-27B model
The numbers, read carefully
The card compares four runs on the tasks all of them graded: the base model, the base model thinking, the adapter, and the adapter thinking.
| Benchmark | Qwen3.8-27B | Base, thinking | Hermes adapter | Adapter, thinking |
|---|---|---|---|---|
| SWE-bench Pro (hard 51) | 29.4% | 45.1% | 43.1% | 51.0% |
| Terminal-Bench 2.1 | 44.7% | 58.8% | 58.8% | 57.6% |
Three readings matter.
- The adapter closes the gap to thinking. With thinking off it lands within two points of the thinking base model on SWE-bench Pro and ties it on Terminal-Bench. That is the claim that matters for cost.
- Thinking on top of the adapter does not clearly help. On Terminal-Bench 2.1 it scores 57.6%, slightly below the adapter alone. On SWE-bench Pro it reaches 51.0%, but the card notes thinking runs were limited by context.
- The sample is small. Hard 51 means 51 tasks. A 13.3-point interval on a 13.7-point gain is wide. Terminal-Bench 2.1 has a better p-value, but 85 tasks is still a small set.
The authors are direct about the setup. Results are one attempt per task, a 100-step budget, a 131,072-token context window, text only, and sampling at temperature 0.8 with top-p 1.0, which matches training. They also say these numbers are not comparable with the ones on the Qwen3.8-27B card.
Less work per task, not just better scores
The efficiency table is where the adapter looks strongest. On tasks both models solved, medians per task fall like this:
| Measure | SWE-bench Pro (adapter / base) | Terminal-Bench 2.1 (adapter / base) |
|---|---|---|
| Turns | 22 / 59 | 8 / 21 |
| Tool calls | 28 / 63 | 7 / 23 |
| Generated tokens | 4.1k / 8.5k | 1.6k / 3.5k |
Against the base model with thinking, the token gap is larger: 5.5k vs 22.9k on SWE-bench Pro and 2.1k vs 7.4k on Terminal-Bench 2.1. The card says most of that difference is reasoning, and the adapter gets there without it.
The card includes a worked example from SWE-bench Pro, a qutebrowser task about leaving percent-encoded parts of a URL intact when incrementing a number. The base model probes Qt's encoding modes for dozens of calls, breaks the test file with a bad patch and runs out of budget. The adapter fixes the regex and the path setter, checks the scenarios and passes in 35 tool calls. One example is not proof, but it shows what "fewer, more targeted actions" looks like.
Why keep thinking off?
This is the most practical advice on the card. Under Hermes Agent, thinking mode sends each turn's reasoning back to the model. Long sessions then fill the 131,072-token window. Many failed thinking-on attempts, for the base model and for the adapter, ended this way. A session that overflowed was scored as a failure for every model alike, and no context compaction was configured.
So the thinking-on rows understate what thinking could do with a compaction setup. The adapter's advantage is partly that it avoids the problem: it reaches thinking-level accuracy without carrying the reasoning tokens.
How to try it
The adapter is a PEFT LoRA on Qwen/Qwen3.8-27B. The practical steps follow the card's description.
- Download the base model Qwen/Qwen3.8-27B and the adapter from the harness-hub org.
- Load the adapter with PEFT, or merge it. The card says the merged checkpoint is about 54 GB in BF16, so plan memory accordingly or quantize it.
- Point Hermes Agent at the served model and keep thinking off.
- Keep the evaluation conditions in mind: a 131,072-token window, no compaction by default, text only.
If you are new to Hermes Agent, start with our Hermes Agent remote VPS and Telegram guide, the Hermes vs OpenClaw comparison and 10 things you can do with Hermes Agent.
How the three adapters compare
Harness-hub has now shipped adapters for three harnesses on the same base model. The numbers below are each card's own and use different harnesses, budgets and task sets, so they are a map of what was reported, not a ranking.
| Adapter | Harness | Headline claim | Where to read it |
|---|---|---|---|
| Pi | Pi coding agent | SWE-bench Pro hard 51 from 39.2% to 64.7%, Terminal-Bench 2.1 from 41.0% to 56.6% | Pi model card |
| OpenCode | OpenCode | About 51% on SWE-bench Pro, with 6 to 8 times fewer tokens | Our OpenCode write-up |
| Hermes | Hermes Agent | 58.8% on Terminal-Bench 2.1, 43.1% on SWE-bench Pro hard 51, 3.5 to 4.4 times fewer tokens | Hermes model card |
What stays constant is the recipe: pick a harness, run reinforcement learning rollouts inside it, keep the reward honest, and ship a small adapter. What changes is the headline, because each harness exposes different tools. That is the argument for adapters over one general fine-tune. A model tuned on Hermes tool names may waste calls on OpenCode's, and the reverse.
Questions people ask about it
Is this a new model? No. It is a LoRA on top of Alibaba's Qwen3.8-27B. You still need the base weights.
Does it work outside Hermes Agent? The card only reports results under Hermes Agent. Use the Pi or OpenCode adapter for those harnesses, and test before assuming transfer.
Is 27B enough for real work? The card's own numbers say a 27B model can solve about 59% of Terminal-Bench 2.1 tasks and about 43% of a hard SWE-bench Pro subset under this setup. That is promising for a self-hosted model and still short of what a well-prompted frontier model delivers in a long session, so match the tool to the job.
What does the token saving mean in money? Fewer generated tokens means less compute on your own GPU, or a smaller bill on a hosted endpoint. The card reports medians per task, not dollars, so measure your own cost.
Why p = 0.092 matters. A p-value above 0.05 on SWE-bench Pro means the gain could be chance on 51 tasks. The Terminal-Bench result clears the usual bar. Together they say the direction is likely right and the size is uncertain.
What this means for builders and teams
The idea that matters is cost. Reasoning tokens are the expensive part of agent loops. If a LoRA can give you thinking-level results with a quarter to a third of the tokens, a self-hosted 27B model becomes a cheaper engine for terminal and repo tasks. Two cautions apply.
- The adapter is harness-specific. It was trained inside Hermes Agent. If you use a different harness, use that harness's adapter or expect smaller gains. The org's point is that you swap adapters per harness.
- Benchmarks are not your repo. Terminal-Bench measures terminal tasks and SWE-bench Pro measures real GitHub issues. Your codebase will differ. Run it on a handful of your own tasks before moving production work.
For a general picture of the benchmark, see our Terminal-Bench 2.0 explainer and the write-up on agent harness engineering, which makes the same argument from the harness side. The Qwen3.8-27B family has other community work too, such as the DSpine-b16 drafter for speculative decoding.
What is claimed and what is verified
| Claim | Source | Status |
|---|---|---|
| 58.8% on Terminal-Bench 2.1 with thinking off | Model card | Authors' own evaluation, paired, p = 0.029 |
| 43.1% on SWE-bench Pro hard 51 | Model card | Authors' own evaluation, p = 0.092, wide interval |
| 3.5 to 4.4 times fewer tokens | Model card | Authors' own measurement |
| A full reproducible recipe is coming | Model card says it will be in the release blog post | Not yet published |
No independent reproduction was available at the time of writing. The authors say the full recipe will come in a release blog post, which would let others check the method.
Bottom line
Qwen3.8-27B-hermes-adapted is a credible, openly licensed way to make a mid-size open model behave more efficiently inside one popular agent. The gain is large in the card's own numbers, the sample is small, and the honest framing in the card helps. If you run Hermes Agent on your own hardware, it is worth a test this week with thinking off. If you do not, the idea behind it, one small adapter per harness, is the part to watch.
Scores and settings reflect the Hugging Face model card as of November 19, 2026.
