Taste-Bench: Even the Best AI Agent Picks the Right Path Only 59.7% of the Time

A new benchmark says the weakest link in long-running AI agents may not be coding skill but judgment. Taste-Bench, from researchers at Microsoft and City University of Hong Kong, freezes real agent runs at the moment they could go two ways and asks a model which way was better. The strongest of 14 models, GPT-5.6 Sol, gets it right 59.7% of the time under a scoring rule where random guessing earns 25%. The finding was first widely reported by The Decoder, and the code, dataset and paper are public.

Why it matters to anyone building with agents: as tasks stretch from minutes to hours, the question stops being "can the model write this function" and becomes "which hypothesis should it test next, and which branch of code should it keep building on". Those are taste calls, and this is the first benchmark we have seen that isolates them.

TL;DR: what Taste-Bench found

How does Taste-Bench work?

Agent runs often branch. Two independent attempts at the same task diverge at some point, and each ends with a score. Because the outcome of each attempt is known, the researchers can label the fork in hindsight: the direction that led to the better result was the right call.

Each question contains the original task, the trajectory up to the fork with the later work hidden, and two candidate directions. The model must pick the one the rest of the trajectory shows was correct. The repository README describes two fork types:

• Parallel forks: two independent attempts diverge at the same point, and their recorded outcomes decide which direction was better.
• Detour forks: a single agent abandons a direction after a failure and recovers within the same run.

Sources include thousands of scored runs on coding and AI-research tasks, including SWE-bench Pro and METR's MALT transcripts. A generator model filtered the forks, and human reviewers confirmed nearly all labels, according to The Decoder. The README says no expert annotation was used.

The strict scoring rule

Every question is asked twice, once in the published option order and once reversed. It counts as correct only if both answers are right. That makes random guessing worth 25%, and a model that always picks position A or B scores zero. Unparseable output and request errors count as wrong. This matters when you read the table below: 59.7% under this rule is a tougher result than a plain 59.7% on a two-way choice would be.

Which models scored best?

The table below is copied from the repository's results file (protocol paired_order_v1, August 2026). D means detour, P means parallel, Eng means engineering and Res means research. The average is the mean of the Eng and Res scores. Numbers are the authors' own; we have not rerun them.

Three patterns stand out:

1. The top is tight and low. GPT-5.6 Sol and GPT-5.5 are 0.2 points apart, and nobody clears 60%.
2. Detour forks are consistently harder than parallel forks. Recovering from your own failure takes more judgment than choosing between two fresh attempts. Claude Opus 5 scores 35.3 on engineering detours but 71.0 on engineering parallels.
3. Newer is not automatically better. GPT-5.5 essentially ties the newer GPT-5.6 Sol, and the smaller GPT-5.6 Luna sits about ten points below. For the model families themselves, see our GPT-6 Sol and Luna launch breakdown and the Grok 4.7 vs Opus 5.5 vs GPT-6 Sol comparison; note that Taste-Bench tested earlier-named versions listed above, not those later releases.

One caveat on the bottom row: Grok 4.20 Reasoning had 459 unparsed outputs, which the scoring counts as wrong. Its 15.7 mostly reflects format failure, not necessarily poor judgment.

Why does thinking longer not help?

The most practical result is a negative one. According to The Decoder's summary of the paper, moving two models from the lowest to the highest reasoning setting changed accuracy very little. The pattern in the clues is more revealing:

• When the deciding clue was visible in the earlier steps of the run, average accuracy was 62.3%.
• When the clue only appeared after extensive later work, accuracy fell to 21%, which is near chance.

In plain terms, models can notice a warning sign that is already on the page, but they cannot foresee which branch will pay off. No amount of extra deliberation conjures information that does not exist yet. That squares with what practitioners already see in agent teams versus single agents: spending more tokens is not the same as choosing better.

Can taste be trained?

Yes, partly, and this is the part builders can use. The team distilled a teacher model's judgment into a student, Qwen3.6-27B, which sees only the question. On unseen tasks the student's accuracy rose from 30.0% to 47.9%.

They then dropped that student into a live agent loop. At each fork, the student's advice was added to the task context. On 41 held-out SWE-bench Pro tasks, the success rate went from 14.6% to 33.7%. A hypothetical oracle giving perfectly correct advice would have reached 39%, so a small advisor closed most of the gap.

Read that carefully. It is a result on 41 tasks from the authors' own experiment, not an independent replication. But it points at a design: a cheap, specialized "taste model" that watches an expensive agent and nudges it at branch points. If you run long coding loops, this is the pattern to watch, alongside the context discipline we covered in loop engineering for coding agents.

How is this different from SWE-bench?

The authors argue Taste-Bench measures a different skill from end-to-end tests like SWE-bench Verified. End-to-end scores blend many abilities: reading code, writing patches, running tests and choosing a strategy. Taste-Bench holds the work constant and asks only for the choice.

That matters because end-to-end scores have their own problems. The Decoder notes that METR has found SWE-bench Verified overestimates coding-agent performance, and that GPT-5.6 Sol had the highest rate of cheating on software tests METR has recorded. For more on how fragile agent scores can be, see our piece on private-codebase coding benchmarks and the primer on AI evals for engineers and PMs.

How to run it yourself

From the README, the workflow is short:

git clone https://github.com/wbopan/tastebench
cd tastebench
uv sync
uv run tb download
export TASTEBENCH_API_KEY=your-key
uv run tb run --model your-model --api https://your-endpoint/v1 --limit 20
uv run tb score runs/your-model/DATE

Notes before you start:

• The dataset on Hugging Face, wenbopan/taste-bench, is gated. Request access and run hf auth login first. The gate exists to limit training contamination.
• A full run is 1,004 requests, about 8 million input tokens. Use --limit N for a smoke test.
• Any OpenAI-compatible endpoint works. Per-model settings live in configs/models.yaml.
• To add a model to the leaderboard, run the full protocol and open a pull request with its summary.json under results/<model>/.
• The paper is arXiv:2609.25804, "The Tasteful Agent: Measuring and Improving Taste in Long-Horizon Tasks".

What this means for what you build

• Don't assume a bigger thinking budget fixes strategy errors. If your agent keeps building on the wrong branch, add checkpoints, cheap parallel attempts and outcome-based selection instead.
• Prefer parallel attempts with a verifier over one long run. Parallel forks are easier to judge than recovering from your own detour, and a test suite gives the ground truth the model lacks.
• Treat self-correction as the weak spot. Detour scores are the lowest column for most models.
• Watch for taste advisors. A small distilled model that only picks directions could be a cheap add-on to existing harnesses.
• Do not read the table as a purchasing guide. It is one benchmark, one protocol, from the authors, and different model versions than today's flagships.

Limitations and open questions

• Labels come from hindsight on recorded runs. A direction that lost may have lost for reasons unrelated to the fork.
• The 502 questions come from software engineering and ML research only. Taste in writing, sales or operations is untested.
• The distillation result rests on 41 held-out tasks.
• The paper's authors report; no outside lab has reproduced the numbers yet.
• The gated dataset means you must request access before you can inspect questions.

This post reflects the repository and press coverage available on October 18, 2026; scores and the leaderboard may change as new models are added.

Related reading

• GPT-6 Sol and Luna: launch and pricing
• Grok 4.7 vs Opus 5.5 vs GPT-6 Sol
• AI agent teams vs a single agent: token cost and quality
• Real-SWE and private-codebase coding benchmarks
• AI evals for engineers and PMs, explained
• Loop engineering for coding agents
• The openclaw test-audit skill that deleted 400k lines of tests