Δτ-Bench: The Agent Benchmark Where Tools Keep Changing (Best Score 42.3%)

Most agent benchmarks freeze the world: the tools, their names and their documentation stay put while the model is scored. Real agents do not get that luxury. Δτ-Bench ("Delta Tau Bench"), introduced in the paper Δτ-Bench: Pushing Agents Toward a Moving Capability (arXiv 2610.34238, submitted October 25, 2026), measures what happens when the tools underneath an agent keep changing. The headline number: the strongest evaluated model or harness reaches only 42.3% task success.

The work was trending on Hugging Face's daily papers with 21 upvotes, and the authors, Shuofei Qiao and colleagues (with a University College London affiliation listed on the paper page), released code and data. This post explains what the benchmark does, what the reported results do and do not say, and what builders can take from it.

TL;DR: what you need to know

What problem does Δτ-Bench try to measure?

Tool-using agents are usually graded on a static snapshot. That hides a failure mode every team running agents in production meets sooner or later: tool drift. An API gets a new version, a function is split in two, two workflows are merged, a documentation page goes stale, or a shinier alternative appears that does the same job in fewer calls.

According to the paper, Δτ-Bench models this structural evolution directly. Independently evolving services reshape how tasks can be accomplished: familiar tools may disappear or be replaced, while more efficient alternatives emerge amid overlapping functionality and noisy documentation. The user's goal stays the same. The benchmark asks two things of the agent:

1. Can it still find an efficient path to the goal when the tool landscape shifts?
2. Can it update the procedural knowledge it has stored, such as skills learned from earlier tasks, instead of replaying an obsolete recipe?

The six versions are produced through capability-preserving transformations, meaning the set of achievable outcomes stays constant while the routes to them change. That design isolates adaptation from raw capability.

How is the benchmark built?

From the repository README, the pieces fit together like this:

• Tasks: goal-driven, multi-turn tasks against executable services and databases, so success is checked by state, not by a judge reading a transcript.
• Splits: 1,191 training tasks and 397 test tasks per version, which together account for the 1,588 total.
• Versions: v1 through v6, with a retained base environment as a reference.
• Metrics: task success rate, tool-retrieval precision, recall and F1, and path-efficiency measures (the README lists SPE, IDR and DAR) across version transitions.

If you have used the original tau-bench style of tool-agent-user simulation, the setup will feel familiar: a simulated user pursues a goal and the agent has to operate the tools to deliver it. The new ingredient is that the environment mutates between versions.

What did the results show?

The reported findings are brief, and the summary available at publication time withholds many details, so treat the following as the complete public picture rather than a full results table:

• The best system scores 42.3%. Even the strongest evaluated model or harness fails more than half of the tasks. The page does not name the models or baselines, and the repository does not include a score table.
• Skill learning does not reliably transfer. Early gains from existing skill-learning methods do not consistently carry over as the ecosystem evolves. A skill library that helps in v1 can stop helping, or hurt, by v4.
• Steward holds up better. The authors' Steward system sustains higher success across changes, with lower skill-maintenance overhead, improved recovery from disrupted workflows, and better adoption of more efficient alternatives. No specific Steward numbers appear in the abstract.

One caution: because the paper is a work-in-progress first version, and the repository says the released dataset includes corrections, numbers may move. Read the PDF before quoting any specific figure.

How does Steward work?

Per the README, Steward manages tools and procedural knowledge together rather than treating them as separate problems. It maintains three kinds of skills (planning, atomic and tool skills), tracks the dependencies between them, and provisions the agent through initial retrieval plus on-demand search while a task runs.

Training builds the skill libraries version by version. In v1, skills are bootstrapped from training trajectories. In v2 through v6, they are evolved for each new ecosystem. Importantly, training does not fine-tune model weights. All adaptation lives in a maintained library of skills and indices, which makes the approach portable across models and cheap to update.

This is close in spirit to how many builders already use agent skills, and to the idea in WikiSkill of compiling agent experience into a persistent store that can evolve. The difference is that Δτ-Bench gives you a way to test whether such a store survives change.

How to try it

The README describes four steps:

1. Create a Python 3.10 or newer virtual environment and install the package.
2. Download the dataset from Hugging Face into data/, set DTAU_BENCH_DATA_ROOT, and run the validate-data command.
3. Copy .env.example to .env and fill in three endpoints: the agent model, the simulated-user model and an embedding model. Load the file in each shell.
4. Run the model-only baseline or Steward. Both build tool indices, run rollouts and generate reports.

Expect to pay for model calls: each task is multi-turn, the user is simulated by a second model, and Steward adds retrieval. The repository does not publish cost estimates, so budget with a small slice of tasks first.

Why does this matter for people building agents?

Three practical lessons follow even before you run anything.

Treat skills as versioned artifacts. If your agent keeps a library of learned procedures, pair it with tests that run when tool definitions change. The benchmark's finding that gains do not reliably carry over is a warning against "learn once, trust forever."

Measure efficiency, not only success. Δτ-Bench scores path efficiency and the adoption of better alternatives. An agent that keeps using a deprecated route still passes the task but wastes calls and money, which matters given how token costs have been moving.

Expect noisy documentation. Overlapping tools and messy docs are part of the benchmark by design. That mirrors the real MCP landscape, where servers from different vendors describe similar capabilities in different ways. Our earlier piece on Agent-Reflex covers a related approach to recovering from broken tools.

How could you drift-test your own agent?

The paper does not prescribe this, but the benchmark's design suggests a cheap version you can run on your own stack. Treat it as our reading, not a method from the authors.

1. Freeze a task set. Pick 20 to 50 real tasks your agent handles today, each with a state-based pass or fail check.
2. Mutate one tool at a time. Rename a function, split an endpoint in two, or hide a tool and expose a near-duplicate. Keep the achievable outcome identical.
3. Re-run with and without your skill library. If the library lowers success after a change, it is stale and needs maintenance rather than more training data.
4. Count calls. Record the number of tool calls per task before and after. A rise with unchanged success is the efficiency loss the benchmark's path metrics are meant to capture.
5. Repeat per release. Wire the run into the same pipeline that ships tool updates, so drift shows up as a failing check instead of a support ticket.

Even a rough version of this tells you whether your agent is adapting or merely replaying.

How does it compare with other agent evaluations?

Static tool-use suites ask whether an agent can use the tools it is given. Δτ-Bench asks whether it can keep doing so after the tools change, which is closer to what teams see after a quarter in production. It is complementary to long-horizon tests such as the Continual Learning Exam, where the best agent also completed under half the levels, and to harness-focused workflows like loop engineering. Different benchmarks, similar message: the ceiling for agents that must adapt is still low.

What are the limits?

• Work in progress. The paper is version 1, and the page has no limitations section.
• Unnamed baselines. The available summary does not say which models produced 42.3%, so you cannot yet map the score to a product.
• Synthetic drift. The ecosystem changes are generated by capability-preserving transformations. Real drift can also break capability, change semantics silently or add rate limits.
• Simulated users. Like other tool-agent-user benchmarks, the user is itself a model, which adds variance.

None of that undermines the idea. It means the right use today is as a stress test and a vocabulary for tool drift, not as a leaderboard to rank vendors.

Related reading

• Agent skills: the complete guide
• WikiSkill: persistent wiki for agent skill evolution
• Agent-Reflex: recovering from broken tools
• What is MCP?
• Continual Learning Exam for agents
• Loop engineering for coding agents

Details are accurate as of October 27, 2026, based on the arXiv listing and the project repository. The paper is a first version and may change.