Microsoft ThinkingBox: A Benchmark That Checks What Agents Actually Changed

A support agent tells you it added a quiet-room preference to your hotel booking. The transcript reads like a clean success. But the booking's special_requests field is still empty — the agent confirmed work it never actually did, and a grader that only reads the final chat response would pass the run anyway. You find out the gap exists at check-in.

Microsoft built ThinkingBox to catch exactly that class of failure, and it's worth covering because it targets a gap most agent benchmarks still have: grading the chat transcript instead of the world the agent left behind.

TL;DR

The problem: a transcript can lie about what happened

Final-response grading only works if the benchmark controls the starting state and every way that state can change. Microsoft's engineering team, led by principal machine learning engineer Liang-Chun Tsai, designed ThinkingBox around four requirements a task needs to reproduce reliably:

1. A known initial state and expected outcome — the task defines exactly what records exist before the run and what they should look like after.
2. A controlled tool surface — the agent can only change state through the tools it's given; the evaluator keeps a separate view of the ground truth.
3. A responsive simulated user — the opening request deliberately withholds details (like a booking reference), forcing the agent to ask rather than guess.
4. A clean, isolated environment per attempt — every run starts from the same baseline with no cross-run contamination.

The quiet-room task, in Microsoft's own example, starts with an empty special_requests list and an open support ticket, and only passes if the preference appears on the booking and the ticket is marked solved with the correct resolution code. The transcript explains how the agent got there; a set of executable assertions over the actual database state decides whether it counts.

def test_quiet_room_request(x: TestContext, judge: Judge):
    assert (
        x.effects["external_booking_v1"]["result_db_hash"]
        == x.effects["external_booking_v1"]["golden_db_hash"]
    )
    assert judge.text_yesno(
        x.response,
        "Does the response say the preference is subject to availability?",
    )

Notably, the assertions don't check which tools the agent called or in what order — only the resulting state. A test that hard-codes a tool-call sequence rejects any agent that finds a valid alternate path (skipping a redundant lookup, retrying after a transient error). ThinkingBox checks outcomes; deterministic assertions cover state, and a narrow LLM-judge question only covers phrasing that can't be reduced to an exact match, like whether the agent disclosed availability isn't guaranteed.

How the runtime enforces isolation

ThinkingBox splits into two repositories: thinkingbox holds the tb CLI, the agent/simulated-user loop, an MCP Session Proxy, and the evaluation harness; thinkingbox-data holds the MCP servers, scenario definitions, and synthetic records for each domain.

The Session Proxy is what makes parallel, repeatable isolation possible. It's not a public-facing tool — it wraps three reserved MCP calls the agent never sees:

• __reserved__init — starts one process per MCP server (not per tool) and seeds it with the task's initial records
• __reserved__geteffects — returns the final state after the run, for the evaluator to compare against the golden database hash
• __reserved__teardown — kills the session's MCP processes, even on a failed run

Each tb infer attempt gets its own session ID and freshly initialized processes, so concurrent trials don't leak state into each other — a real constraint for running the same task 20 times, which is exactly what the benchmark does.

Where ThinkingBox sits against prior agent benchmarks

Microsoft's own comparison table checks each prior benchmark against the four properties above:

Sierra's tau-bench family — covered on explainx.ai via Celeris-1 Magnus's tau3-bench results — is the closest prior work satisfying all four properties. ThinkingBox's contribution is scale and domain breadth: 507 tasks across retail (98), travel and hospitality (104), auto insurance (100), neobank internal IT (104), and consulting IT/HR (101), plus the reusable MCP-based lifecycle around them.

The leaderboard: GPT-5.4 leads, but domain variance is large

Microsoft ran all 12 evaluated models 20 times per task and reported pass@1 averaged across the full 507-task set:

GPT-5.4 is the only model above 50% in every single domain. GPT-5.2, despite a similar-generation aggregate score to DeepSeek-V4-Pro, swings from 70.20% on retail down to 22.40% on auto insurance — a domain-specific policy environment it clearly handles worse. Claude Opus 4.6 shows the same pattern: 74.90% on retail against 14.65% on auto insurance. A single average score hides which domain a given model will actually fail in production.

Capability and reliability are different numbers

The more useful finding sits underneath the leaderboard. Because Microsoft ran every task 20 times, they could separate three distinct questions instead of one:

• pass@1 — how often a single attempt succeeds
• pass@20 — whether at least one of 20 attempts succeeded (can retries find a working path?)
• pass^20 — whether all 20 attempts succeeded (does the agent repeat success reliably?)

GPT-5.4's pass@20 reaches 91.12% — it can find a successful trajectory on almost every task given enough tries — but its pass^20 is only 25.25%. It never once succeeded on 45 of the 507 tasks, and passed all 20 attempts on just 128. Claude Opus 4.6 (37.91% pass@1) and Kimi-K2.6 (37.66% pass@1) look nearly identical on the headline number, but Opus reaches 70.02% pass@20 against Kimi's 84.22% — Kimi finds a working path on more tasks — while Opus's pass^20 of 13.81% beats Kimi's 3.16% — Opus is the one that repeats success once it finds it.

That distinction matters for anything beyond a demo: retries can raise the odds of one good outcome without making an agent dependable enough to remove a human reviewer from the loop.

Where the failures actually happen

Microsoft's team also read the failed traces and categorized the root cause:

Nearly four out of five failures happen during tool execution, not in the final response. Most agents reach the right workflow; they just don't interpret a failed tool call, replan around it, or verify state before declaring success. That's a specific, actionable target for anyone building agent harnesses — the fix is in how the loop handles a bad tool response, not in prompt-tuning the final message.

Try it yourself

Both repositories are open source. The runtime is at github.com/microsoft/thinkingbox; the benchmark tasks, synthetic records, and MCP servers are at github.com/microsoft/thinkingbox-data. The ThinkingBox-Bench v1.0 release documents reproducing the 507-task run and computing pass@1, pass@20, and pass^20 with tb agg. The main repo's README walks through building a custom MCP server, scenario, and test case if you want to grade your own agent against domain-specific side effects instead of a chat transcript.

Benchmark scores, repository structure, and release details reflect Microsoft's ThinkingBox publication as of August 19, 2026 — check the linked repositories for current numbers before relying on any specific figure.

Related on explainx.ai

• Complete guide to AI benchmarks
• How to read AI benchmarks
• Terminal-Bench 2.0: AI agent benchmark evaluation
• Celeris-1 Magnus and tau3-bench agentic evaluation
• State machines for enterprise agent test environments
• What is an agent harness? Complete guide
• AI coding agent evals on real repos
• Goodhart's Law and AI benchmark contamination

Official sources: ThinkingBox runtime on GitHub, ThinkingBox-Bench data on GitHub, Microsoft's ThinkingBox announcement on Command Line.