explainx / blog
Terminal-Bench 2.0 is the industry-standard benchmark for evaluating AI agents on real-world terminal tasks. 89 carefully curated tasks, Harbor framework, and results from GPT-5.5, Claude Opus 4.7, and more.
Jun 12, 2026
ALE is a living benchmark built with 250+ industry experts and 1,490 task instances mapped to the U.S. O*NET occupational taxonomy. Unlike academic tests, it scores agents on long-horizon GUI+CLI work with deterministic evaluators—and frontier systems still fail 97%+ of the hardest tasks.
Jun 10, 2026
Published June 8, 2026, Self-Harness demonstrates how AI agents can autonomously identify weaknesses, propose harness modifications, and validate improvements—turning model-specific failure patterns into concrete executable fixes that boost Terminal-Bench 2.0 pass rates from 40.5% to 61.9%, 23.8% to 38.1%, and 42.9% to 57.1% across three diverse models.
May 2, 2026
AI benchmarking in 2026 has reached a critical inflection point. Traditional benchmarks like MMLU and HellaSwag are saturated above 88% and 95%, while frontier models cluster within statistical noise. This comprehensive guide covers every major benchmark category—from language understanding to agent evaluation—the 37% lab-to-production gap, benchmark gaming vulnerabilities, and what actually matters for production AI systems.
Every spring, the AI benchmarking landscape shifts. What was once challenging becomes saturated. What differentiated frontier models becomes statistical noise. But in May 2025, the Laude Institute, Stanford University, and Snorkel AI released something different: Terminal-Bench 1.0—a benchmark that became, in the words of the creators, a "runaway success," adopted by virtually every frontier lab within months.
Six months later, in November 2025, the team released Terminal-Bench 2.0. This wasn't just an incremental update. It was a proactive response to saturation—raising the bar before models conquered version 1.0, while fixing quality issues the community discovered through intensive usage. The result: 89 carefully curated tasks where frontier models still score below 65%, each task receiving approximately 3 reviewer-hours of human auditing to ensure it's solvable, realistic, and well-specified.
This post does four things: it explains what Terminal-Bench 2.0 actually measures, how it differs from other agent benchmarks like SWE-bench and GAIA, the Harbor framework that powers it, and what the current leaderboard tells us about the state of AI agents in 2026.
Terminal-Bench 2.0 is a carefully curated hard benchmark composed of 89 tasks in computer terminal environments inspired by problems from real workflows. Unlike academic benchmarks that test knowledge recall or isolated code generation, Terminal-Bench measures:
Each task features a unique environment, a human-written solution, and comprehensive tests for verification. Tasks must be completed using only Bash commands through a headless terminal—no GUI, no shortcuts, no structured output templates to lean on.
Every Terminal-Bench 2.0 task consists of:
Scoring is binary and strict: Pass@1 only. Models must pass ALL pytest validation tests to receive credit for a task. A task with 10 tests where 9 pass still scores 0. There are no multiple attempts, no partial credit, no second chances.
The evaluation pipeline is deterministic and transparent:
uv package manageruvx to install pytest and task-specific dependencies with pinned versionsThis approach eliminates the ambiguity and gaming vulnerabilities that plague LLM-judged benchmarks.
Terminal-Bench 2.0 covers diverse domains that reflect real developer and system administrator workflows:
Tasks range from easy to hard:
These difficulty labels are author-estimated for humans and may not reflect agent difficulty—some "easy" tasks trip up frontier models while some "hard" tasks fall to clever tool use.
Terminal-Bench 1.0 launched in May 2025 with 80 tasks and became an instant success. But success brought scrutiny. The community—including the researchers themselves—discovered problems:
As frontier models climbed over 50% success rate on version 1.0, the benchmark risked becoming another saturated metric.
Instead of waiting for saturation, the team raised the bar in November 2025:
Task Quality and Verification
Increased Difficulty
Technical Infrastructure Upgrade
Version 1.0:
Version 2.0:
Specific Improvements
The original Terminal-Bench 1.0 harness worked, but scaling to thousands of evaluations across 16 frontier models and 6 state-of-the-art agents revealed bottlenecks:
Harbor is the answer: an open-source framework for evaluating and optimizing agents in container environments.
Core Components:
harbor run -d [email protected])Supported Agents:
Custom Agent Support: Developers can create custom agents by subclassing BaseInstalledAgent or BaseAgent, receiving the instruction and Docker container, then exploring/manipulating the environment through tool calls (editing files, running Bash commands).
The Evolution:
This infrastructure enabled the 32,155 total trials across models and agents that powered the 2.0 leaderboard.
Sandboxing: Each task runs in isolated Docker containers to prevent cross-contamination and ensure reproducibility.
Contamination Detection: Includes Big-Bench canary string in each repository file to aid in training corpus decontamination. The team acknowledges that private test sets are considered out of scope due to the community investment required, but the canary string provides transparency for model developers.
As of 2026, the Terminal-Bench 2.0 leaderboard shows:
Top Direct Model Scores:
The 65% Ceiling: Despite frontier models approaching human-level performance on many academic benchmarks (MMMU-Pro models within 0.3 points of human experts), they still fail on 18-35% of Terminal-Bench 2.0 tasks—tasks that experienced developers complete routinely.
Top Agent Scores (combining agent scaffolding with frontier models):
The Agent Scaffolding Effect: The same model can perform very differently with different agent implementations. For example, Gemini 2.5 Pro's pass rate improved 17% with Terminus 2 scaffolding over OpenHands—demonstrating that agent design matters significantly.
The benchmark has tested 6 state-of-the-art agents across 16 frontier models with 32,155 total trials:
The 49-point gap between easy and hard tasks affects all models uniformly, suggesting that current agent architectures struggle with similar bottlenecks regardless of underlying model capability.
SWE-bench (Verified: 500 tasks, Pro: 731 tasks):
Terminal-Bench (89 tasks):
Key Difference: SWE-bench measures software-engineering proficiency; Terminal-Bench measures broader operational capabilities and tool-use accuracy. A model can excel at one and struggle at the other—they test distinct skill sets.
GAIA (466 tasks):
Terminal-Bench:
Key Difference: GAIA tests general-assistant reasoning capability; Terminal-Bench tests action execution in live environments. GAIA asks "Can you figure out the answer?"; Terminal-Bench asks "Can you actually make it happen?"
Models show different strengths across benchmarks. In the same week, a model might achieve:
This demonstrates that software-engineering proficiency ≠ general-assistant capability ≠ operational reliability. Each benchmark captures distinct aspects of agent capability.
Important Note: Research from Berkeley RDI has shown that Terminal-Bench, SWE-bench, and GAIA (and other prominent agent benchmarks) can all be exploited to achieve near-perfect scores without solving tasks:
The Terminal-Bench team is aware of these vulnerabilities and continues to improve sandboxing and verification mechanisms, but this highlights an ongoing challenge in creating truly robust evaluation benchmarks.
As the announcement post states: "Current benchmarks either do not measure real-world tasks, or are not sufficiently difficult to meaningfully measure frontier models."
Terminal-Bench 2.0 fills this gap by:
Terminal-Bench 1.0 was a "runaway success"—since its May 2025 launch:
Version 2.0 continues this trajectory while addressing quality concerns and raising difficulty before saturation set in.
Terminal-Bench 2.0 represents a commitment to maintaining the highest quality evaluation infrastructure as AI agent capabilities increase:
While SWE-bench measures software patch correctness and GAIA measures multi-step reasoning with ground-truth answers, Terminal-Bench uniquely evaluates:
This broader scope makes Terminal-Bench particularly valuable for teams building general-purpose agents rather than domain-specific tools.
Frontier models still fail on 18-35% of tasks that humans complete routinely. This is not a transient gap—it reflects fundamental limitations in current agent architectures:
Resolution rate correlates strongly with model capability AND agent orchestration. The 17% improvement Gemini 2.5 Pro saw with better scaffolding demonstrates that:
Terminal-Bench 2.0's tasks are inspired by real workflows, but are they representative of what actually matters in production?
Arguments for relevance:
Arguments for caution:
The 37% gap between lab benchmark scores and real-world deployment performance observed across enterprise agentic AI systems suggests that even Terminal-Bench—among the best available benchmarks—cannot fully predict production readiness.
Version 1.0 saw frontier models climb from 20% (2025) to >50% in six months. Version 2.0 reset the bar, but models are already approaching 73%. If progress continues at this pace:
The team will likely need Terminal-Bench 3.0 within 12-18 months to maintain differentiation at the frontier.
Official Distribution via Harbor:
harbor run -d [email protected]
This command:
Minimum Implementation:
from harbor import BaseInstalledAgent
class MyAgent(BaseInstalledAgent):
def solve_task(self, instruction: str, container):
# Your agent logic here
# Use container.run_bash(command) to execute commands
# Parse outputs and decide next steps
# Return when task complete
pass
Provided Examples:
What a 65% score means:
What to optimize:
For a complete picture of agent capability, use Terminal-Bench alongside:
Each benchmark tests distinct capabilities. High performance on one does not guarantee high performance on others.
Use Terminal-Bench when:
Don't rely solely on Terminal-Bench when:
Research shows that enterprise agentic AI systems exhibit a 37% gap between lab benchmark scores and real-world deployment performance. Terminal-Bench is among the best proxies for production readiness, but:
Recommendation: Use Terminal-Bench for relative comparisons between models and agents, but always validate in your specific production context before deployment.
Based on the 1.0 → 2.0 transition, expect:
Terminal-Bench faces the same challenge as all benchmarks: saturation is inevitable. The question is not "if" but "when" and "how to respond."
Two strategies:
Terminal-Bench currently uses strategy #1. A shift to strategy #2 might be necessary to maintain relevance beyond 2027.
Terminal-Bench 2.0 selected 89 tasks from 229 submissions by 93 contributors through crowdsourcing. This community-driven approach:
Future versions will likely lean more heavily on community contributions, potentially with:
Terminal-Bench 2.0 has earned its place as the de facto standard for AI agent evaluation because it tests what actually matters: Can agents complete real-world operational tasks reliably, recover from errors, and execute multi-step workflows across diverse domains?
The answer in 2026 is: mostly, but not entirely. Frontier models score 65-73% direct, with agent scaffolding pushing top combinations to 81-82%. This leaves a meaningful capability gap that differentiates systems and reveals failure modes invisible to saturated academic benchmarks.
For teams building agents, Terminal-Bench 2.0 provides:
But remember: Terminal-Bench scores are proxies, not guarantees. The 37% lab-to-production gap means you still need to validate in your specific context before trusting an agent with production work.
Read the official resources:
For complementary perspectives on agent evaluation, benchmarking, and production deployment, see our other guides:
Disclosure: This post is editorial commentary on public materials from the Laude Institute, Stanford University, Snorkel AI, and the Terminal-Bench community. For academic or production citations, use the primary paper and official leaderboard data.