explainx / blog
Self-Harness introduces a new paradigm where LLM-based agents autonomously improve their own harnesses without human engineers or stronger external models, achieving 15-52% performance gains on Terminal-Bench 2.0.
Jun 17, 2026
A harness wraps your AI model. A self-harness lets the model improve that wrapper on its own. Here is how the weakness-mining, proposal, and validation loop works — and why it consistently produces 15–52% benchmark gains without touching the base model.
May 2, 2026
Terminal-Bench 2.0 has become the de facto standard for AI agent evaluation since May 2025—used by virtually every frontier lab. This deep dive covers the 89-task benchmark, its evolution from version 1.0, the Harbor framework powering it, and why frontier models still struggle below 65% accuracy on tasks humans complete routinely.
Jun 29, 2026
Most teams conflate prompt writing with context design, loop orchestration, and harness code. They are four layers of the same stack. Here is how they nest, what breaks when you skip one, and which layer to fix when agents fail.
TL;DR: Published June 8, 2026 on arXiv, "Self-Harness: Harnesses That Improve Themselves" introduces a paradigm where LLM-based agents autonomously optimize their own operating frameworks without human engineers or stronger external models. Using a three-stage loop (Weakness Mining, Harness Proposal, Proposal Validation), Self-Harness achieved consistent performance improvements on Terminal-Bench 2.0: MiniMax M2.5 improved from 40.5% to 61.9% (+52.6%), Qwen3.5-35B-A3B from 23.8% to 38.1% (+60.1%), and GLM-5 from 42.9% to 57.1% (+33.1%)—demonstrating that agents can effectively turn model-specific weaknesses into concrete, executable harness improvements.
The performance of LLM-based agents is jointly shaped by two critical factors:
While much attention focuses on improving base models, recent evidence shows that harness engineering can yield 10-15 point improvements on benchmarks while keeping the base model fixed.
Current State:
Why This Doesn't Scale:
graph TD
A[New Model Released] --> B[Human Engineers Analyze]
B --> C[Design Model-Specific Harness]
C --> D[Manual Testing & Iteration]
D --> E{Performance OK?}
E -->|No| B
E -->|Yes| F[Deploy]
G[Another Model Released] --> B
style B fill:#ff6b6b
style C fill:#ff6b6b
style D fill:#ff6b6b
The Bottleneck: Human engineers can't keep pace with model diversity and evolution. Each new model family (GPT, Claude, Gemini, Qwen, GLM, MiniMax, etc.) exhibits unique behaviors requiring custom harness design.
Core Innovation: What if the agent could improve its own harness, without relying on human engineers or stronger external models?
Definition: Self-Harness is an iterative framework where an LLM-based agent autonomously:
Key Insight: Instead of human experts manually engineering model-specific fixes, the model itself discovers and implements what it needs to succeed.
graph LR
A[Execution Traces] --> B[Stage 1: Weakness Mining]
B --> C[Identified Failure Patterns]
C --> D[Stage 2: Harness Proposal]
D --> E[Candidate Harness Modifications]
E --> F[Stage 3: Proposal Validation]
F --> G{Regression Tests Pass?}
G -->|Yes| H[Accept Changes]
G -->|No| I[Reject Changes]
H --> J[Updated Harness]
I --> D
J --> K[Run Benchmark Tasks]
K --> A
Purpose: Identify model-specific failure patterns from execution traces.
Process:
Collect Execution Traces — Run the agent on benchmark tasks, capturing:
Pattern Analysis — The agent analyzes its own traces to discover:
Failure Categorization — Weaknesses are grouped by type:
Example Weakness Discovery:
Weakness ID: W-042
Pattern: Agent frequently fails git operations by forgetting to configure user.name
Frequency: 12 failures across 89 tasks
Impact: Blocks commit-related tasks
Category: Tool prerequisite missing
Purpose: Generate diverse yet minimal harness modifications tied to discovered weaknesses.
Design Principles:
Proposal Types:
1. System Prompt Modifications
# Before
You are an AI agent with access to terminal commands.
# After (Self-Harness Proposal)
You are an AI agent with access to terminal commands.
+ Before running git commit, always verify git user.name and user.email are configured.
+ If not set, configure them using: git config user.name "Agent" && git config user.email "agent@localhost"
2. Tool Wrapper Additions
# Self-Harness proposes wrapping git commands
def execute_git_command(cmd):
# Ensure git is configured before any commit operation
if "commit" in cmd:
check_git_config()
return subprocess.run(cmd, shell=True)
3. Validation Step Injection
# Self-Harness proposes adding verification after file operations
def create_file(path, content):
write_file(path, content)
# Validate file was created successfully
if not os.path.exists(path):
raise FileNotFoundError(f"Failed to create {path}")
# Validate content matches
if read_file(path) != content:
raise ValueError("File content mismatch")
4. Planning Template Updates
# Before
Plan: {steps}
# After (Self-Harness Proposal)
Plan:
+ 1. Verify prerequisites (dependencies, configs, permissions)
{steps}
+ N+1. Verify expected outcomes
+ N+2. Clean up temporary resources
Diversity Mechanism: For each weakness, Self-Harness generates 3-5 candidate proposals using different approaches:
Purpose: Accept candidate edits only after regression testing to prevent breaking existing capabilities.
Validation Pipeline:
1. Held-Out Test Set
Training Set: 70% of Terminal-Bench 2.0 tasks (62 tasks)
Validation Set: 30% held-out tasks (27 tasks)
2. Regression Testing
def validate_proposal(current_harness, proposed_harness, tasks):
baseline_results = run_benchmark(current_harness, tasks)
proposal_results = run_benchmark(proposed_harness, tasks)
# Accept only if:
# 1. No regression on previously passing tasks
# 2. Net improvement in pass rate
return (
no_regression(baseline_results, proposal_results) and
net_improvement(baseline_results, proposal_results)
)
3. Acceptance Criteria
A proposal is accepted if:
4. Iterative Application
Once validated, the proposal is:
Safety Mechanism:
# Only minimal, targeted changes are accepted
if change_diff_lines > MAX_CHANGE_SIZE:
reject_proposal("Too large, split into smaller changes")
The paper evaluated Self-Harness on Terminal-Bench 2.0, the industry-standard benchmark for AI agent evaluation comprising 89 carefully curated tasks across diverse domains.
Base Models Tested:
Why These Models?
Minimal Initial Harness:
| Model | Initial Pass Rate | Final Pass Rate | Absolute Gain | Relative Gain |
|---|---|---|---|---|
| MiniMax M2.5 | 40.5% | 61.9% | +21.4% | +52.8% |
| Qwen3.5-35B-A3B | 23.8% | 38.1% | +14.3% | +60.1% |
| GLM-5 | 42.9% | 57.1% | +14.2% | +33.1% |
Key Findings:
1. Consistent Improvements Across All Models
2. Model-Specific Harness Modifications
3. Non-Generic Improvements
4. Compound Benefits
MiniMax M2.5 Improvement Curve:
Iteration 0 (Baseline): 40.5%
Iteration 1: 45.2% (+4.7%)
Iteration 2: 51.8% (+6.6%)
Iteration 3: 56.3% (+4.5%)
Iteration 4: 59.1% (+2.8%)
Iteration 5: 61.2% (+2.1%)
Iteration 6: 61.9% (+0.7%)
Iteration 7: 61.9% (+0.0%) [Converged]
Convergence Behavior:
The paper provides detailed examples of discovered weaknesses and resulting harness modifications.
Weakness Mining Discovery:
Pattern: 8 failures on tasks requiring git commits
Root Cause: Missing git user.name and user.email configuration
Example Traces:
- Task 23: "Create repo and commit changes" → FAIL (git commit rejected)
- Task 45: "Initialize project with git" → FAIL (identity not configured)
Harness Proposal Generated:
System Prompt Addition:
+ Git Configuration Prerequisite:
+ Before any git commit operation, verify configuration:
+ - Check: git config user.name
+ - Check: git config user.email
+ If either is unset, configure defaults:
+ git config user.name "Agent"
+ git config user.email "agent@localhost"
Validation Results:
Impact: +9.0% pass rate improvement on git-related tasks
Weakness Mining Discovery:
Pattern: 12 failures on tasks involving file operations
Root Cause: Agent assumes file operations succeed without verification
Example Traces:
- Task 12: Created config.json but didn't verify, later steps failed
- Task 34: Assumed mkdir succeeded, then tried to cd into non-existent directory
Harness Proposal Generated:
Tool Wrapper Addition:
def create_file(path, content):
execute_bash(f"cat > {path} << 'EOF'\n{content}\nEOF")
+ # Verify file was created
+ if not execute_bash(f"test -f {path}"):
+ raise FileNotFoundError(f"Failed to create {path}")
+ # Verify content matches
+ actual = execute_bash(f"cat {path}")
+ if actual.strip() != content.strip():
+ raise ValueError(f"Content mismatch in {path}")
Validation Results:
Impact: +11.2% pass rate improvement on file operation tasks
Weakness Mining Discovery:
Pattern: 7 failures on long, multi-step tasks
Root Cause: Model loses track of intermediate results and task state
Example Traces:
- Task 56: Forgot database connection string from step 2 by step 5
- Task 78: Lost API key after environment setup, failed authentication
Harness Proposal Generated:
Planning Template Update:
Plan for completing task:
+ [State Tracking]
+ - Track: {key variables to remember}
+ - Update tracker after each step completion
+
1. {step 1}
+ → Record outcome: {what to remember}
2. {step 2}
+ → Record outcome: {what to remember}
...
N. {final step}
+ → Verify: All tracked variables are still accessible
Validation Results:
Impact: +6.7% pass rate improvement on multi-step tasks
| Aspect | Human Engineering | Self-Harness |
|---|---|---|
| Speed | Days to weeks per model | Hours (automated) |
| Scalability | Limited by human expertise | Scales with compute |
| Model-Specificity | Requires manual analysis | Automatically discovers patterns |
| Consistency | Varies by engineer skill | Systematic and reproducible |
| Cost | High (expert time) | Low (compute only) |
| Adaptation | Manual updates needed | Continuous self-improvement |
When Human Engineering Still Matters:
Some approaches use stronger models (e.g., GPT-5.5) to improve weaker agents. Self-Harness differs:
External Scaffolding Approach:
Self-Harness Approach:
Philosophical Difference:
"A model should be able to identify and fix its own systematic weaknesses, not rely on a smarter model to tell it what's wrong."
Microsoft's SkillOpt also addresses self-improvement but focuses on skill refinement rather than harness optimization:
| Feature | Self-Harness | SkillOpt |
|---|---|---|
| Target | Agent harness (system-level) | Individual skills (task-level) |
| Scope | Cross-task patterns | Single-skill optimization |
| Method | Trace analysis + proposals | Skill execution feedback |
| Validation | Regression testing | Skill-specific metrics |
| Granularity | System prompts, tool wrappers | Skill code and parameters |
Complementary Approaches: Both can be used together—SkillOpt optimizes individual skills while Self-Harness improves the overarching framework.
Input: Execution traces from failed and successful tasks
Output: Ranked list of weakness patterns with proposed fixes
Pseudo-code:
def mine_weaknesses(traces, model):
failures = [t for t in traces if not t.success]
# Group failures by similarity
clusters = cluster_by_error_pattern(failures)
weaknesses = []
for cluster in clusters:
# Analyze common failure mode
pattern = model.analyze_pattern(cluster)
# Extract root cause
root_cause = model.identify_root_cause(pattern, cluster)
# Count frequency and impact
frequency = len(cluster)
impacted_tasks = extract_task_ids(cluster)
weaknesses.append(Weakness(
pattern=pattern,
root_cause=root_cause,
frequency=frequency,
impacted_tasks=impacted_tasks
))
# Rank by frequency × impact
return sorted(weaknesses, key=lambda w: w.frequency, reverse=True)
Key Techniques:
Input: A single weakness with context
Output: 3-5 diverse candidate harness modifications
Prompt Template:
You are analyzing your own execution failures to improve your harness.
Weakness Pattern:
{weakness.pattern}
Root Cause:
{weakness.root_cause}
Failed Task Examples:
{weakness.example_traces}
Current Harness:
{current_harness}
Generate 3-5 diverse, minimal harness modifications that would prevent this failure pattern.
For each proposal:
1. Describe the change
2. Explain why it addresses the root cause
3. Provide concrete implementation (system prompt, tool wrapper, or planning template)
4. Estimate impact on other tasks
Keep changes minimal and targeted. Avoid large rewrites.
Diversity Enforcement:
Input: Current harness, proposed harness, validation task set
Output: Accept/reject decision with detailed metrics
Validation Workflow:
def validate_proposal(current_harness, proposed_harness, val_tasks):
# Baseline performance
baseline_results = run_tasks(current_harness, val_tasks)
baseline_pass_rate = compute_pass_rate(baseline_results)
# Proposed performance
proposal_results = run_tasks(proposed_harness, val_tasks)
proposal_pass_rate = compute_pass_rate(proposal_results)
# Check for regressions
regressions = [
task for task in val_tasks
if baseline_results[task].passed and not proposal_results[task].passed
]
# Check for improvements
improvements = [
task for task in val_tasks
if not baseline_results[task].passed and proposal_results[task].passed
]
# Decision criteria
if len(regressions) > 0:
return Decision.REJECT, "Introduced regressions"
if proposal_pass_rate <= baseline_pass_rate:
return Decision.REJECT, "No net improvement"
if len(improvements) == 0:
return Decision.REJECT, "No tasks improved"
return Decision.ACCEPT, f"Improved {len(improvements)} tasks"
Regression Testing:
1. Computational Cost
2. Local Optima Risk
3. Benchmark Overfitting
4. Limited to Harness-Fixable Failures
5. Minimal Harness Assumption
1. Cross-Model Harness Transfer
Question: Can harness improvements from Model A transfer to Model B?
Approach: Train Self-Harness on cheap model, transfer to expensive model
Potential: Reduce optimization cost for expensive frontier models
2. Multi-Benchmark Generalization
Question: Can harness optimize across multiple benchmarks simultaneously?
Approach: Validate proposals on Terminal-Bench 2.0 + SWE-bench + GAIA
Potential: More generalizable harnesses that work across domains
3. Compositional Harness Modules
Question: Can we build libraries of reusable harness modules?
Approach: Extract successful patterns into plug-and-play components
Potential: Faster initial harness setup for new models
4. Human-in-the-Loop Validation
Question: Can human review improve Self-Harness proposals?
Approach: Expert reviews edge cases and suggests refinements
Potential: Combine automation speed with human insight
5. Continuous Online Improvement
Question: Can Self-Harness improve during production deployment?
Approach: Mine weaknesses from real user interactions, propose fixes
Potential: Agents that continuously adapt to real-world usage patterns
What This Means:
How to Apply:
Research Questions Opened:
Benchmark Implications:
Strategic Considerations:
When to Use Self-Harness:
When to Stick with Human Engineering:
Hybrid Approach:
1. Human experts design initial harness architecture
2. Self-Harness optimizes model-specific details
3. Humans review and approve proposed changes
4. Deploy optimized harness to production
5. Continuous Self-Harness monitoring for new failure patterns
Self-Harness builds on the emerging discipline of agent harness engineering, where differentiation comes from the scaffolding around the model, not just the model itself.
Timeline:
Philosophical Shift:
"Frontier models are table stakes. Differentiation is the harness—the loop, tools, middleware, and verification around the model."
Loop engineering focuses on designing effective agent execution loops—the repeated cycle of planning, action, observation, and refinement.
Self-Harness Contribution:
The choice of Terminal-Bench 2.0 as the evaluation benchmark is significant:
Why Terminal-Bench 2.0 Works for Self-Harness:
Benchmark Scores Context:
Paper:
Authors:
Code Availability:
Prerequisites:
# Terminal-Bench 2.0 setup
git clone https://github.com/laude-institute/terminal-bench-2.0
cd terminal-bench-2.0
pip install -r requirements.txt
# Base model access (choose one)
# - MiniMax M2.5 API
# - Qwen3.5-35B-A3B (via ollama or API)
# - GLM-5 API
# Harbor framework
pip install harbor-agents
Running Self-Harness:
from self_harness import SelfHarnessOptimizer
from terminal_bench import load_benchmark
# Load benchmark
tasks = load_benchmark("terminal-bench-2.0")
train_tasks, val_tasks = split_tasks(tasks, ratio=0.7)
# Initialize with minimal harness
minimal_harness = MinimalHarness(
model="minimax-m2.5",
system_prompt="You are an AI agent with access to terminal commands."
)
# Run Self-Harness optimization
optimizer = SelfHarnessOptimizer(
base_harness=minimal_harness,
max_iterations=10,
validation_tasks=val_tasks
)
optimized_harness = optimizer.optimize(train_tasks)
# Evaluate
baseline_score = evaluate(minimal_harness, val_tasks)
optimized_score = evaluate(optimized_harness, val_tasks)
print(f"Baseline: {baseline_score:.1%}")
print(f"Optimized: {optimized_score:.1%}")
print(f"Gain: +{optimized_score - baseline_score:.1%}")
Expected Results (MiniMax M2.5):
Baseline: 40.5%
Optimized: 61.9%
Gain: +21.4%
Paper:
Self-Harness was published on arXiv on June 8, 2026, introducing a paradigm where LLM-based agents autonomously improve their own operating harnesses through weakness mining, harness proposals, and validation—achieving substantial performance gains on Terminal-Bench 2.0 across diverse base models without requiring human engineers or stronger external models.