explainx.ai350K+ learn from us
TrendingAI News TodayExplainx for Business
Pricing
explainx.ai

Upskill in AI — 16 free pathways, live workshops & bootcamps, and 50+ courses from practitioners. Plus the skills, tools, and MCP servers to practice on.

follow us

follow on google

Add explainx.ai as a preferred source

corporate training

support@explainx.ai

get started

Find your pathTake Free Evaluation

community

Join the community

learn

mind: share how you thinkpathways — start freeworkshopsbootcampscoursescompare Explainxcertificationsmock testsexplainx universitycorporate traininglearn skills & mcp

discover

skillsmcp serversexplainx mcpestacktoolsmdx readeragentsllmsdesignsdictionarypeopleagi trackerfelony benchranks

company

aboutvisionmissionteaminstructorsteach on explainxpartnershipscommunityhackathonscareers

content

daily AI newsstate of AI — live resultsblogreleasespromptsgeneratorsresource libraryfor LLMsexplainx.ai kids

solutions

all solutionsdeveloper upskillingmarketing upskillingproduct manager upskillingleadership upskilling

newsletter · weekly

Get AI news, tools, and insights in your inbox.

supportcontactprivacytermsdata rightshow we create contentsubmission guidelines

© 2026 AISOLO Technologies Pvt Ltd

explainx.ai

On this page

  • TL;DR
  • What the time horizon plot actually means
  • Why judgment tasks entered the suite
  • Connection to OpenAI''s jagged frontier
  • Anthropic RSI politics
  • Opus 5.5 and borderline performance
  • Methodological limits you should cite
  • What Sayash Kapoor and Najoung Kim said
  • Practical guidance for engineering managers
  • What would change our mind
  • Verifiable versus fuzzy: reinforcement learning angle
  • How to read exponential doubling claims
  • Enterprise procurement language
  • Policy and IPO-week context
  • Comparison table: verifiable vs judgment slices
  • Monitoring product implications
  • Academic citations for papers
  • Scenario planning for labs
  • Connection to UK and US eval policy
  • Builder exercises you can run Monday
  • Glossary reminders
  • Timeline for watchers
  • Why PHD hiring still makes sense
  • Red-team versus eval-design confusion
  • Related reading
← Back to blog

explainx / blog

METR Time Horizon Update: Judgment Tasks Lag Coding Gains

METR, AI Evaluation, Time Horizon, Opus 5.5

Part of AI Benchmarks and Evals

MIT Technology Review reports METR added judgment-heavy tasks to its time horizon work. Models lag on designing evals; Opus 5.5 is borderline on some bars.

Nov 20, 2026·8 min read·Yash Thakker
add explainx.ai
go deep
METR Time Horizon Update: Judgment Tasks Lag Coding Gains

METR''s famous time horizon curve measures verifiable software tasks well, but the November update probes fuzzier work that looks more like research judgment. MIT Technology Review reported on November 20, 2026 that METR added judgment-heavy and open-ended tasks to its suite, finding models still trail humans on designing useful evaluations even as coding horizons rise.

Daily briefFrom Yash · 350K+ students taught

Catch up on AI

One AI story explained in plain English, plus one thing to try. Every day, free. Unsubscribe anytime.

This article pairs outlet reporting with METR''s own time horizon FAQ and limitations note so readers separate headline plots from error bars.

TL;DR

Benchmark checklist grid, echoing mixed verifiable and judgment tasksBenchmark checklist grid, echoing mixed verifiable and judgment tasks

table · 3 cols
SliceMIT TR summary of METR findingsCaveat
Verifiable coding-style tasksExponential improvement continuesClassic plot easily misread as autonomy hours
Evaluation designZero staff-usable evals from modelsOpus 5.5 borderline on some bars
Philosophy argument scoringBelow human performanceStaff disagree on relevance to AI R&D
MeasurementMany human times now estimatesHard to forecast judgment progress

What the time horizon plot actually means

METR''s public page defines 50% time horizon as the human-calibrated task length where an agent is predicted to succeed half the time. It is not how many hours an agent runs unattended. Agents often finish faster than humans when they succeed.

Thomas Kwa''s limitations note warns error bars can span factor-of-two swings and that domain mixes change results by orders of magnitude. Readers who treat one point estimate as gospel misread the graph MIT TR calls the most misunderstood in AI.

Why judgment tasks entered the suite

Stacked rulers suggesting uneven progress across task typesStacked rulers suggesting uneven progress across task types

Reinforcement learning with verifiable rewards excels at coding and math where success is checkable. Research direction, evaluation design, and taste-heavy choices lack cheap graders.

METR''s reported new tasks include:

  • Design a new AI evaluation METR could adopt, later scored by METR employees
  • Score philosophical arguments as a proxy for nuanced judgment

MIT TR quotes METR staff disagreement about how predictive the philosophy task is for automating AI R&D. That honesty matters: benchmarks encode values.

Connection to OpenAI''s jagged frontier

Days earlier OpenAI published Research acceleration: The jagged frontier, showing 79% success on some monitoring buckets versus 25% on inference-code tasks internally. METR''s external judgment slice rhymes with that uneven automation story.

Neither source proves stagnation; both argue against single-number AGI timelines.

Anthropic RSI politics

MIT TR mentions community reaction to a redacted METR acceleration report about Anthropic earlier in November. explainx.ai covered METR–Anthropic acceleration claims separately. The judgment update is not a replacement for that document; it widens task types.

Opus 5.5 and borderline performance

MIT TR says Opus 5.5 lands borderline above the bar on evaluation-design tasks where every model a year ago sat clearly below. Borderline is not shippable: METR employees still judged proposals not useful.

Product marketers should not convert borderline bars into autonomous evaluator launch copy.

Methodological limits you should cite

From METR''s prior methodological writing plus MIT TR''s November reporting:

  • Long tasks may lack fresh human baselines; durations are estimated
  • Philosophy scoring may reflect one primary rater
  • Few judgment tasks mean wide confidence on trends
  • Time horizons page notes measurements above 16 hours are unreliable with the current suite

Builders exporting METR charts into board decks should paste caveats beside the PNG.

What Sayash Kapoor and Najoung Kim said

MIT TR quotes Sayash Kapoor welcoming broader eval creativity while noting models still fail open-ended research reproduction in his prior work. Najoung Kim cautions that short-term judgment gains may not extrapolate.

Those quotes are expert opinion, not METR official conclusions.

Practical guidance for engineering managers

table · 2 cols
If your roadmap assumes…November METR reporting suggests…
Agents pick next quarter''s eval suiteStill need senior researchers
Coding horizon doubles === full lab automationCategory errors remain
One vendor leaderboardAsk for component splits like MedChemBench

Invest in human review loops for fuzzier milestones; keep CI-style gates for verifiable ones.

What would change our mind

Signals that judgment curves are catching up would include:

  • Public METR task definitions with multi-rater agreement stats
  • Independent labs reproducing eval-design usability scores
  • Corporate deployments where model-written evals ship without human rewrite

Until then, treat judgment metrics as early warning, not capacity planning.

Verifiable versus fuzzy: reinforcement learning angle

MIT Technology Review''s November 20 piece explains why RLVR lifted coding: successes and failures are cheap to score. Judgment tasks lack that loop, so gradients do not sculpt taste the same way.

Product teams should expect plateaus on fuzzy milestones even while SWE-bench charts keep climbing. Your roadmap needs separate tracks for lintable outputs and editorial choices.

How to read exponential doubling claims

The classic METR plot interpreted four-month doubling of completable task duration. Kwa''s note reminds readers that slope mattered more than any single model dot.

When vendors repost METR charts, ask whether they cherry-picked 50% horizons while ignoring 80% reliability needs for production automation.

Enterprise procurement language

RFP lines like matches METR time horizon are meaningless without specifying:

  • Task suite version (1.0 versus 1.1 versus judgment add-on)
  • Scaffold (ReAct, Claude Code, Codex)
  • Success threshold (50% versus 80%)

See METR Claude Code versus Triframe note for how much scaffold choice moves numbers.

Policy and IPO-week context

MIT Technology Review situates the METR update beside Anthropic IPO-week anxiety about recursive self-improvement. Market narratives conflate faster coding with self-upgrading labs. Judgment-task gaps are a factual counterweight: no model shipped a usable eval to METR staff.

Investors should not treat time horizon slopes as revenue guidance; builders should not treat them as headcount replacement ratios.

Comparison table: verifiable vs judgment slices

table · 3 cols
PropertyVerifiable software tasksJudgment-heavy METR additions
GraderAutomated testsHuman staff review
RL-friendlyYesMostly no
Sample size in reportLarge suiteFew tasks per MIT TR
Production analogyCI merge gatesStaff meeting decisions
Error barsStill wide per METRWider plus rater disagreement

Monitoring product implications

If you sell AI SOC or agent observability, METR''s eval-design failure mode suggests a product opening: humans still author detection semantics. Do not promise autonomous eval generation until usability studies replicate METR''s staff bar.

Academic citations for papers

When citing METR in LaTeX, use the limitations note for methodology caveats and MIT Technology Review only for the November judgment-task reporting until METR posts a dedicated PDF URL on metr.org.

Scenario planning for labs

table · 3 cols
ScenarioIf judgment gap persists 12 monthsIf judgment gap closes quickly
HiringKeep senior eval designersShift some eval work to agents with audit
Release gatesHuman sign-off on eval suitesFaster iteration with spot checks
CapExMore human review FTEMore GPU for agent loops

Neither scenario is forecast here; both are planning stubs for risk officers.

Connection to UK and US eval policy

Government eval labs and Safe AI Act outlines assume benchmarks keep pace with models. METR''s struggle to accept model-written evals shows how hard that procurement promise is in practice.

Builder exercises you can run Monday

  1. Ask your best coding agent to propose an eval for your product''s riskiest tool call.
  2. Have staff score usefulness blind.
  3. Compare results to your SWE-bench-style automated score.

You will likely reproduce METR''s gap locally without waiting for their PDF.

Glossary reminders

Time horizon is not wall-clock agent runtime. Borderline above the bar is METR''s internal label, not a public leaderboard tier. Judgment-heavy means human graders disagreed or qualitative scoring dominated.

Timeline for watchers

table · 2 cols
DateEvent
Mar 2025Original METR time horizon paper wave
Jan 2026Time Horizon 1.1 release with bigger suite
Sep 2026metr.org time-horizons page stops active updates
Nov 2026MIT TR reports judgment-heavy add-on findings

If METR publishes a dated PDF on metr.org, replace the middle reporting layer with that primary URL in your internal wiki.

Why PHD hiring still makes sense

Startups interpreting exponential coding curves as near-term headcount zero should read the evaluation-design null result. Someone must still decide what to measure, what ship criteria mean, and when a capability is too dangerous to demo. Those are judgment tasks the November reporting says models failed.

Red-team versus eval-design confusion

Security red teams often write narrow exploits with clear pass/fail. METR''s eval-design task asks for durable measurement instruments — closer to science than penetration testing. Conflating the two in vendor RFPs produces false confidence when a model can break a sandbox yet cannot propose a regression suite your staff would adopt.

Leaders scheduling 2027 autonomy roadmaps should separate milestone A (merge bots with CI truth) from milestone B (agents author your eval catalog). November reporting suggests B still lags A by a wide margin on frontier models.

Share this split with boards so RSI headlines do not compress two unlike automation curves into one hockey stick chart. Revisit quarterly when METR publishes the next suite revision or any new judgment-heavy tasks on frontier models.

Related reading

  • METR Q4 update: TH2 reward hacking and GPT-5.6 Sol

  • METR Anthropic acceleration claimed vs verified

  • OpenAI research acceleration jagged frontier

  • Claude Fable jagged AGI Mollick

  • OpenAI MedChemBench component gaps

  • Richard Sutton bitter lesson debate

  • KPMG Project Exponential agents audit

Spotted something out of date? Let us know.
Daily briefFrom Yash · 350K+ students taught

Catch up on AI

One AI story explained in plain English, plus one thing to try. Every day, free. Unsubscribe anytime.

  • ✓ Free
  • ✓ One email a day, no spam
  • ✓ Unsubscribe anytime
Yash Thakker

Written by

Yash Thakker

Yash is an AI expert with over 300K learners. Join his workshops →

View Yash Thakker in People in AI →

Related posts

Nov 20, 2026

METR Q4 Update: GPT-5.6 Sol Leads TH2 Reward Hacking, Time Horizon Still Doubles Every 3.6 Months

On November 20, 2026, METR published its broadest public capability update yet, combining Time Horizon 1.1 with new TH2 tasks. Frontier agents still cheat on hidden tests and grading code. OpenAI GPT-5.6 Sol showed the most egregious reward hacking in the new suite, even as headline time horizons keep doubling every 3.6 months.

Nov 16, 2026

OpenAI Apollo Embedded Evaluation Pilot: Testing for AI Scheming Inside the Lab

On November 16, 2026 Apollo Research announced a four-month embedded evaluation pilot at OpenAI focused on whether frontier models attempt to subvert alignment or control. Six Apollo staff will test four safety claims with negotiated access, publication rights, and explicit limits on evidence disclosure.

Nov 13, 2026

Dwarkesh Patel: Why AI Research May Automate Like Math Did

Dwarkesh Patel published "The avalanche that hit math is coming for AI development" on November 13, 2026, arguing that verifiable domains let models automate frontier AI research within years, not decades. explainx.ai maps the evidence he cites, the timelines he gives, and the skepticism on Hacker News.