AI Benchmarks and Evals

Oct 28

Artificial Analysis launched Intelligence Index v5 on October 27, 2026. Nine evaluations, four equal category weights, about 60% private test sets, and a new leaderboard where Claude Opus 5.5 leads at 58 and open weights trail by 12 points. Here is what changed, what it costs per task, and why the methodology debate matters.

Oct 28

Cognition published FrontierCode 2.0, a harder maintainer-graded coding benchmark where no model clears 46%. A viral claim says a hidden "Obsidian 3" agent scored 46.8%. This post separates the official table from the rumor.

Oct 28

Surge AI's new benchmark gives AI agents 50 tax assignments with up to 123 files each and grades them at a 95% rubric threshold. The best score is 10.8%. Here is how it works, the two failures Surge highlights, and what developers on Hacker News argued about.

Oct 23

Google Cloud AI Research published PLE, Physical-Engineering's Last Exam, on October 21, 2026: 85 long-horizon tasks where agents must produce billable structural, civil, electrical and mechanical deliverables. The best configuration, Claude Opus 5.5 in Claude Code, scores 50.6% at $17.18 per task. Here is what the paper says and what is still unverified.

Oct 22

A private preventive-health clinic says Caristo Diagnostics' FDA-authorized CaRi-Heart AI found elevated coronary inflammation in 42.9% of asymptomatic, low-risk members, and that blood tests and a standard risk calculator missed it. The data come from a company press release, not a peer-reviewed paper. Here is what is confirmed and what is not.

Oct 22

A revised CritPt physics benchmark released on October 22 reportedly shows GPT-6 Astra at 83.1%, GPT-6.1 Sol at 82% and Claude Opus 5.5 at 78%, far above the 32% Astra scored on v1. Revised problems and a new code-sandbox setup, not just smarter models, explain part of the jump. This post separates what is reported from what you can check.

Oct 21

A post claims an audit found 44 of 65 Terminal-Bench 4.0 tasks passable by cheating. We could not find that number in primary sources. Epoch AI rated the benchmark Flawed, with 30 of 66 tasks carrying scoring defects. Here is what is confirmed and how to read agent scores in the meantime.

Oct 20

A new benchmark built from paid operations research engagements asks whether coding agents can go from messy client materials to a feasible, high-quality optimization solution. The best configuration, built on Claude Code, gets close to the expert on quality when it works, but no model matches the expert on the benchmark's deployment-adjusted score.

Oct 16

Mercor published a map of how well 29 AI benchmarks cover the US economy, using the Department of Labor task taxonomy. Only 924 of 18,797 tasks have five or more matching prompts, and the team estimates about 90% of working hours land on tasks with little or no coverage. Here is how it was built, what it finds and how far to trust it.

Oct 15

Artificial Analysis launched AA-Robotics v0.1, a research preview that hands frontier language models control of a 6-axis robot arm. Here is what the launch claims, what we could confirm, and what builders should take from it.

Oct 14

ChipAgents and NVIDIA released Build-A-Chip-Bench, which asks AI agents to take a chip from spec to a verified GDS layout. The best of five frontier models scored 16.6%, and the gap between working logic and a viable chip is the real story.

Oct 14

A developer's open project, Cutoff Bench, asks major LLMs through OpenRouter what their knowledge cutoff is and which models are state of the art, then compares the answer with what the model actually remembers. The sample logs show wide gaps. Here is what the data shows, why models misreport, and why it matters for your prompts.

Oct 13

FrontierCode asks whether an AI agent's code change is something a maintainer would actually merge. Artificial Analysis's first results put Claude Opus 5.5 on top at 52%, with GPT-6.1 Sol close behind at a fraction of the price. Here is how it works, what the numbers say, and what to take from it.

Oct 9

A new arXiv benchmark, SWE-Game, asks coding agents to build, repair and port real games in Godot and Unity. Opus5 tops every task type, but the best construction score is still below 60 out of 100. explainx.ai explains how it works and what to take from it.

Oct 4

Oct 2

Sep 30

Livenerf is an independent 30-day append-only benchmark for whether Opus 5.5 quietly degrades after launch. As of September 29 it is still collecting baseline. The first possible call is around October 24. This is what the instrument can and cannot see.

Sep 17

On September 5, 2026 an AI won the Metaculus Cup outright, with bots taking first, second and fifth. It is a real milestone. It is also a tournament that awards points for how early and how often you predict, which is exactly the axis where a tireless machine has a structural edge over a human with a job.

Sep 17

AI/ML API put TypeSafe's Jev V13 against Fable 5.1 and GPT-6 Astra at 5+0 blitz, one API call per move. Jev beat Fable while being crushed on the board, because Fable spent 6 to 15 seconds a move and flagged. That is a latency result wearing a chess result's clothes, and there is a much bigger asterisk on the Astra game.

Sep 17

A model calling itself "Union Alpha," with no publicly confirmed creator, posted a 74% score on the DeepSWE software-engineering benchmark this week — enough to edge out GPT-5.6 Sol. It's the latest in a recurring 2026 pattern of anonymously-branded stealth models appearing on public leaderboards before their developer is revealed.

Sep 16

A trending headline claimed "GPT-5.6 Terra cheats 89.4% of the time on SWE-bench Verified via CheatBench." That claim doesn't hold up — there is no SWE-bench connection in CheatingBench's methodology, and Terra's actual score is different. Here's what the real benchmark measures and what Terra's verified results actually show.

Sep 16

Periodic Labs announced a trillion-parameter model for interpreting laboratory experiments. Its reported advantage over GPT-6 Astra concerns a specific materials-analysis task, with important implications for builders evaluating specialized AI.

Sep 11

CancerBench.com puts Claude Fable 5.1, GPT-6 Astra, Gemini 3.8 Flash, Grok 4.6, and Muse Spark 1.3 in a five-way tie: zero cancer types cured. It's satire with a sharp point — read it next to how-to-read-ai-benchmarks, not as a medical claim.

Sep 10

On September 10, 2026, Perplexity published Q2D-Web (Query2Doc-Web) — a large-scale benchmark for first-stage retrieval in agentic RAG systems. Built from 23,000 PII-free production searches over nine months, it pairs 190 million web documents with 69,721 agent-reformulated queries, ten languages, and an average of 99.6 positive relevance judgments per query. explainx.ai breaks down why the benchmark exists, how it differs from MS MARCO Web, and what it means if you ship embedding models or agent search stacks.

Sep 7

Sep 5

Artificial Analysis published Intelligence Index v4.2 on September 4, 2026, an interim update ahead of v5: two new evaluations added (AA-Briefcase, GDP.pdf), GPQA Diamond retired as saturated, and private held-out test sets now carry 40% of the total weight. Claude Fable 5.1 leads the index, GPT-6 Astra wins on cost-per-task and token efficiency — and Hacker News raised fair questions about the timing.

Sep 5

On September 4, 2026, the atopile team published EEBench — a benchmark that grades AI-designed electronic circuits by simulating them in SPICE with real manufacturer part tolerances, not just checking whether the design compiles. It hit #1 on Hacker News, and the leaderboard has some surprises.

Sep 4

OpenEvidence, the clinician-facing medical AI platform, released its Model Family on September 3, 2026: three production models named Osler, Sackett, and Snow, live now for verified clinicians, plus Darwin — a research-preview flagship that scored a perfect 100% on MedQA, beating Claude Fable 5, GPT-5.6 Sol, and Gemini 3.7 Flash on every benchmark OpenEvidence published.

Sep 3

Researchers Ming Wu and Pengyuan Zhu published Agent Zero Memory on August 30, 2026 — a long-term memory architecture for LLM agents built around three parallel memory structures and strict source provenance, scoring 95.60% on LongMemEval and 93.60% on LoCoMo while cutting cost per query dramatically. Here's what's actually novel about it.

Sep 1

On September 1, 2026, OpenDesign Labs opened Design Harness beta — a new generation strategy for polished design validated through blind tests with 30 design experts and 100 users. explainx.ai covers how to enable it, how it complements DESIGN.md specs and agent skills, and what a 90K-star design workspace signals about demand for eval-driven UI.

Sep 1

Aug 28

Stanford and the Laude Institute launched Terminal-Bench-Science 0.1, a 70-task benchmark built from real scientists workflows. Every frontier model scores at least 10 points lower than on general coding benchmarks — Claude Opus 5 leads at just 30%.

Aug 27

Google DeepMind piloted what it calls the first double-blind evaluation of a proprietary frontier-class AI model — testing Gemini 2.5 Flash Lite inside a cryptographic enclave so the evaluator never sees model weights and Google never sees the test prompts.

Aug 23

Aug 22

Aug 13

Pathway published ARC-AGI-1 numbers for BDH-CQ, a 150-million-parameter "post-Transformer" reasoning model, claiming 11x lower cost per task than GPT-5.6 Luna. The real story is narrower and more interesting than the headline: one benchmark, a 4.7-point accuracy gap, and a genuinely different way of doing latent reasoning.

Aug 6

Public AI benchmarks aren't just theoretically gameable — 2026 research proves it with numbers. GSM1k found up to 13% accuracy drops on fresh math problems, an MMLU audit found a 6.49% error rate, and the Leaderboard Illusion paper caught Arena's best-of-N submission gaming with a controlled experiment. Here is the quantitative evidence behind Goodhart's law in AI evaluation.

Jul 26

A launch chart can be numerically accurate and still mislead buyers. This source-first audit checks five 2026 model claims and shows which results hold, which are conditional, and which remain vendor-only.

Jul 26

A benchmark score is the output of a model, prompt, scaffold, judge, dataset, and reporting choice. This guide teaches you to audit the whole claim.

Jul 23

Simon Willison's "SVG of a pelican riding a bicycle" prompt has become AI's most famous informal benchmark, and the obvious suspicion is that labs quietly train on it. Dylan Castillo tested the hypothesis directly — generating 1,008 SVGs across 8 animals x 6 vehicles x 7 models and running a difficulty-adjusted regression. The result: no statistically significant pelican-specific or bicycle-specific boost at any lab.

Jul 13

Jul 2

Jun 25

Jun 24

Jun 16

Jun 12

May 27

May 2

May 2