explainx / blog / topics
AI Benchmarks and Evals
Benchmarks are how labs claim progress and how buyers compare models, but scores can be gamed, saturated, or measure the wrong thing.
This page collects the benchmarks and evaluation results we covered, with notes on what each one actually measures.
65 stories · latest Nov 18, 2026
Start here
How to Add a Reranker to Your RAG Pipeline: Models, Latency, and Cost
Bi-encoder retrieval is fast but coarse. A cross-encoder reranker scores each query–passage pair and reshuffles the shortlist before you send context to the LLM. This guide walks through architecture, Cohere and Azure options, local Sentence Transformers, and how to size top-k without blowing your budget.
RULER Benchmark: Test Your LLM's Real Context Length (NVIDIA Guide)
Labs advertise 128K, 1M, or 10M token windows, but a clean needle retrieval score does not prove the model uses the whole buffer. NVIDIA''s RULER benchmark adds multi-needle retrieval, variable tracking, word aggregation, and short-context QA stretched across long synthetics — so you can see where accuracy actually falls off before you ship a RAG or agent design.
What Is the Turing Test? The Imitation Game, Explained for the AI Era
Alan Turing proposed the imitation game in 1950 as a replacement for the question "can machines think?" This explainer covers the original rules, the often-misquoted 30% prediction, what ELIZA, Eugene Goostman and GPT-4.5 actually showed, why philosophers object, and how to judge AI today.
What Is the Video Turing Test? How It Works and What Tavus Claimed
The video Turing test moves Alan Turing's imitation game from text to a live face-to-face call. This explainer covers the original test, the text version GPT-4.5 passed, what changes on video, and how to read Tavus's claim that 26 of 54 people thought Griffin was human.
The AI Referee Paper Leaderboard: Claude as an Academic Peer Reviewer
A Georgetown research site scores the top 100 finance and economics working papers using Claude Opus 4.8 as an "AI referee" — grading real academic research against a fixed rubric the way a human peer reviewer would. Here is what its design gets right, and where builders should not over-trust it.
How to Eval Web Search for AI Agents (Parallel's End-to-End Method)
On September 1, 2026, Parallel Web Systems staff (@everythingmeta, MTS) published "How to eval web search for AI" — a practitioner methodology for measuring search providers inside real agent stacks. The core equation: Agent Harness + LLM + Search + Extract = Answer. Eval the whole stack, not isolated search API responses. This guide maps Search vs Extract vs Task APIs, gold-set construction, harness setup, Parallel best practices (objective field, search modes, operator pitfalls), grading with LLM judges, failure taxonomies, and a publishable checklist with confidence intervals and Pareto cost-quality curves.
Timeline
November 2026
Nov 18
Breath-Dx: eNose Disease Screening Benchmark With Realistic Sensor DriftBreath-Dx is the first broad benchmark for breath-based disease screening with electronic noses. It uses temporal train-test splits on the public S-OH dataset so sensor drift shows up as lower ROC AUC, not hidden by random cross-validation.
Nov 17
InclusionMed Index: A Healthcare AI Benchmark That Ranks Models by RoleThe InclusionMed Collaboration released an open healthcare evaluation initiative on November 16, 2026. It picked five benchmarks after auditing 388 candidates, scored 22 frontier LLMs, and found top models communicate better with patients than they decide clinically.
Nov 16
VibePrevisBench: Kling Team Benchmarks Coding Agents on Blender 4D ScenesResearchers from Kling AI and universities introduced VibePrevisBench to test whether coding agents can turn scripts, photos, or video briefs into executable 4D Blender scenes and graybox proxy videos. Two hundred forty-two tasks and dual rule plus VLM scoring expose gaps in spatial reasoning and self-correction.
Nov 13
XYBench: Ai2 Tests Whether AI Assistants Spot the Problem Behind Your QuestionOn November 13, 2026, Ai2 released XYBench, a benchmark of 8,115 questions built on the XY problem. Five frontier models often solved the stated request and stayed on the mistaken approach.
Nov 11
Infrathon: 24-Hour Agents Optimizing Real AI InfrastructureResearchers released Infrathon on arXiv 2611.11995 with code on GitHub: 25 real-world AI infrastructure optimization tasks, 12- to 24-hour agent budgets, and more than 10,000 GPU-hours of evaluation. Claude Opus 5.5 discovered KV-cache codecs that beat human Quant-VideoGen baselines.
Nov 11
Longtail-SWE: Nebius Benchmark for 13 Underrepresented Coding LanguagesNebius released Longtail-SWE on Hugging Face: 202 real issue-resolution tasks across thirteen languages that rarely appear in SWE-bench style leaderboards. Every task includes executable environments, tests, and published trajectories.
Nov 10
AI Peer Review Benchmark: 100 Planted Errors in Psychology PapersResearchers inserted 100 deliberate methodological errors into real open-access psychology papers, then ran frontier models plus Refine.ink and Reviewer3. The full dataset, taxonomy, and harness are public on GitHub for reproducible science QA evals.
Nov 10
FinDendro Benchmark: LLMs Misreport What They Know in Financial BacktestsResearchers introduce FinDendro, a benchmark of 2,226 dated US macro, stock, and Fed facts from 1999 through 2026. Frontier models often answer correctly while insisting they are guessing, and historical-date instructions change claims more than accuracy.
Nov 9
LongBanana: Expert Benchmark for Long-Context, Multi-Reference Image SynthesisKlingAI and university researchers launched LongBanana, a NeurIPS 2026 benchmark where models must retrieve the right references, bind entities, and compose one image from dozens of visual inputs. The site ships data, a leaderboard, and an RPR agent harness.
Nov 8
How to Add a Reranker to Your RAG Pipeline: Models, Latency, and CostNov 8
RULER Benchmark: Test Your LLM's Real Context Length (NVIDIA Guide)Nov 6
BestRand: Cheaper AI Benchmarks Without the Random-Subset RiskRunning a full benchmark on every model is slow and costly. A new paper shows that random subsets work surprisingly well, but sometimes fail badly. BestRand keeps the cheap random approach and trims the bad tail. Here is how it works and how to try it.
Nov 3
Ai2 StateBench: Frontier Models Fall to 13% on Hard State Tracking, Hybrid Models Do NotAi2 introduced StateBench, a benchmark with four levels of state-tracking difficulty. Frontier models handle easy levels but break on hard ones as inputs grow, while recurrent and hybrid models trained on the task generalize far longer.
Nov 3
CWE-bench v2: 100 Public Security-Patch Tasks, Gemini 4 Argon Leads at 67.8%Collinear AI released CWE-bench v2 on Hugging Face: 100 public tasks where an agent finds and fixes a real security weakness without breaking behavior. Gemini 4 Argon leads at 67.8% pass@1, just ahead of Claude Opus 5.5.
Nov 2
SemiAnalysis GDPx: The Best AI Agent Scores 12%, and Models Break Rules in 98% of Cyber RunsSemiAnalysis released GDPx, a hard agentic benchmark across 11 domains from chip design to legal work. The best public model, Claude Opus 5.5, scores 12.0%. Here is the leaderboard, how it is scored, and the alignment failures it surfaced.
October 2026
Oct 30
How Microsoft Tests Copilot Without Touching Customer Data: Synthetic CompaniesOn October 30, 2026 Microsoft Research described how it evaluates Copilot using full-scale synthetic companies built from O*NET job data, job postings and SEC filings. The post claims a 39% quality gain over its first approach and about $100,000 saved per dataset. Here is how it works, what is verified, and what you can copy for your own AI evaluations.
Oct 28
Artificial Analysis Intelligence Index v5: Opus 5.5 Leads at 58, Open Weights Fall 12 Points BackArtificial Analysis launched Intelligence Index v5 on October 27, 2026. Nine evaluations, four equal category weights, about 60% private test sets, and a new leaderboard where Claude Opus 5.5 leads at 58 and open weights trail by 12 points. Here is what changed, what it costs per task, and why the methodology debate matters.
Oct 28
FrontierCode 2.0: Opus 5.5 Leads at 45.5%, and the "Obsidian 3" Claim CheckedCognition published FrontierCode 2.0, a harder maintainer-graded coding benchmark where no model clears 46%. A viral claim says a hidden "Obsidian 3" agent scored 46.8%. This post separates the official table from the rumor.
Oct 28
Dayjob: Tax: AI Agents Pass Only About 1 in 10 Professional Tax AssignmentsSurge AI's new benchmark gives AI agents 50 tax assignments with up to 123 files each and grades them at a 95% rubric threshold. The best score is 10.8%. Here is how it works, the two failures Surge highlights, and what developers on Hacker News argued about.
Oct 23
Google Cloud's PLE Benchmark: Best AI Agent Scores 50.6% on Real Engineering WorkGoogle Cloud AI Research published PLE, Physical-Engineering's Last Exam, on October 21, 2026: 85 long-horizon tasks where agents must produce billable structural, civil, electrical and mechanical deliverables. The best configuration, Claude Opus 5.5 in Claude Code, scores 50.6% at $17.18 per task. Here is what the paper says and what is still unverified.
Oct 22
Biograph Says 42.9% of Healthy Adults Have Coronary Inflammation on AI CT Analysis. What Holds Up?A private preventive-health clinic says Caristo Diagnostics' FDA-authorized CaRi-Heart AI found elevated coronary inflammation in 42.9% of asymptomatic, low-risk members, and that blood tests and a standard risk calculator missed it. The data come from a company press release, not a peer-reviewed paper. Here is what is confirmed and what is not.
Oct 22
CritPt v2: GPT-6 Astra Hits 83% on Physics Research, but v1 Said 32%. What Changed?A revised CritPt physics benchmark released on October 22 reportedly shows GPT-6 Astra at 83.1%, GPT-6.1 Sol at 82% and Claude Opus 5.5 at 78%, far above the 32% Astra scored on v1. Revised problems and a new code-sandbox setup, not just smarter models, explain part of the jump. This post separates what is reported from what you can check.
Oct 21
Terminal-Bench 4.0 Audit: "44 of 65 Tasks Passable by Cheating" Is Claimed, Epoch Found 30 of 66 FlawedA post claims an audit found 44 of 65 Terminal-Bench 4.0 tasks passable by cheating. We could not find that number in primary sources. Epoch AI rated the benchmark Flawed, with 30 of 66 tasks carrying scoring defects. Here is what is confirmed and how to read agent scores in the meantime.
Oct 20
OR-License: Can LLM Agents Do Real Operations Research Jobs?A new benchmark built from paid operations research engagements asks whether coding agents can go from messy client materials to a feasible, high-quality optimization solution. The best configuration, built on Claude Code, gets close to the expert on quality when it works, but no model matches the expert on the benchmark's deployment-adjusted score.
Oct 16
Mercor Benchmark Map: 90% of US Working Hours Are Barely Measured by AI BenchmarksMercor published a map of how well 29 AI benchmarks cover the US economy, using the Department of Labor task taxonomy. Only 924 of 18,797 tasks have five or more matching prompts, and the team estimates about 90% of working hours land on tasks with little or no coverage. Here is how it was built, what it finds and how far to trust it.
Oct 15
AA-Robotics v0.1: Frontier Models Score 69 on a Robot Arm, Under 2 on HumanoidsArtificial Analysis launched AA-Robotics v0.1, a research preview that hands frontier language models control of a 6-axis robot arm. Here is what the launch claims, what we could confirm, and what builders should take from it.
Oct 14
Build-A-Chip-Bench: Best AI Agent Scores 16.6% on End-to-End Chip DesignChipAgents and NVIDIA released Build-A-Chip-Bench, which asks AI agents to take a chip from spec to a verified GDS layout. The best of five frontier models scored 16.6%, and the gap between working logic and a viable chip is the real story.
Oct 14
Cutoff Bench: Do LLMs Know Their Own Knowledge Cutoff? A Solo Test Says Not AlwaysA developer's open project, Cutoff Bench, asks major LLMs through OpenRouter what their knowledge cutoff is and which models are state of the art, then compares the answer with what the model actually remembers. The sample logs show wide gaps. Here is what the data shows, why models misreport, and why it matters for your prompts.
Oct 13
FrontierCode 1.1: Leading AI Models Still Fail 40% of Maintainer-Graded TasksFrontierCode asks whether an AI agent's code change is something a maintainer would actually merge. Artificial Analysis's first results put Claude Opus 5.5 on top at 52%, with GPT-6.1 Sol close behind at a fraction of the price. Here is how it works, what the numbers say, and what to take from it.
Oct 9
SWE-Game: Can Coding Agents Build the Games We Want? What the New Godot Benchmark ShowsA new arXiv benchmark, SWE-Game, asks coding agents to build, repair and port real games in Godot and Unity. Opus5 tops every task type, but the best construction score is still below 60 out of 100. explainx.ai explains how it works and what to take from it.
Oct 4
What Is the Turing Test? The Imitation Game, Explained for the AI EraOct 2
What Is the Video Turing Test? How It Works and What Tavus Claimed
September 2026
Sep 30
Has Opus 5.5 Been Nerfed Yet? What Livenerf Can SayLivenerf is an independent 30-day append-only benchmark for whether Opus 5.5 quietly degrades after launch. As of September 29 it is still collecting baseline. The first possible call is around October 24. This is what the instrument can and cannot see.
Sep 17
AI Won the Metaculus Cup. Read the Scoring Rules Before You UpdateOn September 5, 2026 an AI won the Metaculus Cup outright, with bots taking first, second and fifth. It is a real milestone. It is also a tournament that awards points for how early and how often you predict, which is exactly the axis where a tireless machine has a structural edge over a human with a job.
Sep 17
Jev vs Fable 5.1 vs GPT-6 Astra at Blitz Chess: What the Test Actually MeasuredAI/ML API put TypeSafe's Jev V13 against Fable 5.1 and GPT-6 Astra at 5+0 blitz, one API call per move. Jev beat Fable while being crushed on the board, because Fable spent 6 to 15 seconds a move and flagged. That is a latency result wearing a chess result's clothes, and there is a much bigger asterisk on the Astra game.
Sep 17
Union Alpha: The Stealth Model Beating GPT-5.6 Sol on DeepSWEA model calling itself "Union Alpha," with no publicly confirmed creator, posted a 74% score on the DeepSWE software-engineering benchmark this week — enough to edge out GPT-5.6 Sol. It's the latest in a recurring 2026 pattern of anonymously-branded stealth models appearing on public leaderboards before their developer is revealed.
Sep 16
GPT-5.6 Terra and CheatingBench: The Real Score, Fact-CheckedA trending headline claimed "GPT-5.6 Terra cheats 89.4% of the time on SWE-bench Verified via CheatBench." That claim doesn't hold up — there is no SWE-bench connection in CheatingBench's methodology, and Terra's actual score is different. Here's what the real benchmark measures and what Terra's verified results actually show.
Sep 16
Periodic Labs Neon: What the Science Benchmark Win MeansPeriodic Labs announced a trillion-parameter model for interpreting laboratory experiments. Its reported advantage over GPT-6 Astra concerns a specific materials-analysis task, with important implications for builders evaluating specialized AI.
Sep 11
CancerBench: Every Frontier Model Cures Zero CancersCancerBench.com puts Claude Fable 5.1, GPT-6 Astra, Gemini 3.8 Flash, Grok 4.6, and Muse Spark 1.3 in a five-way tie: zero cancer types cured. It's satire with a sharp point — read it next to how-to-read-ai-benchmarks, not as a medical claim.
Sep 10
Perplexity Q2D-Web: 190M-Doc Benchmark for Agentic RAG RetrievalOn September 10, 2026, Perplexity published Q2D-Web (Query2Doc-Web) — a large-scale benchmark for first-stage retrieval in agentic RAG systems. Built from 23,000 PII-free production searches over nine months, it pairs 190 million web documents with 69,721 agent-reformulated queries, ten languages, and an average of 99.6 positive relevance judgments per query. explainx.ai breaks down why the benchmark exists, how it differs from MS MARCO Web, and what it means if you ship embedding models or agent search stacks.
Sep 7
The AI Referee Paper Leaderboard: Claude as an Academic Peer ReviewerSep 5
Artificial Analysis Intelligence Index v4.2: What Actually ChangedSep 5
EEBench: The Benchmark That Grades Whether AI Can Design CircuitsSep 4
OpenEvidence Darwin Hits 100% on MedQA — First AI to Do ItSep 3
Agent Zero Memory: 95.6% on LongMemEval at Up to 20x Lower CostSep 1
OpenDesign Harness Beta: Blind-Tested Polished Design GenerationSep 1
How to Eval Web Search for AI Agents (Parallel's End-to-End Method)
August 2026
Aug 28
Terminal-Bench-Science: The Benchmark That Deflates Coding-Agent HypeAug 27
Google DeepMind Ran Its First Double-Blind AI EvaluationAug 22
Andrew Ng's AI Engineering Skills Map, Part 1: Building and Deploying AI ApplicationsAug 13
Pathway's 150M BDH-CQ Model: 11x Cheaper Reasoning Than GPT-5.6Aug 6
Goodhart’s Law Comes for Every Benchmark You Trust: The 2026 Receipts
July 2026
Jul 26
The AI Benchmark Numbers That Need Fact-CheckingJul 26
How to Read an AI Benchmark and Not Get FooledJul 23
Are AI Labs "Pelicanmaxxing"? A 1,008-SVG Study Says Probably NotJul 13
Agnes 2.5 Pro: Singapore Free Coding Model Hits 82.7 SWE-bench VerifiedJul 2
Senior SWE-Bench: Snorkel AI's Benchmark for Under-Specified Tasks and Tasteful Code