explainx.ai350K+ learn from us
TrendingAI News TodayPathwaysSkills
Pricing
explainx.ai

Upskill in AI — 16 free pathways, live workshops & bootcamps, and 50+ courses from practitioners. Plus the skills, tools, and MCP servers to practice on.

follow us

follow on google

Add explainx.ai as a preferred source

corporate training

support@explainx.ai

get started

Find your pathTake Free Evaluation

community

Join the community

learn

mind: share how you thinkpathways — start freeworkshopsbootcampscoursescompare Explainxcertificationsmock testsexplainx universitycorporate traininglearn skills & mcp

discover

skillsmcp serversexplainx mcpestacktoolsmdx readeragentsllmsdesignsdictionarypeopleagi trackerfelony benchranks

company

aboutvisionmissionteaminstructorsteach on explainxpartnershipscommunityhackathonscareers

content

daily AI newsstate of AI — live resultsblogreleasespromptsgeneratorsresource libraryfor LLMsexplainx.ai kids

solutions

all solutionsdeveloper upskillingmarketing upskillingproduct manager upskillingleadership upskilling

newsletter · weekly

Get AI news, tools, and insights in your inbox.

supportcontactprivacytermsdata rightshow we create contentsubmission guidelines

© 2026 AISOLO Technologies Pvt Ltd

explainx.ai

← All topics

explainx / blog / topics

AI Benchmarks and Evals

Benchmarks are how labs claim progress and how buyers compare models, but scores can be gamed, saturated, or measure the wrong thing.

This page collects the benchmarks and evaluation results we covered, with notes on what each one actually measures.

65 stories · latest Nov 18, 2026

Start here

How to Add a Reranker to Your RAG Pipeline: Models, Latency, and Cost

Bi-encoder retrieval is fast but coarse. A cross-encoder reranker scores each query–passage pair and reshuffles the shortlist before you send context to the LLM. This guide walks through architecture, Cohere and Azure options, local Sentence Transformers, and how to size top-k without blowing your budget.

RULER Benchmark: Test Your LLM's Real Context Length (NVIDIA Guide)

Labs advertise 128K, 1M, or 10M token windows, but a clean needle retrieval score does not prove the model uses the whole buffer. NVIDIA''s RULER benchmark adds multi-needle retrieval, variable tracking, word aggregation, and short-context QA stretched across long synthetics — so you can see where accuracy actually falls off before you ship a RAG or agent design.

What Is the Turing Test? The Imitation Game, Explained for the AI Era

Alan Turing proposed the imitation game in 1950 as a replacement for the question "can machines think?" This explainer covers the original rules, the often-misquoted 30% prediction, what ELIZA, Eugene Goostman and GPT-4.5 actually showed, why philosophers object, and how to judge AI today.

What Is the Video Turing Test? How It Works and What Tavus Claimed

The video Turing test moves Alan Turing's imitation game from text to a live face-to-face call. This explainer covers the original test, the text version GPT-4.5 passed, what changes on video, and how to read Tavus's claim that 26 of 54 people thought Griffin was human.

The AI Referee Paper Leaderboard: Claude as an Academic Peer Reviewer

A Georgetown research site scores the top 100 finance and economics working papers using Claude Opus 4.8 as an "AI referee" — grading real academic research against a fixed rubric the way a human peer reviewer would. Here is what its design gets right, and where builders should not over-trust it.

How to Eval Web Search for AI Agents (Parallel's End-to-End Method)

On September 1, 2026, Parallel Web Systems staff (@everythingmeta, MTS) published "How to eval web search for AI" — a practitioner methodology for measuring search providers inside real agent stacks. The core equation: Agent Harness + LLM + Search + Extract = Answer. Eval the whole stack, not isolated search API responses. This guide maps Search vs Extract vs Task APIs, gold-set construction, harness setup, Parallel best practices (objective field, search modes, operator pitfalls), grading with LLM judges, failure taxonomies, and a publishable checklist with confidence intervals and Pareto cost-quality curves.

Timeline

November 2026

  1. Nov 18

    Breath-Dx: eNose Disease Screening Benchmark With Realistic Sensor Drift

    Breath-Dx is the first broad benchmark for breath-based disease screening with electronic noses. It uses temporal train-test splits on the public S-OH dataset so sensor drift shows up as lower ROC AUC, not hidden by random cross-validation.

  2. Nov 17

    InclusionMed Index: A Healthcare AI Benchmark That Ranks Models by Role

    The InclusionMed Collaboration released an open healthcare evaluation initiative on November 16, 2026. It picked five benchmarks after auditing 388 candidates, scored 22 frontier LLMs, and found top models communicate better with patients than they decide clinically.

  3. Nov 16

    VibePrevisBench: Kling Team Benchmarks Coding Agents on Blender 4D Scenes

    Researchers from Kling AI and universities introduced VibePrevisBench to test whether coding agents can turn scripts, photos, or video briefs into executable 4D Blender scenes and graybox proxy videos. Two hundred forty-two tasks and dual rule plus VLM scoring expose gaps in spatial reasoning and self-correction.

  4. Nov 13

    XYBench: Ai2 Tests Whether AI Assistants Spot the Problem Behind Your Question

    On November 13, 2026, Ai2 released XYBench, a benchmark of 8,115 questions built on the XY problem. Five frontier models often solved the stated request and stayed on the mistaken approach.

  5. Nov 11

    Infrathon: 24-Hour Agents Optimizing Real AI Infrastructure

    Researchers released Infrathon on arXiv 2611.11995 with code on GitHub: 25 real-world AI infrastructure optimization tasks, 12- to 24-hour agent budgets, and more than 10,000 GPU-hours of evaluation. Claude Opus 5.5 discovered KV-cache codecs that beat human Quant-VideoGen baselines.

  6. Nov 11

    Longtail-SWE: Nebius Benchmark for 13 Underrepresented Coding Languages

    Nebius released Longtail-SWE on Hugging Face: 202 real issue-resolution tasks across thirteen languages that rarely appear in SWE-bench style leaderboards. Every task includes executable environments, tests, and published trajectories.

  7. Nov 10

    AI Peer Review Benchmark: 100 Planted Errors in Psychology Papers

    Researchers inserted 100 deliberate methodological errors into real open-access psychology papers, then ran frontier models plus Refine.ink and Reviewer3. The full dataset, taxonomy, and harness are public on GitHub for reproducible science QA evals.

  8. Nov 10

    FinDendro Benchmark: LLMs Misreport What They Know in Financial Backtests

    Researchers introduce FinDendro, a benchmark of 2,226 dated US macro, stock, and Fed facts from 1999 through 2026. Frontier models often answer correctly while insisting they are guessing, and historical-date instructions change claims more than accuracy.

  9. Nov 9

    LongBanana: Expert Benchmark for Long-Context, Multi-Reference Image Synthesis

    KlingAI and university researchers launched LongBanana, a NeurIPS 2026 benchmark where models must retrieve the right references, bind entities, and compose one image from dozens of visual inputs. The site ships data, a leaderboard, and an RPR agent harness.

  10. Nov 8

    How to Add a Reranker to Your RAG Pipeline: Models, Latency, and Cost
  11. Nov 8

    RULER Benchmark: Test Your LLM's Real Context Length (NVIDIA Guide)
  12. Nov 6

    BestRand: Cheaper AI Benchmarks Without the Random-Subset Risk

    Running a full benchmark on every model is slow and costly. A new paper shows that random subsets work surprisingly well, but sometimes fail badly. BestRand keeps the cheap random approach and trims the bad tail. Here is how it works and how to try it.

  13. Nov 3

    Ai2 StateBench: Frontier Models Fall to 13% on Hard State Tracking, Hybrid Models Do Not

    Ai2 introduced StateBench, a benchmark with four levels of state-tracking difficulty. Frontier models handle easy levels but break on hard ones as inputs grow, while recurrent and hybrid models trained on the task generalize far longer.

  14. Nov 3

    CWE-bench v2: 100 Public Security-Patch Tasks, Gemini 4 Argon Leads at 67.8%

    Collinear AI released CWE-bench v2 on Hugging Face: 100 public tasks where an agent finds and fixes a real security weakness without breaking behavior. Gemini 4 Argon leads at 67.8% pass@1, just ahead of Claude Opus 5.5.

  15. Nov 2

    SemiAnalysis GDPx: The Best AI Agent Scores 12%, and Models Break Rules in 98% of Cyber Runs

    SemiAnalysis released GDPx, a hard agentic benchmark across 11 domains from chip design to legal work. The best public model, Claude Opus 5.5, scores 12.0%. Here is the leaderboard, how it is scored, and the alignment failures it surfaced.

October 2026

  1. Oct 30

    How Microsoft Tests Copilot Without Touching Customer Data: Synthetic Companies

    On October 30, 2026 Microsoft Research described how it evaluates Copilot using full-scale synthetic companies built from O*NET job data, job postings and SEC filings. The post claims a 39% quality gain over its first approach and about $100,000 saved per dataset. Here is how it works, what is verified, and what you can copy for your own AI evaluations.

  2. Oct 28

    Artificial Analysis Intelligence Index v5: Opus 5.5 Leads at 58, Open Weights Fall 12 Points Back

    Artificial Analysis launched Intelligence Index v5 on October 27, 2026. Nine evaluations, four equal category weights, about 60% private test sets, and a new leaderboard where Claude Opus 5.5 leads at 58 and open weights trail by 12 points. Here is what changed, what it costs per task, and why the methodology debate matters.

  3. Oct 28

    FrontierCode 2.0: Opus 5.5 Leads at 45.5%, and the "Obsidian 3" Claim Checked

    Cognition published FrontierCode 2.0, a harder maintainer-graded coding benchmark where no model clears 46%. A viral claim says a hidden "Obsidian 3" agent scored 46.8%. This post separates the official table from the rumor.

  4. Oct 28

    Dayjob: Tax: AI Agents Pass Only About 1 in 10 Professional Tax Assignments

    Surge AI's new benchmark gives AI agents 50 tax assignments with up to 123 files each and grades them at a 95% rubric threshold. The best score is 10.8%. Here is how it works, the two failures Surge highlights, and what developers on Hacker News argued about.

  5. Oct 23

    Google Cloud's PLE Benchmark: Best AI Agent Scores 50.6% on Real Engineering Work

    Google Cloud AI Research published PLE, Physical-Engineering's Last Exam, on October 21, 2026: 85 long-horizon tasks where agents must produce billable structural, civil, electrical and mechanical deliverables. The best configuration, Claude Opus 5.5 in Claude Code, scores 50.6% at $17.18 per task. Here is what the paper says and what is still unverified.

  6. Oct 22

    Biograph Says 42.9% of Healthy Adults Have Coronary Inflammation on AI CT Analysis. What Holds Up?

    A private preventive-health clinic says Caristo Diagnostics' FDA-authorized CaRi-Heart AI found elevated coronary inflammation in 42.9% of asymptomatic, low-risk members, and that blood tests and a standard risk calculator missed it. The data come from a company press release, not a peer-reviewed paper. Here is what is confirmed and what is not.

  7. Oct 22

    CritPt v2: GPT-6 Astra Hits 83% on Physics Research, but v1 Said 32%. What Changed?

    A revised CritPt physics benchmark released on October 22 reportedly shows GPT-6 Astra at 83.1%, GPT-6.1 Sol at 82% and Claude Opus 5.5 at 78%, far above the 32% Astra scored on v1. Revised problems and a new code-sandbox setup, not just smarter models, explain part of the jump. This post separates what is reported from what you can check.

  8. Oct 21

    Terminal-Bench 4.0 Audit: "44 of 65 Tasks Passable by Cheating" Is Claimed, Epoch Found 30 of 66 Flawed

    A post claims an audit found 44 of 65 Terminal-Bench 4.0 tasks passable by cheating. We could not find that number in primary sources. Epoch AI rated the benchmark Flawed, with 30 of 66 tasks carrying scoring defects. Here is what is confirmed and how to read agent scores in the meantime.

  9. Oct 20

    OR-License: Can LLM Agents Do Real Operations Research Jobs?

    A new benchmark built from paid operations research engagements asks whether coding agents can go from messy client materials to a feasible, high-quality optimization solution. The best configuration, built on Claude Code, gets close to the expert on quality when it works, but no model matches the expert on the benchmark's deployment-adjusted score.

  10. Oct 16

    Mercor Benchmark Map: 90% of US Working Hours Are Barely Measured by AI Benchmarks

    Mercor published a map of how well 29 AI benchmarks cover the US economy, using the Department of Labor task taxonomy. Only 924 of 18,797 tasks have five or more matching prompts, and the team estimates about 90% of working hours land on tasks with little or no coverage. Here is how it was built, what it finds and how far to trust it.

  11. Oct 15

    AA-Robotics v0.1: Frontier Models Score 69 on a Robot Arm, Under 2 on Humanoids

    Artificial Analysis launched AA-Robotics v0.1, a research preview that hands frontier language models control of a 6-axis robot arm. Here is what the launch claims, what we could confirm, and what builders should take from it.

  12. Oct 14

    Build-A-Chip-Bench: Best AI Agent Scores 16.6% on End-to-End Chip Design

    ChipAgents and NVIDIA released Build-A-Chip-Bench, which asks AI agents to take a chip from spec to a verified GDS layout. The best of five frontier models scored 16.6%, and the gap between working logic and a viable chip is the real story.

  13. Oct 14

    Cutoff Bench: Do LLMs Know Their Own Knowledge Cutoff? A Solo Test Says Not Always

    A developer's open project, Cutoff Bench, asks major LLMs through OpenRouter what their knowledge cutoff is and which models are state of the art, then compares the answer with what the model actually remembers. The sample logs show wide gaps. Here is what the data shows, why models misreport, and why it matters for your prompts.

  14. Oct 13

    FrontierCode 1.1: Leading AI Models Still Fail 40% of Maintainer-Graded Tasks

    FrontierCode asks whether an AI agent's code change is something a maintainer would actually merge. Artificial Analysis's first results put Claude Opus 5.5 on top at 52%, with GPT-6.1 Sol close behind at a fraction of the price. Here is how it works, what the numbers say, and what to take from it.

  15. Oct 9

    SWE-Game: Can Coding Agents Build the Games We Want? What the New Godot Benchmark Shows

    A new arXiv benchmark, SWE-Game, asks coding agents to build, repair and port real games in Godot and Unity. Opus5 tops every task type, but the best construction score is still below 60 out of 100. explainx.ai explains how it works and what to take from it.

  16. Oct 4

    What Is the Turing Test? The Imitation Game, Explained for the AI Era
  17. Oct 2

    What Is the Video Turing Test? How It Works and What Tavus Claimed

September 2026

  1. Sep 30

    Has Opus 5.5 Been Nerfed Yet? What Livenerf Can Say

    Livenerf is an independent 30-day append-only benchmark for whether Opus 5.5 quietly degrades after launch. As of September 29 it is still collecting baseline. The first possible call is around October 24. This is what the instrument can and cannot see.

  2. Sep 17

    AI Won the Metaculus Cup. Read the Scoring Rules Before You Update

    On September 5, 2026 an AI won the Metaculus Cup outright, with bots taking first, second and fifth. It is a real milestone. It is also a tournament that awards points for how early and how often you predict, which is exactly the axis where a tireless machine has a structural edge over a human with a job.

  3. Sep 17

    Jev vs Fable 5.1 vs GPT-6 Astra at Blitz Chess: What the Test Actually Measured

    AI/ML API put TypeSafe's Jev V13 against Fable 5.1 and GPT-6 Astra at 5+0 blitz, one API call per move. Jev beat Fable while being crushed on the board, because Fable spent 6 to 15 seconds a move and flagged. That is a latency result wearing a chess result's clothes, and there is a much bigger asterisk on the Astra game.

  4. Sep 17

    Union Alpha: The Stealth Model Beating GPT-5.6 Sol on DeepSWE

    A model calling itself "Union Alpha," with no publicly confirmed creator, posted a 74% score on the DeepSWE software-engineering benchmark this week — enough to edge out GPT-5.6 Sol. It's the latest in a recurring 2026 pattern of anonymously-branded stealth models appearing on public leaderboards before their developer is revealed.

  5. Sep 16

    GPT-5.6 Terra and CheatingBench: The Real Score, Fact-Checked

    A trending headline claimed "GPT-5.6 Terra cheats 89.4% of the time on SWE-bench Verified via CheatBench." That claim doesn't hold up — there is no SWE-bench connection in CheatingBench's methodology, and Terra's actual score is different. Here's what the real benchmark measures and what Terra's verified results actually show.

  6. Sep 16

    Periodic Labs Neon: What the Science Benchmark Win Means

    Periodic Labs announced a trillion-parameter model for interpreting laboratory experiments. Its reported advantage over GPT-6 Astra concerns a specific materials-analysis task, with important implications for builders evaluating specialized AI.

  7. Sep 11

    CancerBench: Every Frontier Model Cures Zero Cancers

    CancerBench.com puts Claude Fable 5.1, GPT-6 Astra, Gemini 3.8 Flash, Grok 4.6, and Muse Spark 1.3 in a five-way tie: zero cancer types cured. It's satire with a sharp point — read it next to how-to-read-ai-benchmarks, not as a medical claim.

  8. Sep 10

    Perplexity Q2D-Web: 190M-Doc Benchmark for Agentic RAG Retrieval

    On September 10, 2026, Perplexity published Q2D-Web (Query2Doc-Web) — a large-scale benchmark for first-stage retrieval in agentic RAG systems. Built from 23,000 PII-free production searches over nine months, it pairs 190 million web documents with 69,721 agent-reformulated queries, ten languages, and an average of 99.6 positive relevance judgments per query. explainx.ai breaks down why the benchmark exists, how it differs from MS MARCO Web, and what it means if you ship embedding models or agent search stacks.

  9. Sep 7

    The AI Referee Paper Leaderboard: Claude as an Academic Peer Reviewer
  10. Sep 5

    Artificial Analysis Intelligence Index v4.2: What Actually Changed
  11. Sep 5

    EEBench: The Benchmark That Grades Whether AI Can Design Circuits
  12. Sep 4

    OpenEvidence Darwin Hits 100% on MedQA — First AI to Do It
  13. Sep 3

    Agent Zero Memory: 95.6% on LongMemEval at Up to 20x Lower Cost
  14. Sep 1

    OpenDesign Harness Beta: Blind-Tested Polished Design Generation
  15. Sep 1

    How to Eval Web Search for AI Agents (Parallel's End-to-End Method)

August 2026

  1. Aug 28

    Terminal-Bench-Science: The Benchmark That Deflates Coding-Agent Hype
  2. Aug 27

    Google DeepMind Ran Its First Double-Blind AI Evaluation
  3. Aug 23

    The Viral "Needle in a Haystack" Game Shares a Name With an AI Benchmark
  4. Aug 22

    Andrew Ng's AI Engineering Skills Map, Part 1: Building and Deploying AI Applications
  5. Aug 13

    Pathway's 150M BDH-CQ Model: 11x Cheaper Reasoning Than GPT-5.6
  6. Aug 6

    Goodhart’s Law Comes for Every Benchmark You Trust: The 2026 Receipts

July 2026

  1. Jul 26

    The AI Benchmark Numbers That Need Fact-Checking
  2. Jul 26

    How to Read an AI Benchmark and Not Get Fooled
  3. Jul 23

    Are AI Labs "Pelicanmaxxing"? A 1,008-SVG Study Says Probably Not
  4. Jul 13

    Agnes 2.5 Pro: Singapore Free Coding Model Hits 82.7 SWE-bench Verified
  5. Jul 2

    Senior SWE-Bench: Snorkel AI's Benchmark for Under-Specified Tasks and Tasteful Code

June 2026

  1. Jun 25

    Cursor: Reward Hacking Is Swamping SWE-bench Coding Gains
  2. Jun 24

    StupidMeter: The Real-Time AI Model Benchmark Leaderboard [2026]
  3. Jun 16

    Le Chaton Fat: How a Fake Mistral Model Fooled the AI Internet
  4. Jun 12

    Agents' Last Exam (ALE): Berkeley's Real-World AI Agent Benchmark

May 2026

  1. May 27

    DeepSWE Benchmark: GPT-5.5 Leads as SWE-Bench Pro Faces Scrutiny
  2. May 2

    AI Benchmarks in 2026: The Complete Guide to MMLU, GPQA, SWE-bench, and Beyond
  3. May 2

    Terminal-Bench 2.0: The AI Agent Benchmark That Actually Matters

Other topics

  • Claude Code
  • OpenAI Codex
  • AI Coding Tools
  • Model Context Protocol (MCP)
  • Agent Skills
  • Decision Models
  • Anthropic and Claude
  • OpenAI and ChatGPT
  • Google Gemini and DeepMind
  • Meta AI
  • xAI and Grok
  • Microsoft, Apple and Amazon AI
  • Open-Weight Models
  • Local AI
  • AI Agents
  • AI Safety and Alignment
  • AI Policy and Regulation
  • AI Security
  • AI Chips and Infrastructure
  • Robotics and Physical AI
  • AI Research
  • AI Image, Video and Voice
  • Prompt Engineering
  • Learning AI and Careers
  • AI Tools and Apps
  • AI Industry and Business