AI Peer Review Benchmark: 100 Planted Errors in Psychology Papers

Scientific teams increasingly run drafts through frontier models before human peer review. Until November 2026, there was little public ground truth for how well those tools catch methodological mistakes.

Paul Litvak's benchmark, released on GitHub under the Dawes Institute with an arXiv paper, plants 100 known errors across 10 open-access psychology papers, then scores 14 configurations spanning Claude, GPT, Gemini, Refine.ink, and Reviewer3.

TL;DR

How the benchmark was built

Litvak describes an iterative process on his writeup:

1. Start from real published open-access psychology papers (originals preserved in original papers/).
2. Build a 62-subcategory taxonomy of detectable issues weighted by replication relevance (refine_taxonomy_v4.md in the repo).
3. Plant 10 errors per paper — both insertions (wrong stats, flipped effect directions) and deletions (missing disclosures, removed transformations).
4. Reject an early version that was too easy and saturated foundation models instantly.

Warning in the repo: modified papers/ are deliberately corrupted watermarked derivatives — never cite them as real literature.

The ground-truth table lives in error_insertions.csv with original text, modified text, category, and description.

Type-safe guardrails concept for checking planted errors in scientific manuscripts

Systems tested and scoring harness

The benchmark/ directory includes:

• Provider adapters for Anthropic, OpenAI, Google
• Integrations for commercial tools Refine.ink and Reviewer3
• Scoring via fuzzy matching plus an LLM judge comparing reviewer comments to planted errors
• benchmark/writeup_numbers.py to reproduce headline metrics

For foundation models, Litvak reran papers three times to gauge stability; reruns did not surface many new catches (temperature sweeps could go further).

Results builders should internalize

Single-system ceiling: Even the best reported individual model landed at 71% recall on planted issues — far from a safety-critical peer review replacement.

Ensembling lever: Because models catch partially uncorrelated errors, unioning 14 configurations reached 93/100. For lab workflows, that supports multi-model review lanes (different vendors, different prompts) rather than one Claude or GPT pass.

Omission blind spot: All seven errors no system caught involved missing information — deleted methodology, not wrong numbers present on the page. That matches human peer review pain points but is worse for LLMs trained to critique visible text.

Commercial tools: The writeup notes Refine.ink contributed many unique catches but at high per-error cost; rankings among close systems should be read as patterns, not precision leaderboards, given small n=100 and model version drift since the original run.

Contamination and leaderboard hygiene

The README warns that because ground truth is public, future models may train on modified papers. Scores for models released after August 2026 should be treated as potentially contaminated. Litvak frames the release as a methods demo portable to other fields with open-access corpora — not an eternal leaderboard.

That caution parallels how we report claimed vs verified vendor benchmarks elsewhere on explainx.ai.

Commercial tools versus foundation models in the same harness

Litvak's sweep places Refine.ink and Reviewer3 beside frontier APIs using the same planted manuscripts. Reported patterns worth noting even if exact ranks drift:

Society deals: Litvak ties urgency to economics and psych societies exploring AI screening. If your startup sells AI peer review, expect contract clauses demanding published evals like this one within 12 months.

Teaching use: Professors can assign students to find planted errors manually before revealing error_insertions.csv — a pedagogical bridge between AI in coursework integrity and real methods training.

How to run or extend it

git clone https://github.com/Dawes-Institute/ai-peer-review-benchmark.git
cd ai-peer-review-benchmark
# Configure keys per benchmark/README, then run harness scripts under benchmark/

Extension ideas for ML engineers:

• Port the harness to arxiv PDF ingestion for CS conferences
• Add false-positive scoring against unmodified originals (still missing)
• Pair with Opus formalization workflows for math-heavy venues

Limitations to disclose in your own evals:

• Field is psychology only
• No FP rate — systems may over-flag benign issues
• Planted errors may be harder or easier than organic mistakes

Taxonomy depth: what kinds of errors were planted

The repository's refine_taxonomy_v4.md organizes 62 subcategories across 10 top-level classes linked to replication risk. Planted errors covered 50 subcategories — not the full taxonomy, but wider than a single-stats checklist.

Representative insertion failures systems caught more often:

• Sign errors — flipped effect directions contradicting narrative.
• p-value gymnastics — inconsistent test reporting vs methods text.
• Causal overclaim — observational designs described as experiments.

Representative omission failures that defeated everyone:

• Removed data-preparation sentences — reviewers must notice missing steps.
• Deleted exclusion criteria — invisible unless you know the original paper.
• Dropped disclosure language on conflicts or preregistration.

That split explains why GPT-5.5 at 71/100 still misses nearly a third — and why human peer review focused on methods archaeology remains load-bearing.

Reproducing headline numbers safely

From the README workflow:

1. Clone Dawes-Institute/ai-peer-review-benchmark.
2. Never cite paths under modified papers/ in real manuscripts.
3. Configure API keys per benchmark/ provider docs.
4. Run benchmark/writeup_numbers.py after your sweep to compare against published 71 / 30 / 93 figures.

Judge dependency: Scoring uses LLM-as-judge alignment between free-text reviewer comments and error_insertions.csv. If your judge model drifts, reproduced scores shift — log judge version.

Ensembling recipe for labs:

• Run Claude, GPT, and Gemini with different system prompts (methods-first vs stats-first vs ethics-first).
• Union unique flagged spans; human triage only on disagreements — cheaper than triple human review.

Policy and product context

Commercial AI review is moving from informal Slack usage to society partnerships — Litvak notes Refine.ink deals with economics societies. Benchmarks like this will shape publisher due diligence the same way Australia's mandatory AI incident reporting is shaping deployment governance.

For coding paper pipelines, compare to Alibaba Open Code Review — different modality, same question: what does automated critique actually catch?

Workshop context and why psychology first

The arXiv note lists acceptance at Cognitive Security for Human-AI Systems (CogSec 2026), a non-archival workshop at ACM HCOMP / CI 2026. Psychology papers were chosen because open-access originals with clear methods sections make controlled corruption feasible without copyright fights.

Porting to ML conferences means navigating double-blind norms — you cannot plant errors in submitted PDFs without editorial permission. Safer ports use synthetic papers or withdrawn preprints with author consent.

Cognitive security framing: Peer review is a human–human trust ritual. AI assists sit in the supply chain attackers may poison with subtle omissions — the benchmark's seven universal misses are the scary class for grad students trusting one ChatGPT pass before submission.

Frequently misunderstood scores

Readers often misread 71/100 as "71% accurate reviewer". Litvak's metric is recall on planted errors, not precision on all comments. A model might emit fifty plausible-sounding critiques with 29 missed plantings and unknown false positives.

Similarly 93/100 union does not mean ensemble is production ready — it still missed seven omissions, and real papers may contain errors outside the 62-category taxonomy.

Version drift: The writeup warns models and commercial APIs upgrade constantly. Re-run the harness quarterly if you cite numbers in procurement docs.

Related on explainx.ai

• Opus 5.5 CPython and Lean formalization bugs
• Findendro temporal self-knowledge for finance backtests
• Claude Fable 5.2 METR RSI evaluation
• OpenAI agents spoofed tool calls in evals
• Australia mandatory AI incident reporting
• AI labs output token inflation pricing
• Semantic caching for LLM app cost

Sources: arXiv:2611.12820, GitHub repository, Paul Litvak writeup.

Quick FAQ for lab managers

Should we block AI review tools? The benchmark says tools help but do not replace methods experts — especially on omissions.

Can we use this in hiring? Yes for eval engineer interviews: ask candidates to design new plantings and scoring rules.

Will journals accept AI-assisted review? Policy is moving faster than benchmarks; cite CogSec lineage when proposing internal pilots.

How big is the error budget? One hundred plantings across ten papers is enough to expose systematic blind spots but not to rank models within three points with confidence — treat close scores as ties.

Ethics reminder: The modified manuscripts are watermarked derivatives. Do not submit them to journals, post them as preprints, or use them as negative SEO against original authors. The benchmark exists to score reviewers, not to pollute the literature.

When you present results to PIs, show recall on plantings and comment volume side by side so they see that chatty reviewers are not necessarily thorough reviewers.

Archive API model IDs in the same JSON as scores so you can explain a sudden jump six months later.

Scores and system list follow the repository EVAL_WRITEUP as of November 2026.