Google AI Scientist at ICML 2026: ScientistOne, Chain-of-Evidence, and the End-to-End Research Pipeline
Google Research demoed "AI Scientist" at ICML 2026 — the ScientistOne pipeline with Problem Investigator, Parallel Explore-Exploit discovery, and Claim Verifier. CoE audit results, vs Co-Scientist, and what X is asking.
On July 7, 2026, Google Research posted on X inviting ICML attendees to "Experience AI Scientist" — a multi-agent system meant to automate the end-to-end scientific research pipeline. The booth slot was 6:00pm at #B206, with Jinsung Yoon and Rui Meng listed as contacts, tagged #ICML2026.
That marketing name — "AI Scientist" — lines up with an ICML Expo Talk Panel from the day before: The End-to-End AI Scientist: Automating Discovery and the Research Pipeline (July 6, 2026, Hall D2), same authors. The published research system with the matching three-stage architecture is ScientistOne — Towards Human-Level Autonomous Research via Chain-of-Evidence (Google Cloud AI Research, arXiv 2605.26340).
This post maps the booth narrative to verified paper claims only. No invented benchmark numbers, no conflating separate Google research products.
No.MARS = budget-aware MCTS AutoML agent (shared authors). ScientistOne uses MLE-Bench only as a generalization test.
Does it kill PhD jobs?
The system automates pipeline mechanics on bounded benchmarks — authors still note limits on novelty, significance, and wet-lab domains. Humans remain in the loop for real discovery judgment.
"Slop in, slop out"?
Fair skepticism. CoE Audit exists because prior "AI Scientist" systems produced fluent but unverifiable papers — up to 21% hallucinated references in baselines. ScientistOne's claim is architectural provenance, not guaranteed breakthroughs.
Token spend at Google?
Experiments standardized on Gemini 3.1 Pro with generous iteration budgets (paper notes up to 20 solver iterations per task for baselines). Replicating full runs outside Google infra is costly — treat public numbers as research-scale, not laptop-scale.
Failed experiments?
PEE prunes losing branches, filters spec violations, feeds metrics back across I × B iterations — details in Discovery stage.
Rediscovering 1987 work?
Claim Verifier checks citations against retrieved bibliography; CoE does not certify scientific novelty — see FAQ on prior art.
Three names, one booth — keep them straight
Google's autonomous-research story is not one monolith:
Separate ICML 2026 research (Yoon, Meng, Pfister, et al.)
The ICML panel abstract also mentions ScholarPeer (automated peer review with a "historian" literature approach for novelty) and PaperBanana (methodology figure generation). Those appear in the talk abstract; ScientistOne's paper evaluates manuscript quality with ScholarPeer but centers verification on Claim Verifier + CoE Audit inside the pipeline.
The pipeline — Problem Investigator → Discovery → Paper + Claim Verifier
The architecture Google showed at ICML matches ScientistOne Figure 1 and the booth diagram below:
Stage 1: Problem Investigator
Inputs: task definition + seed papers.
Process (from arXiv §4.1 and Appendix B):
Build a citation graph via scholarly database queries.
Filter to an elite pool (~500 papers) with tier scores for methodology relevance and problem alignment.
Run multi-round investigation — Librarian, Researcher, and SubdomainWriter agents across ~3 rounds targeting ~100 structured paper notes and 5–15 research directions.
Emit an Experiment Brief (research landscape, concrete experiment plan, 25–40 references traceable to read PDFs).
Design intent: citations must come from retrieved full text, not model memory. In CoE Audit, systems without retrieval grounding hit up to 21% hallucinated references; ScientistOne reports 0/337.
This is the opposite of "prompt and pray" — closer to Mollick-style specs: define the task, define evidence, define how you will test claims.
Ideator: reads the PI brief, generates candidate approaches, scores novelty and feasibility, expands top ideas into proposals.
The booth diagram labels conservative vs unconventional generation tracks before proposals fan out. The arXiv text describes novelty/feasibility scoring and distribution across branches — treat C/U as visual shorthand for diverse ideation, not a separate published benchmark metric.
Parallel Explore-Exploit (PEE): proposals scatter into parallel branches (B1…B4 in the diagram; paper uses configurable B branches). Each branch runs:
After each iteration, top-K branches survive; remaining slots refill from fresh ideation on winners. Loop for I iterations × B branches. A best-run selector picks the highest-scoring clean solution, then runs ablations.
What happens to failed experiments? They are expected and discarded — pruned branches, spec-violation drops, and low scores never reach Stage 3. Feedback (distillation, metrics) can inform the next ideation round. This is loop engineering at research scale: explore widely, exploit winners, stop on violations.
Stage 3: Paper Writer + Claim Verifier
Paper Writer — five stages (arXiv §4.3, Appendix B.4):
Stage
Role
Conceive
Build a markdown "research representation" with inline evidence tags on every factual claim.
Ground
Deterministic validation — scores match logs, baselines trace to PI brief, artifacts exist.
Critic
LLM audit for gap–approach alignment, overclaims, missing comparisons.
Resolve
Rewrite or drop unsupported claims; loop until convergence or abort if grounding ratio too low.
Compose
Per-section LaTeX writers — provenance before prose.
Claim Verifier: extract claims → verify against declared evidence (numeric tolerance on logs, bibliography + abstract entailment for citations, log overlap for methods) → refinement pass rewrites or removes failures → final_paper.pdf only if no blocking violations remain.
That directly targets the "slop" failure mode: pretty PDFs where numbers, citations, and code disagree.
Chain-of-Evidence (CoE) and the Integrity Audit
ScientistOne's thesis: verifiability is architectural, not a post-hoc spell-check.
CoE requires every claim to trace to code, data, or literature through the full run.
CoE Integrity Audit (post-hoc, cross-system) runs four checks on deliverables (paper.tex, solution code, references.bib):
Check
What it catches
I1 Score Verification
Paper score vs golden evaluator re-run
I2 Specification Violation
Metric gaming, invalid submissions
I3 Reference Verification
Hallucinated or mismatched citations (API + LLM disambiguation)
I4 Method–Code Alignment
Paper describes algorithm X, code implements Y
Headline results (75 papers, five systems, five ADRS tasks — project site):
System
Score Verif.
Spec Violation
Halluc. Refs
Method–Code
Sakana ASv2
5/12
10/15†
0/159
5/15
AutoResearchClaw
5/12
0/15
3/196
3/15
DeepScientist
11/12
0/15
42/201
5/15
AI-Researcher
9/12
1/15
21/222
12/15
ScientistOne
12/12
0/15
0/337
14/15
† Sakana cross-system I2/I4 comparisons are confounded by scaffolding in submitted code — see paper.
Solver performance still matches or beats human expert baselines on ADRS tasks; generalization tests include MLE-Bench (e.g., Gold on 3D Object Detection where DeepScientist scored 0.0000) and Parameter Golf SOTA as of the April 27, 2026 cutoff.
What the Claim Verifier does — and does not — catch
X threads asked whether Claim Verifier flags accidental rediscovery — e.g., reinventing a 1987 result.
What it does verify (per paper):
Citation keys resolve to real publications retrieved into the bibliography.
Abstract entailment for specific citation assertions.
Numerical claims match evaluator logs within tolerance.
Method descriptions align with experimental logs and (via CoE I4) solution code.
What it explicitly does not guarantee:
ScholarPeer serves as a scalable proxy for review quality but does not replace human expert evaluation … CoE Integrity Audit … is itself limited to structural integrity, not scientific novelty or significance.
So: if the Problem Investigator never retrieved the 1987 paper, the pipeline will not magically flag "you rediscovered Smith et al. 1987." If that paper is in the PI brief and the draft claims novelty anyway, Critic and ScholarPeer-style review are the layers meant to push back — still imperfect.
For the agent vs agency debate — whether tools like this are "co-scientists" or autonomous agents — see Eric Xing's Critique of Agent Model: marketing labels outrun what the harness actually controls.
"Auto research race" — who else is in the pool?
July 2026 social chatter compared Google and Anthropic. Map that to what shipped publicly:
Lab / project
Focus (verified)
Google ScientistOne
End-to-end papers + CoE on ADRS / MLE-Bench
Google Co-Scientist
Hypothesis partner for biologists — human executes experiments
Sakana AI Scientist v2
Baseline in CoE Audit — strong solvers, weak reference integrity
Adaption AutoScientist
Automated model training loop — different problem (our guide)
Anthropic (July 2026)
Interpretability (J-space, J-lens, NLAs) — not an end-to-end paper factory
Karpathy autoresearch
Minimal self-improving training loop — inspiration cited across the ecosystem
The race is real on harness + verification, not on replacing peer review overnight. Agent harness engineering already showed scaffolding beats model swaps on Terminal-Bench; ScientistOne argues the same for research integrity.
ICML panel extras — ScholarPeer and PaperBanana
The ICML panel abstract adds components beyond the core ScientistOne diagram:
ScholarPeer — automated peer review; panel mentions a "historian" approach to check novelty against historical literature (ScholarPeer arXiv:2601.22638, cited in ScientistOne for review scores).
Architecture, audit tables, and benchmark numbers follow Google Cloud AI Research's ScientistOne paper and project site as of July 7, 2026. ICML booth branding ("AI Scientist") may differ from arXiv product naming; verify against primary sources before citing in academic or investor materials.
