CritPt v2: GPT-6 Astra Hits 83% on Physics Research, but v1 Said 32%. What Changed?

A revised version of the CritPt research-physics benchmark landed on October 22, and the reported scores look dramatic: GPT-6 Astra (max) at about 83%, up from roughly 32% on the original. A CritPt author framed the longer arc as progress from about 29% in late 2025 to 83% now. But a score jump this large between versions of the same benchmark begs the obvious question: did the models improve, or did the test change? According to the reports, mostly the test changed. This post separates the reported numbers from what we could verify.

TL;DR

What CritPt measures

The CritPt paper on arXiv describes 71 composite research challenges designed to simulate entry-level research projects, decomposed into 190 simpler checkpoint tasks. They were written by more than 50 active physics researchers from their own work, and cover condensed matter, quantum physics, astrophysics, high energy physics, fluid dynamics, biophysics and more. Every problem has an answer format that is hard to guess (arrays, symbolic expressions, Python functions) and is graded automatically.

The Artificial Analysis CritPt page adds that each challenge took 40-plus hours of review on average, and that the paper's original headline was bleak: the best base model, GPT-5 (high), scored 4.0%, rising to roughly 10% with coding tools. That is why a headline of 83% sounds like a different universe.

What the v2 reports say

Pulling together the posts that circulated on October 22 (from Artificial Analysis and a CritPt author), the claims are:

• v2 is a set of 70 research-level challenges released October 22.
• The team revised 47 of them to fix mistakes and underspecification, after reports from Anthropic and Hugging Face.
• Models are now evaluated agentically with a code-execution tool in an offline sandbox.
• Top scores: GPT-6 Astra (max) 83.1%, GPT-6.1 Sol (max) 82%, Claude Opus 5.5 (max with fallback) 78%, GPT-5.6 Sol (max) 75%.
• On v1, GPT-6 Astra scored about 32%.
• Artificial Analysis reportedly said the increase is primarily due to the revised problem set rather than the added coding tools.
• The CritPt team offers free access to its grading API for approved labs, and Artificial Analysis plans to update its Intelligence Index in late October to include the updated figures.

We checked the Artificial Analysis CritPt v2 page, which confirms the top three: GPT-6 Astra (Max) 83.1%, GPT-6.1 Sol (Max) 82.0% and Claude Opus 5.5 (Max, Default Fallback) 77.7%, with 11 models listed. The page describes v2 as a revision in which the authors and physics experts "corrected underspecified statements, errors, and typos," and says models solve problems with a code-execution tool in an offline Linux sandbox. The fourth-place GPT-5.6 Sol figure (75%), the count of 47 revised problems and the attribution of the jump to the revised problems rather than tools come from posts on X, which we did not verify. The v1 page states that its results run all 70 problems without an agent harness, which is itself a large difference from v2's tool setup.

Why a benchmark can jump from 32% to 83%

Three separate things changed at once, and it is worth listing them because they stack.

1. Problem fixes. If a problem is underspecified or its reference answer is wrong, even a perfect solver loses points. Fixing 47 of 70 challenges removes that ceiling. A model that "failed" a flawed problem can now pass it without being any smarter.
2. Tools. Giving models a sandboxed code tool lets them check arithmetic, run simulations and test candidate formulas. Artificial Analysis's v1 page says its v1 runs used no agent harness, while v2 uses a code sandbox. The reports say tools are not the main driver, but we could not verify that, and it changes what is being measured: model plus tool, not raw reasoning.
3. Model progress. Real, but entangled with the first two. Only a re-run of the same v2 problems on older models would isolate it, and the figure of 28.9% for GPT-5 (high) with a coding tool in November 2025 comes from v1, not v2.

The practical lesson is the one we have made before about Artificial Analysis Index revisions: never compare scores across benchmark versions, and check whether the problems, the tools or the grader changed.

Reading the leaderboard order

The reported ranking has GPT-6 Astra and GPT-6.1 Sol within a point of each other (83.1% vs 82%). With 70 problems, each problem is worth about 1.4 percentage points, so a one-point gap is less than one problem. Without confidence intervals or repeated runs, treat the top two as tied. Claude Opus 5.5 at 78% is about three to four problems behind. Our own head-to-head posts, such as Gemini 4 Argon vs Opus 5.5 vs Grok 4.7 vs GPT-6 Astra, show how often small gaps like this flip across benchmarks.

One commenter on X also pointed to GPT-6.1 Sol delivering these results at a cost similar to a cheaper open model; that is one person's observation, and we did not verify any cost-per-task figures.

How this compares with other physics evals

CritPt is not the only physics test. TheoPhysEval reported GPT-6 Astra solving 49 of 50 problems at a cost of $2 to $40 each, using a different problem set and grading approach. The two tests converge on a similar story: on well-posed, checkable physics problems, frontier models are now very strong. They differ in construction, so you cannot add or average them.

What this does and does not mean

• It does mean that on structured physics problems with verifiable answers, top models with code tools now clear most of a benchmark that stumped them a year ago.
• It does not mean models can independently do physics research. Real research involves choosing questions, handling ambiguous setups and judging whether an answer is plausible, none of which a guess-resistant, machine-graded test captures.
• It does mean benchmark maintenance matters. External reports from Anthropic and Hugging Face led to fixes, which is how benchmarks should evolve, and also why scores before and after are not comparable.
• It does not mean the 83% is fully stable. Benchmark answers can still contain errors, and models could have seen related material.

A checklist for any revised benchmark

CritPt v2 is a good worked example of a pattern you will see again, so here is a short checklist you can reuse whenever a benchmark ships a new version with higher scores.

1. Count what was revised. Here 47 of 70 problems changed, which is two thirds of the test. That is closer to a new benchmark than a patch.
2. Ask who found the errors. The reports credit Anthropic and Hugging Face. External error reports are a sign of healthy maintenance, but the labs reporting errors also have an interest in how their own models score, so look for the list of changes.
3. Separate tool access from reasoning. An offline code sandbox is a legitimate setup, but a score with tools answers a different question than a score without them.
4. Look for a bridge run. The most useful number is the same models on v1 and v2 side by side. Astra at 32% on v1 and 83% on v2 is not a clean bridge, because the tool setup differs between the two runs. A v1 problem set run with the v2 sandbox, or the reverse, would isolate the cause.
5. Check problem count and variance. With 70 problems, one problem is about 1.4 points, and a single run at maximum reasoning effort can vary.
6. Wait for the index. Artificial Analysis says it will fold the updated figures into its Intelligence Index in late October, which will show how the revision shifts rankings across all models, not only the top three.

For builders choosing a model for scientific work, the takeaway is practical rather than philosophical. If your tasks look like CritPt, with clearly specified inputs, checkable outputs and room to run code, the reported results suggest the leading models are now useful assistants. If your tasks are open-ended, such as deciding which experiment to run next, this benchmark says little, and you should test on your own examples.

If you want to check yourself

1. Read the paper for problem structure and the grading method.
2. Open the Artificial Analysis CritPt page for the current leaderboard and methodology once v2 is published there.
3. If you are an approved research lab, request the free grading API through the address listed on that page and run your own models.
4. Report scores with the version label (v1 or v2), the tool setting and the reasoning effort, for example "max".

Related reading

• TheoPhysEval: GPT-6 Astra solves 49 of 50 physics problems
• Artificial Analysis Intelligence Index v4.2
• Gemini 4 Argon vs Opus 5.5 vs Grok 4.7 vs GPT-6 Astra
• Claude Sonnet 5.5 vs GPT-6 Astra
• GPT-6.1 Sol and Astra cost efficiency
• Epoch Capabilities Index and Opus 5.5

Scores are as reported on October 22, 2026 and may be revised when the v2 leaderboard is fully published.