TheoPhysEval: GPT-6-Astra Solves 49 of 50 Physics Research Problems

A new benchmark built by physicists says the best AI agents can now do most of a working theoretical physicist's routine calculations. TheoPhysEval has 50 unpublished research-level problems, and GPT-6-Astra running in OpenAI's Codex solved 49 of them, about 98 percent, at an average cost under 3 dollars per problem. The paper is on arXiv (2610.16733), the problems and test setup are on Hugging Face, and The Decoder put the figure at about 2.40 dollars apiece.

The headline needs care. These are "turn-the-crank" problems, not new discoveries, and the authors themselves point to memorization. But the result is a useful data point for anyone following GPT-6-Astra and the race with Claude Opus 5.5.

TL;DR: the numbers

Scores and costs come from the paper as summarized by The Decoder; check the arXiv paper for exact tables.

How was the TheoPhysEval benchmark built?

The authors wanted problems that cannot be copied from a paper. Contributors wrote 50 new problems, often by changing a few values or conditions in a calculation from their own recent work, so the method is known but the answer is not published anywhere.

The paper describes a typical problem as one a leading expert could solve in about a day or less, while specialists in neighboring fields would need much longer. Each agent received no human hints and ran twice: once with web search and once offline. Agents could write scripts for numerical and symbolic work.

This design echoes the debate we covered in Nature's work on AI designing physics experiments: AI is clearly useful in physics, but the open question is what kind of work it is doing.

Who scored what, and why the gaps matter

The spread between models is wide. GPT-6-Astra and Opus 5.5 are close, but the next tier drops sharply: Gemini 3.8 Flash at 68 percent, the open-weight DeepSeek-V4.1-Flash at 54 percent and GLM 5.3 at 42 percent.

The Decoder reports a revealing failure mode: on a black hole problem, Gemini and GLM copied five-dimensional formulas unchanged when the task required four dimensions. That is a pattern-matching error, applying a remembered formula without checking that its assumptions match the question.

If you are choosing a model for technical computation, the useful comparison is the cost-adjusted one. Our GPT-6.1 Sol vs Astra cost analysis covers how to think about price per solved task, and the GPT-6-Astra vs Claude Fable 5.1 comparison covers the head-to-head.

Astra's one miss, and the answer-key corrections

Astra's single failure was a problem asking for an equation that generates a given curve. Every model failed it. The authors concede it may look unfair but argue it reflects realistic research tasks.

There is a second detail worth noting. After follow-ups with experts fixed small errors in the reference solutions, Astra's answers were correct every time, according to The Decoder. That suggests the true ceiling on this set may be limited by the answer key rather than the model. It also fits a separate study cited by the same article, which found that some theoretical physics benchmarks are graded incorrectly.

The memorization finding

This is the most important result in the paper. When the authors removed web search and background papers, accuracy changed only slightly. Astra and Opus both held 92 percent offline.

The authors read this as evidence that leading models have absorbed much of the theoretical physics literature. For a reader, the implication is twofold. First, the models are genuinely strong at recalling and adapting known methods. Second, a high score on a modified-from-the-literature problem does not tell you the model can invent a new method. Related work such as OpenAI's Navier-Stokes agent-swarm result and the Epoch capabilities index are better places to look for signals of novelty and general capability.

A useful way to think about it: a student who has read every textbook and paper can still ace a problem set made of variations on known derivations. That does not make them a researcher, but it does make them a very fast and cheap calculator for the routine part of the job, which is often where weeks go.

How does it compare with CritPt and other physics tests?

The paper says agents score higher on TheoPhysEval than on CritPt, a physics-research benchmark, though the authors consider their problems harder on average. That is a notable claim: either TheoPhysEval problems are easier to solve in practice because they are well-posed modifications of known work, or newer models are simply better than those tested on CritPt. The two are not mutually exclusive, and the authors do not claim a controlled comparison.

Benchmark saturation is the real risk here. With 49 of 50 solved, the set no longer separates the top agents, and the authors expect rising compute needs for harder problems. They suggest research budgets may start to matter in theoretical physics the way they do in machine learning.

Why the benchmark design matters as much as the score

Most public benchmarks leak. Once a problem set is online, later models can train on it, and scores climb for reasons unrelated to ability. TheoPhysEval tries to avoid that in three ways: the problems are unpublished, they are tweaks of the contributors' own recent calculations, and the answers are verifiable rather than judged by another model.

Each choice has a cost. Unpublished problems stop being unpublished the moment the dataset goes on Hugging Face, so the set has a limited shelf life as a clean test. Modified calculations favor models that have read the underlying papers, which is exactly what the offline result shows. And verifiable answers restrict the set to problems with a single checkable output, leaving out the open-ended reasoning that makes up much of real research.

Spanning 11 subfields helps. A model that only knows one corner of physics would not reach 98 percent across the set, so the result is evidence of breadth rather than a trick in one area. Even so, 50 problems is a small sample: one problem is two percentage points, which is why the gap between Astra's 98 percent and Opus's 92 percent is only three problems, and why small differences between top models should not be over-read.

What the authors say is still missing

The authors argue the strongest models may already beat top human researchers at concrete calculations. What they say is still missing is building theories and choosing good questions, which they call taste. That echoes OpenAI's own framing in its research acceleration push: tools that speed up the work, with humans choosing the direction.

How can you check the result yourself?

Because the dataset and test setup are public, a reader with API access can reproduce part of it. A sensible approach:

1. Download the dataset from the Hugging Face page and read the problem format.
2. Run one or two problems in your agent of choice, once with search and once without, as the authors did.
3. Compare against the reference answers, and note that the authors corrected some small errors in the key, so check the latest version.
4. Track the token cost per problem. The reported 2.40 dollar average is a useful yardstick for your own harness.

If your results differ a lot from the paper, the likely causes are the agent scaffold (Codex and Claude Code are different harnesses), the reasoning effort setting, and the tool permissions allowed.

What this means for what you build or pay

• For researchers: a 116-dollar full-benchmark run suggests routine derivations and checks are now cheap to delegate, as long as a human verifies results against known limits.
• For tool builders: the offline result means you may not need retrieval for well-known physics, but you do still need verification, such as unit checks and numerical cross-checks.
• For evaluators: treat a modified-literature benchmark as a memorization-plus-adaptation test. Pair it with genuinely new problems.
• For budgets: the open-weight models trailed by 40 to 55 points here, so cost savings from cheaper models carry real accuracy loss on this kind of task.

Caveats and what is not verified

• OpenAI has not, in the sources we found, published its own announcement of this result; the numbers come from the independent paper.
• Figures for models other than Astra come from The Decoder's summary; confirm them in the paper.
• The roughly 2.40 dollar figure is a reported average; the arXiv abstract says only under 3 dollars.
• Agent scaffolds differ (Codex for Astra, Claude Code for Opus), so the comparison is of products, not raw models.

Related reading

• GPT-6-Astra launch: benchmarks and pricing
• Claude Opus 5.5 launch: benchmarks and pricing
• GPT-6-Astra vs Claude Fable 5.1
• GPT-6.1 Sol and Astra cost efficiency
• OpenAI Navier-Stokes agent swarm
• AI designing physics experiments (Nature)

Scores and costs are as reported on October 15, 2026 and may be revised in later versions of the paper.