GPT-6-Astra solves 49 of 50 physics research problems at $2.40 apiece

The team tested commercially available AI agents and gave them no human hints. GPT-6-Astra, running in OpenAI's Codex coding environment, came out on top. It found the right answer for 49 of 50 problems, or 98 percent, at an average cost of under $3 per problem and just under $116 for the whole benchmark.

The benchmark tests routine research work, not grand challenges

The collection deliberately leaves out major open questions and focuses on what the authors call "turn-the-crank" problems, where researchers apply known methods to a slightly modified question. Contributors often tweaked a few values in a calculation from one of their own recent papers so the new result couldn't be guessed from the old one. According to the researchers, problems like these are often handed to PhD students as a first step into research.

An example from black hole research shows how demanding these problems still are. Solving it means finding an older paper, carrying its equations over from a five-dimensional to a four-dimensional spacetime, and then computing the result with a custom program. Coordinator Suvrat Raju, a co-author of the older paper, needed about a day. The authors estimate that before modern agents arrived, only a handful of experts had both the physics knowledge and the experience with these computational methods. Even they would have needed several days.

The problems span eleven subfields of theoretical physics, from fluid flows and biological processes to particle physics and cosmology. Each agent ran the test twice, once under realistic conditions with web search and once in isolation without internet access.

Anthropic's Opus 5.5, running in Claude Code, came in second at 92 percent. Google's Gemini 3.8 Flash scored 68 percent, while the open-weight models DeepSeek-V4.1-Flash and GLM 5.3 reached 54 and 42 percent. The weaker models sometimes made mistakes an expert would have caught right away. On the black hole problem, Gemini and GLM copied the formulas from the older paper unchanged during the web search run. Those formulas apply to five dimensions, not the required four.

Internet access made surprisingly little difference. Without web search, Astra and Opus each scored 92 percent. Many problems depend on formulas from research papers that the models usually didn't receive. The authors take this as a sign that leading language models have effectively memorized large parts of the theoretical physics literature.

With web search, Astra solved everything except one problem that stumped every model. It asked for an equation that produces a given curve. The experts' solution relies on special mathematical forms that only count as the required differential equation in a broader sense. The authors admit the problem might seem "unfair," but they consider it realistic because researchers often have to infer the underlying physics from imprecise data.

Several AI misses turned out to be flaws in the answer key

For several problems, follow-ups with the experts revealed small, hard-to-spot errors in their reference solutions. Once those were fixed, Astra's answer was correct every time.

This pattern could also explain an apparent contradiction. The agents do better on TheoPhysEval than on some other physics benchmarks, such as CritPt, even though the authors consider their own problems harder on average. A recent study found that some theoretical physics benchmarks are graded incorrectly and in some cases badly underestimate what the models can do.

Anyone who wants to check the results can download the problems and test setup as a dataset on Hugging Face.

Why strong calculations haven't led to physics breakthroughs

The authors write that some of the strongest current models may already have surpassed the best human researchers on concrete calculations, even in specialized fields. They call this a major milestone.

The impact is even more visible in mathematics, they note, where AI agents have already solved important open problems. According to OpenAI, that includes the Navier-Stokes problem about the equations that describe fluid flow, one of the famous Millennium Prize Problems. Theoretical physics hasn't seen a comparable breakthrough yet.

The authors explain that a concrete calculation with a verifiable result only comes at the end of the research process. Before that, someone has to build a theory and ask it the "right" questions. When a problem turns out to be too hard, a good researcher switches to a related problem that's simpler and just as interesting. Current agents aren't good at that.

Physicists also share a common, if far from unanimous, sense of what makes a problem interesting. Agents don't seem to have this "taste" yet. When asked to design new problems, they get lost in details a human would find boring.

The researchers stress that these observations can't be quantified and aren't a forecast of where AI is headed. They think AI agents could do top-level research on their own in the near future, but that will take more than the ability to solve problems.

AI is already changing how physicists train and who can compete

The authors already see consequences for training. Until recently, some of the benchmark problems would have worked well as first research projects for early-career researchers. That's no longer appropriate unless the goal is explicitly just to work through the material.

They also point to a structural problem: Theoretical physics has historically needed far less money than experimental research, which has kept opportunities in the field relatively equal. Capable AI agents could change that. The benchmark problems are still cheap to solve, and Astra's full run cost about $116. For harder problems, though, the authors expect resource needs to climb sharply. At that point, a research group's budget could have more say over what it can accomplish, even in theoretical physics. The authors say this raises questions about access and equal opportunity in the field.

Mathematics is already having these debates, for example around the Leiden Declaration. The authors argue theoretical physics should have them too, before AI upends the field. At least one school of thought in physics has so far urged researchers to "shut up and calculate." If taste and vision start to matter more, physicists may need to "stop and think" instead.