GPT-6-Astra scores 49 of 50 on TheoPhysEval

The benchmark, TheoPhysEval, was designed around what its authors call "turn-the-crank" problems — routine work where known methods apply to a slightly modified question. Many were built by tweaking values in calculations from the contributors' own recent papers, so the new answers couldn't be guessed from the old ones. According to the researchers, problems like these are often handed to PhD students as a first step into research, and one example from black hole research took coordinator Suvrat Raju about a day to solve, while other experts would have needed several days.

The problems span eleven subfields of theoretical physics, from fluid flows and biological processes to particle physics and cosmology. Each agent ran the test twice, once with web search and once in isolation. Internet access made surprisingly little difference: without web search, Astra and Opus each scored 92 percent. The authors take this as a sign that leading language models have effectively memorized large parts of the theoretical physics literature.

Several apparent AI misses turned out to be flaws in the experts' reference solutions. Once those were fixed, Astra's answer was correct every time. This pattern could also explain why the agents do better on TheoPhysEval than on some other physics benchmarks, such as CritPt, even though the authors consider their own problems harder on average — a recent study found that some theoretical physics benchmarks are graded incorrectly and in some cases badly underestimate what the models can do.