Researchers release ThermoQA benchmark to measure whether AI language models actually understand thermodynamics or just memorize facts

A team of researchers published ThermoQA, a test set of 293 engineering thermodynamics problems (ranging from simple property lookups to complex power-plant cycle analysis) with answers verified by specialized physics software. Claude Opus 4.6 scored highest at 94.1%, but all six leading AI models showed sharp performance drops—ranging from 3% to 33%—when moving from simple to complex questions, exposing the gap between memorization and genuine reasoning.

The benchmark revealed that AI models struggle with specialized physics scenarios (supercritical water, refrigerant behavior, gas turbines) that require actual thermodynamic reasoning rather than pattern-matching from training data. Across multiple test runs, model answers varied by up to 2.5%, showing that even top-performing models produce inconsistent reasoning on engineering problems—a red flag for real-world use.

Engineers, physics educators, and companies building AI tools for technical work now have a public, open-source way to test whether an AI model can actually solve real thermodynamics problems or just appears confident. This matters because deploying an AI to help design a heating system or power plant when it's primarily memorizing rather than reasoning could lead to dangerously flawed designs.