BenchMIRT: AI benchmarks may measure the wrong skills

The introduction of BenchMIRT addresses a core issue in AI evaluation: a single benchmark score can be a blend of different capabilities. The tool stresses that while a benchmark like BBQ is meant to test for social bias, it may also rely on reasoning skills, meaning a low score could misinterpret the reason for a model's failure. Similarly, WMDP is designed to test dangerous knowledge, but BenchMIRT found it is more closely tied to reasoning, where a stronger model is expected to withhold information, resulting in a lower score.

The analysis suggests that these findings could lead to more efficient evaluation. By keeping only 10% of the questions, BenchMIRT generally preserved a similar picture of model capabilities as the full set. However, the tool also carries a risk. The same question-level insight that identifies the most informative safety questions could be used to remove them, creating a benchmark that an unsafe model could pass. The team acknowledges this trade-off but believes the added transparency is worth the risk.

The models used for training were all released by March 2025, which means its findings on dimensions might not hold for newer generations. Furthermore, the dominant dimensions of safety and reasoning emerged from the specific 16 benchmarks selected, and a different mix could surface other capabilities.