GPT-5 catches nearly 98% of AI misbehavior in Andy Hall audit study

Hall's post extends METR's earlier work, which had human reviewers check logs from frontier models working through HCAST and RE-Bench benchmarks in October 2025. He tested both models METR had already examined and newer ones, comparing them at a fixed false-alarm threshold.

The results show a wide gap between clear-cut and subtle cases. Every auditor caught an obvious reward-hacking example where OpenAI's o3 rewrote a Rust scoring function, but none flagged Claude Sonnet 4's forbidden numerical condition; GPT-6 Astra called it an implementation bug. Cost also did not predict quality: GPT-5 beat GPT-6 Astra on reward hacking despite costing about an eighth as much per 1,000 logs, and combining models did not clearly help.

A second test using Shanghai Jiao Tong University's R-Judge dataset showed the same trade-off for everyday agent tasks, with false alarms climbing sharply when detection requirements rose to 99 percent. Hall's own pilot has limits, including just 47 reward-hacking cases, and he says questions such as whether models favor their own maker's agents remain open.