OpenAI paper: AI can't say "I don't know"

The paper draws a line between two very different explanations for hallucinations. One is structural: facts that appear only once in training data, such as a person's birthday, cannot be inferred from patterns, so the authors argue a share of errors on that kind of fact is statistically unavoidable. The other is about incentives, and the authors treat it as the more important one.

Their classroom analogy is blunt: on a four-choice question, a blind guess has a one-in-four chance, while writing "I don't know" guarantees zero. Under binary scoring, they show mathematically, answering is always the better bet for any level of confidence. The SimpleQA numbers make the same point in practice, and the contrast between the two models is stark even though their accuracy rates are close.

The proposed fix changes the arithmetic so that an uncertain guess has negative expected value, which would give honesty a payoff for the first time. Whether that happens hinges on benchmark makers rewriting their scoring rules, since the paper's authors direct their recommendation at the major evaluation metrics themselves rather than at model training alone.