AI's "90%" isn't always 90% — calibration and ambiguity

The piece walks through how a standard image classifier produces a probability: raw class scores go through Softmax so they sum to 1, then training minimizes Cross Entropy against one-hot labels. That process does not, on its own, make the output a trustworthy real-world frequency. The author points to Guo et al.'s finding that deep neural networks are not necessarily well calibrated and that depth, width, regularization and Batch Normalization can affect calibration, measured by ECE — while noting ECE itself has limits, such as sensitivity to how confidence bins are grouped.

The deeper issue raised is definitional. If an image contains both a cat and a dog, asking 100 people might yield 60 choosing cat and 40 choosing dog, yet the dataset may carry only a single "cat" label. The author is careful to say this does not mean such images must always split 60:40, and that this mismatch with human response distributions should not be conflated with ordinary ECE degradation, since ECE measures calibration against the given label.

That same logic is then extended to LLMs. Asked for Japan's most famous university with a confidence score, ChatGPT returned "University of Tokyo — confidence: 95%," but the author points out that "famous" could mean domestic recognition, global recognition, or research standing, so correctness cannot be uniquely evaluated. Cole et al. are cited as finding that, for ambiguous questions, looking at agreement across multiple generated answers was effective alongside model likelihoods. The closing caution is that an AI's percentage should prompt the question "90% of what?" rather than being read directly as correctness.