Why asking LLMs for confidence scores is scientifically meaningless

The author begins with a direct claim: asking an LLM to generate a confidence score for its own response is completely useless from everything they can tell. They note this conversation has recurred at multiple companies multiple times, suggesting a broader misunderstanding about what these systems can do. The pattern they keep seeing is a request for a JSON object with a response key and a confidence key, usually expressed as a continuous 0–100 scale.

The author acknowledges that Anthropic and others have published real research on model introspection. Anthropic's interpretability work found that Claude plans several words ahead when writing a rhyming couplet—evidence of some internal planning process beyond pure next-token prediction. More recent work on "emergent introspective awareness" showed that models can notice concepts injected into their activations and report on them accurately. However, the researchers themselves placed strong caveats on these findings: the capability is highly unreliable and context-dependent. The author argues that noticing something injected into activations under lab conditions is "a very long way from quantifying how likely your paragraph about a customer's refund policy is to be right."

The author then invokes a core logical problem: if models were to use their thinking trace to self-evaluate their responses, the question immediately regresses into "what's the confidence of the confidence score itself?" and then "what's the confidence of that?"—a recursive loop with no foundation. They note that DeepMind's paper "Large Language Models Cannot Self-Correct Reasoning Yet" found that when models attempt self-correction using only their inherent capabilities, without external feedback, performance often degrades. This directly contradicts the assumption that asking for a confidence score will yield a trustworthy proxy.

The author identifies a second major problem: the definition of confidence is ambiguous. Is the model confident in the correctness of the response, the coherence of the response, or that it has attempted to fulfill the user's request? These are three entirely different questions with different failure modes, yet a single float flattens them all into one number. Without a heuristic that bounds the concept, the scores are effectively useless.

A third issue arises because confidence is not uniform across a response. In classical machine learning, methods like Platt scaling, isotonic regression, and temperature scaling allow post-hoc correction of overconfident neural networks, and these corrections are validated against held-out ground truth using reliability diagrams and Brier scores. None of this machinery exists when someone adds a confidence key to a JSON schema. The author notes that even at the token level—the native LLM unit—likelihood values vary wildly: some tokens are structural glue with near-certain probabilities, while others are load-bearing factual claims. Collapsing that distribution into one number the model then verbalizes is "not a measurement. It's a vibe."

The author cites semantic entropy, introduced by Farquhar et al. in Nature, as a more rigorous approach: sample the model multiple times, cluster responses by meaning rather than wording, and measure entropy across those clusters. This method exists precisely because token-level likelihood conflates "I could have phrased this five different ways" with "I don't actually know this." Crucially, semantic entropy is a measurement performed on the model from the outside, not a number the model reports about itself.

The author then turns to research on what actually works. They cite Lin et al.'s work showing that models can be trained to emit calibrated verbal confidence when fine-tuned on purpose-built task distributions. Tian et al. found that with the right prompting strategy and RLHF, models can verbalize probabilities better calibrated than their own conditional probabilities, and that prompting plus temperature scaling can cut expected calibration error by more than half. Anthropic's "Language Models (Mostly) Know What They Know" found encouraging results asking models to estimate the probability their own answer is true. However, all of these approaches require extra work: fine-tuning, deliberate prompting strategy, constrained answer format, or explicit post-hoc scaling. None of it comes for free.

The author strongly criticizes the 0–100 continuous scale, citing Xiong et al.'s empirical evaluation showing how badly overconfident verbalized scores tend to be. The most damning finding comes from "Rescaling Confidence: What Scale Design Reveals About LLM Metacognition," which tested six models and found that more than 78% of responses landed on just three values, with one model reporting exactly 100 on 68% of instances. Coarser categorical scales performed better on metacognitive sensitivity than the 0–100 baseline. This directly contradicts the intuition that a continuous spectrum is more informative.

Finally, the author notes that confidence is almost always a proxy for correctness in practice, particularly in RAG (retrieval-augmented generation) use cases where someone wants the model to signal whether it retrieved the right documents and grounded its response correctly. The author warns that optimizing hard toward such a confidence score would reduce the model's willingness to synthesize across sources, thereby harming the system's value proposition. The risk is not that the score is unreliable (though it is), but that chasing it could degrade the quality of synthesis—the very thing that makes LLMs valuable in retrieval systems.

The core problem the author identifies is that confidence scoring has become a cargo-cult practice across the industry without the scientific foundation it appears to have. While researchers at Anthropic and other labs have published work on intrinsic model awareness and self-correction, those papers consistently carry caveats: the abilities are unreliable, context-dependent, and context-specific. Yet teams ship confidence scores as if they were a reliable, out-of-the-box feature. The author's framing of this as a "psychological safety trick" cuts to the real issue: it makes stakeholders feel safer without delivering actual safety.

The most damning evidence comes from published benchmarks on confidence elicitation. A paper on rescaling confidence found that models don't treat the 0–100 scale as a continuous spectrum at all; they cluster on round numbers. This is not a minor quirk—it directly undermines the assumption that a numeric score carries meaningful gradation. Research supporting categorical confidence (Lin et al., Tian et al., Anthropic's own work) does exist, but every study requires either fine-tuning, explicit prompting strategy, constrained answer formats, or post-hoc scaling. None of that happens when a team adds a confidence key to a Pydantic model and ships it.

The author also highlights a second-order risk: optimizing hard toward a confidence score in retrieval scenarios may actually degrade synthesis quality, because the model learns to avoid cross-source reasoning in order to keep every fact locally grounded. This transforms confidence scoring from a harmless comfort measure into something that can actively harm the system's output quality.

No comments yet. Be the first to share your thoughts!