Study examines how well LLM confidence scores match actual classification accuracy

The author explored whether AI-generated confidence scores from large language models (LLMs that understand and generate text) used for document classification align with real-world accuracy, using injury classification data from the 2024 NEISS.

LLMs generate confidence scores in two main ways: by prompting the model to estimate its own confidence in the output, or by directly extracting token-level probabilities (numerical confidence values for individual words) from the model. The author used a "top versus all" calibration approach with isotonic regression (a statistical method that maps probabilities to observed accuracy) to adjust raw confidence scores to match true accuracy rates—for example, remapping an original token probability of .85 to a calibrated probability of .61.

Calibrated probabilities can be applied in production by building a calibration model on sample cases, validating on a separate hold-out set, then applying the adjusted scores to future classifications.