Confidence and Calibration: Does the Model Know It's Wrong?

A model can be wrong and know it, wrong and not know it, or right for reasons that make its confidence meaningless. Calibration is the statistical machinery for telling these apart, and it is worth learning properly because the sloppy version — treating a logprob as a probability of being correct — fails in a specific and predictable way. The definition A predictor is calibrated if, among all the predictions it made with stated confidence p , a fraction p turn out correct. Say it makes a thousand predictions at 70% confidence; about seven hundred should be right. That is the whole property, and note what it is not: it is not accuracy. A weather model that says “30% chance of rain” every single day in a climate where it rains 30% of days is perfectly calibrated and completely useless. Calibration and discrimination are separate axes, and you want both. The relevance to hallucination is direct. If a model were well calibrated on its own answers, you would not need to detect hallucination at all — you would threshold on confidence and route the low-confidence cases to a human or to a search. The reason that does not work out of the box is the subject of the rest of this page. Reading a reliability diagram The plot everyone shows and few label. Both axes run from 0 to 1. x-axis: predicted confidence. Predictions are sorted into bins — conventionally ten equal-width bins, [0.0, 0.1), [0.1, 0.2) and so on — by the confidence the model stated. For a multiple-choice answer that confidence is the softmax probability of the chosen option. y-axis: observed accuracy. Within each bin, the fraction of predictions that were actually correct. The diagonal. y = x is perfect calibration. Points below the diagonal mean the model was more confident than it deserved: overconfidence. Points above mean it was underconfident. The bin counts. Almost always drawn as a histogram underneath, and they matter — a bin holding twelve predictions can sit anywhere, and a diagram without them invites