Confidence is not the same thing as accuracy

What it is

The model says “I am 80% sure,” which sounds like “it gets this class of item right eight times in ten.” The first is a self-score of this output. The second is a hit frequency under repetition. Confidence is not accuracy: one number describes how determined the model currently seems; the other describes how often it has been right. If the interface spends the first number as the second, the user is buying a monologue as if it were statistics.

This entry splits the two quantities. Whether they are calibrated, and how much a high-confidence miss hurts, come after.

Why it happens

Classifiers and generators can both emit a 0–1 score: max softmax, sequence log-probability, agreement under self-consistency sampling. Those scores optimise an internal objective and need not track external correctness. The default semantics of a percent, for people, is frequency — “eighty percent” is eight in ten. The two semantics collide on one piece of UI and frequency wins, because that is how percents work in ordinary language.

Generative output often has no single “correct” to score against. A fluent fake citation can carry a high sequence probability. The score is rewarding fluency; the user is purchasing correctness. They are not the same good.

Studying it

Take a batch of items with model scores, label them independently right or wrong, and draw a reliability diagram: actual hit rate in each score bucket. Also ask “what does this 80% mean,” and count how many people say “about eight in ten correct.” Independent variables: the name of the score (confidence / certainty / probability), whether a “this is not an accuracy rate” note is shown. Dependent variables: chosen semantics, subsequent handover.

Report user semantics and true calibration separately. A note that changes semantics while calibration stays poor just moves the problem to the next card.

Where it stops holding

On systems with external labels and a score trained to be a probability, the two numbers can be pulled close — they remain two quantities that happen to agree. A forecast’s “probability of precipitation” is a promise about frequency; a generate box’s “certainty” usually is not. On creative tasks with no right or wrong, accuracy as a quantity does not exist; if a confidence is still shown, it can only mean internal agreement, and must be renamed. This entry does not treat the harm pattern of miscalibration. It treats the substitution of concepts.

Applying it

• Do not label an internal score accuracy or correctness. Call it the model’s self-rated certainty, and add that it is not a historical hit rate.
• If you have a real hit rate, show that, not the internal score. The hit rate must name its scope: “on invoice-field extraction, 82% of the last thousand were right.”
• By default, do not show a 0–100 certainty bar on creative output, or it will be read as accuracy.
• Check: point at an 80% and ask “if we did ten items like this.” If the answer is “about eight right” and you have no such frequency evidence, the label is passing confidence off as accuracy.

Related

• Same group: L1.04.2 Uncalibrated confidence misleads judgement · L1.04.3 High-confidence errors are more harmful than low-confidence ones
• Nearby: L1.03 Visualizing uncertainty · L1.08 Confidence displays and their misreadings · L5.03 Trust calibration
• Search terms: confidence versus accuracy · reliability diagram · self-reported confidence