Uncalibrated confidence misleads judgement

What it is

If a score calls itself 80%, then items tagged 80% should, in the long run, be right about eight times in ten. When they are not, the score is uncalibrated. Modern nets often pile mass in the high bins; the 80% bucket may actually hit 55%. People still bet 80. The misleading is not that they fail to understand percents. It is that this percent does not keep its contract.

Once we know confidence is not accuracy, calibration asks: can this number at least be used as a rough probability.

Why it happens

Training objectives (cross-entropy, preference alignment) do not directly punish “high score, low hit.” Temperature, instruction tuning, and RLHF also sharpen the distribution, so scores sit high. Reliability diagrams drawn on classifiers show up again on generators’ self-consistency scores: the middle sags, the ends spike. Handing those numbers to a decision maker copies the model’s overconfidence into the person’s.

People rarely get to run their own calibration experiment. Items cannot be reproduced, feedback is delayed, use is once-off, so a base rate is never learned in use. An uncalibrated number then keeps being spent as a calibrated one.

Studying it

Reliability diagrams, ECE, Brier score are the standard metrics. Human studies add a layer: the same items with calibrated versus uncalibrated scores, watching bets, reviews, handover. Independent variables: temperature scaling or histogram binning on or off, whether the UI says “these scores run high.” Dependent variables: human calibration (their confidence versus their correctness), over-handover.

A lab that only tasks people with already-calibrated scores will conclude “showing confidence helps.” If the field is still feeding raw scores, that conclusion does not travel.

Where it stops holding

A score that has been temperature-scaled and maintained on the same data distribution can be used as a coarse probability — until the distribution moves (new domain, new prompt style) and the calibration expires at once. Open generation with no right/wrong labels cannot define calibration; showing a number is already pretending it could. If users ignore the score entirely, miscalibration does not currently hurt judgement — which only means the score is waste, not that it is calibrated. This entry does not unfold “high-score misses hurt more”; that needs a separate look at the harm path.

Applying it

• Before shipping any certainty number, draw a reliability diagram on your own task distribution. If it does not line up, do not show a number; band it or show nothing.
• If you do show it, calibrate first (temperature scaling, binning) and redo after drift. Calibration is maintenance, not one release.
• Tell people plainly that the high band still errs. Do not paint the high band as a green light.
• Check: sample recent live traffic that carried a score, label it by hand, bucket by displayed band. If a band that calls itself high is near chance, take that band off the interface.

Related

• Same group: L1.04.1 Confidence is not the same thing as accuracy · L1.04.3 High-confidence errors are more harmful than low-confidence ones
• Nearby: L1.03 Visualizing uncertainty · L1.08 Confidence displays and their misreadings · L5.03 Trust calibration
• Search terms: uncalibrated confidence · expected calibration error · reliability diagram