High-confidence errors are more harmful than low-confidence ones

What it is

Wrong is wrong — true for fact-checking. False for action. A miss tagged “low certainty” still gets a glance; a miss tagged “high certainty” gets used. Harm from a high-confidence error comes from switching review off, not from being more wrong than a low-confidence miss.

The same fake number travels farther with a green light than with a yellow one.

Why it happens

Confidence is used in the interface as a switch on attention. The high band means “you may skip the check,” which is why it is shown at all. Once that switch is wired to a false proposition, the skipped check is an amplifier: the error enters mail, a record, a quote, wearing the system’s endorsement. Low-confidence errors tend to stop in a draft.

In the automation-bias literature, people review less when advice wears a trusted look. A generate surface adds fluent sentences to a high score — two looks of trust. The line from that literature is here a confidence bar: the fuller the bar, the less the manual check, the dearer the miss that gets through.

Studying it

Plant equally severe errors, half with a high score, half with a low score (or an uncertainty mark). Watch detection, whether it is sent, how far it spreads. Independent variables: score band, a forced pause or not, whether the error sits in a hard-to-see detail (number, name). Dependent variables: misses, outbound sends, later blame.

Severity has to match real consequence. A lab “pick the wrong picture” will understate high-confidence errors because there is no outbound channel.

Where it stops holding

If low confidence is drawn as failure, people will abandon a whole passage that could have been patched — another cost, which does not deny that high-confidence errors are dearer. When the user is a domain expert and the task is to find faults, the gap between bands shrinks. With no confidence shown at all, harm is gated by prose and layout instead — fluent errors are still dear, they just no longer pass through this switch. This entry assumes the interface is showing some kind of certainty. It does not discuss how to calibrate.

Applying it

• Admission to the high band must be strict: better to mark many correct items medium than to put a wrong item in high. High is a privilege, not a default.
• High-band results that contain numbers, dates, names, or money still get one cheap check (a source, a second calculation, or “confirm these three figures”). High must not waive every check.
• Treat a discovered high-band error as an incident, not an ordinary miss: ask why the score was high, and lower the display band for that region.
• Check: in staging, hand-audit high-band samples. The high-band error rate must sit well below the mid and low bands; otherwise remove the high band and leave the product unbanded. High-band misses outrank an equal number of low-band misses in the fix queue.

Related

• Same group: L1.04.1 Confidence is not the same thing as accuracy · L1.04.2 Uncalibrated confidence misleads judgement
• Nearby: L1.08 Confidence displays and their misreadings · L4.02 Automation bias · L3.03 Hallucination and the fact-checking burden
• Search terms: high-confidence error · confident mistake · automation bias