The bias strengthens as the system gets more accurate

What it is

The more often the system is right, the less people check the next one. Accuracy here is not a cushion. It is fuel for the bias. Bias grows with automation reliability describes the threshold for “do I open the source this time” being raised by recent hits, inside a single decision.

When a product sells accuracy, it is also selling less review.

Why it happens

People calibrate a heuristic: the last few were right, so the expected return on checking the next one drops. Under load, checking is the first thing dropped, because under high accuracy it almost never “wins.” When a bad piece of advice finally appears, it lands exactly where checking has already been skipped. This shares a learning curve with monitoring going slack after a long quiet stretch, but the observation window differs: here the measure is how the acceptance rate of advice moves with recent hits, not how scan frequency decays with quiet time.

If the UI hides the checking entry deeper after a run of successes (“you trust it, we simplified”), it has turned that learning curve into a feature.

Studying it

Let people live through a stretch of high-hit advice, then insert a bad item, and compare with a baseline that inserts errors from the start. Independent variables: prior hit rate, length of the hit streak, where the bad advice sits. Dependent variables: follow-through on the error, whether a check still happens, subjective “do I still need the source.”

Separate recent hits from the long-run baseline. Comparing only “an accurate system” with “an inaccurate system” tangles complacency, trust and bias; what you want is whether, for the same person, checking drops after hits go up.

Where it stops holding

When accuracy is low enough that advice regularly embarrasses itself, people under-use it and refusal, not bias, is the main problem. If subjective accuracy is pinned by calibration information (regular failure samples), the upward drift is held down. This entry does not discuss whether explanatory text can save the situation, nor how the bias operates as a one-shot heuristic.

Applying it

• Do not use a run of successes to put away the checking entry. When hit rate rises, keep the check step for high-consequence items as it was.
• Periodically play back a caught bad suggestion to the operator, to reset the “lately it has been right” heuristic — not to shame.
• Check: after a high-hit stretch, insert one conflicting suggestion and compare check rate and release rate before and after. If checking dropped and release did not, accuracy is fuelling the bias.

Related

• Same group: L4.02.1 People tend to accept system advice without checking · L4.02.3 Providing explanations does not necessarily weaken the bias
• Nearby: L4.03 Automation Complacency · L5.09 Overtrust and Trust Collapse · L1.09 Intervention Points in the Human Loop
• Search terms: automation bias · reliability and compliance · misuse of automation