A single severe error can destroy long-built trust

What it is

A tax assistant is used for a year; every quarter it adds up right. One quarter it files the wrong entity type, and the penalty lands on the user. A year of smooth running goes offline that day. One error, severe enough, can destroy trust that took a long time to accumulate. What collapses is not a slowly recalibrated bias. It is trust as willingness to hand over, emptied in one stroke.

Severe means the consequence is irreversible or hard to reverse, and is attributed to the system. Ordinary small misses that pile up into overtrust are not this entry’s object.

Why it happens

Long-built trust is a generalisation: it will not harm me where it matters. The generalisation is held up by many low-stakes successes, but its content is about the high-stakes place. One injury there falsifies the generalisation directly; earlier successes, not being in the same consequence band, cannot offset it. People do not spend “it got the tax amount right forty times” against “it got the entity wrong” — the units differ.

Behaviour after collapse is withdrawal: auto-filing off, back to manual, telling colleagues not to use it. Withdrawal is much faster than the year it took to start handing filing over. Lee and See noted that trust can be rewritten sharply by a single negative event; what to mark here is the severity threshold — past it, the talk is not about nudging an estimate, it is that the relationship broke.

Studying it

After a history of success, inject one high-consequence failure (real or high-fidelity outbound, a penalty, an unrecallable send), and contrast the same count of low-consequence failures. Independent variables: severity of consequence, whether the failure can be attributed to the system, length of the success history. Dependent variables: cliff-drop in willingness to hand over, withdrawal behaviour, the decision to take the system out of the workflow.

Labs struggle to mint a real penalty. Use currency, reputation, or irreversibility the participant cares about, or you will measure rating wobble, not collapse.

Where it stops holding

If the user never handed the task over at the high-stakes place — always treated it as a draft — “severe” may not reach collapse, only a bad review. A tool whose reliability was declared very low has no long-built trust to collapse. If after one failure trust is still there, only lower, that is an asymmetric update, not this emptying. A streak of small successes pushing trust above reliability is also not collapse.

Applying it

• Band features by consequence. Bands that can cause penalties, outbound mail, or irreversible writes do not get a “it was accurate for a year” waiver from checking; keep an independent check where it matters.
• On high-consequence paths, pre-build emergency recall, human takeover, and an incident account, on the assumption that collapse can happen on any one run.
• Do not let praise from a low-consequence setting underwrite high-consequence automation.
• Check: walk an exercise in which the error really goes out, and see whether the workflow still has a second gate. No gate, and you are betting you will not meet that one time; if you do, long-built trust is paid out with it.

Related

• Same group: L5.04.2 Trust recovers far more slowly than it is built · L5.04.3 How an error is handled affects trust more than the error itself
• Nearby: L5.10 Asymmetric Effect of First Failures on Trust · L5.09 Overtrust and Trust Collapse · L5.03 Trust Calibration
• Search terms: catastrophic trust collapse · severe automation failure · trust destruction