Uncorrectable context judgements lose user trust

What it is

A context judgement users cannot correct gets judged untrustworthy — regardless of its actual accuracy. The basis of trust is not "it is usually right" but "when it is wrong, I have recourse": correctability is the precondition of trust, with accuracy a secondary reference.

This is not an emotional demand but a rational strategy: facing a system that can err and admits no influence, the only safe stance is distrust. Users cannot audit the judgement's internals; their only probe of the system is "can I affect it" — and the correction channel is that probe. Without a probe, the system falls outside the space of trust entirely.

Why it happens

Why does correctability precede accuracy? Three layers:

• Undiscoverable errors are tolerable; unmanageable ones are not. For an individual user, whether the 5% error share of a 95%-accurate judgement lands on them is luck; with a correction channel, error is a manageable cost; without one, it is an uninsured risk. People price uninsured risk far above its probability — rather fewer features than judgements with no floor under them.
• Perceived control carries trust. A stable finding in human factors: operators' trust in automation is moderated by perceived control, not just objective performance; operators stripped of means of influence show declining trust and reliance even with automation performance unchanged. Context judgement turns the operator into a pure spectator — without even a seat from which to watch — and correction is the sole outlet of control.
• Uncorrectability blocks the two-way learning channel. The normal path of trust-building includes repair: system errs, user corrects, system fixes, trust rebounds — above where it started. An uncorrectable system closes that uphill road, leaving a one-way descent: every error is pure deduction with no recovery mechanism. Over time the trust ledger can only thin.

Studying it

• Trust-in-automation research offers mature constructs and scales: trust in automation is composed of performance, process transparency, and purpose alignment dimensions, and measures of the interventability dimension (manipulating correction opportunity for erroneous automation) transfer directly — manipulate "correctable or not", hold everything else, and observe trust scales and reliance behaviour.
• The classic human-factors work on trust-reliance calibration anchors the theory: overtrust causes disuse-through-complacency, distrust causes over-monitoring; correction capability enters the model as a trust moderator.
• Evaluation paradigm: in long deployments, compare the trust trajectories (scales + reliance behaviour) of correctable versus uncorrectable versions of the same judgement, and track the post-error trust recovery curve — the correctable group's trust rebounds past baseline after repair, the uncorrectable group declines monotonically; the shapes of those two curves are the finding.

One methodological caution: self-reported trust and reliance behaviour diverge — users say they trust while behaviourally routing around the feature (no longer relying on it). Measurement must be dual-track: alongside scales, track functional reliance (trigger counts, active invocation rates) as behavioural trust, using the latter to calibrate the former.

Where it stops holding

• Erosion concentrates on high-consequence judgements. Low-consequence mistakes (an off recommendation) don't need correcting — users don't keep score. Investment in correctability should be allocated by consequence: the most consequential judgements get the most complete correction, not an even spread across the system.
• A channel that exists but is hard to use counts as absent. A feedback form buried five menus deep reads to users as "cannot correct". Correctability is measured by actual reachability (how many steps, how visible, when available), not by presence on a feature list.
• A perfunctory channel is worse. An entry point that leads nowhere — no current-instance effect, no future update — confirms "speaking is pointless": one betrayal more than having no entry point at all. Open a correction channel only when both of its semantic layers can be honoured; otherwise keep it closed.

Applying it

• Tier correction by consequence: high-consequence judgements (security, automation triggers, states visible to others) must carry in-place, dual-semantics (now + future) correction; low-consequence judgements get lightweight correction or none.
• Manage the reachability of correction as a metric: steps from error surfacing to completed correction ≤ 2, entry-point visibility, effect receipt — three items in acceptance criteria, not in "future optimisation".
• Monitor behavioural proxies of trust: reliance trends on judgement-driven features; when reliance on a feature slides persistently, check its judgement's correctability before its accuracy — trust problems usually surface before performance problems.
• How to check: post-error trust-recovery A/B — the same error, one arm correctable-and-honoured, one arm not, tracking scales and reliance for 30 days. A visibly higher recovery curve in the correctable arm is the proof that the channel is earning trust back.

Related

• Same group: Z2.08.1 Correction entry points must appear the moment the judgement takes effect · Z2.08.2 Corrections must update future judgements, not just the current one · Z2.08.4 Frequent corrections signal systematic bias in sensors or models
• Nearby: Z3.03 Failure of intent inference · Z7.03 Modifiability
• Search terms: trust in automation · correctability · calibrated trust · reliance on automation