Identical situations should produce identical automation behaviour

What it is

The first cornerstone of predictability: behavioural consistency — same conditions, same inputs, the automation does the same thing. Users build a causal model of "which conditions lead to which behaviour" by repeated observation; when behaviour wobbles randomly (sometimes triggers, sometimes not; sometimes fast, sometimes slow), no model forms and every prediction degenerates into a coin flip.

Consistency is not "never changes". Heating coming on earlier in cold weather, weekends running a different schedule — behaviour varying with real conditions is the content of the model. Users can tell the two kinds of change apart: same conditions, different behaviour breaks the model; conditions changed, behaviour followed is the model.

Why it happens

Trust in automation is predictive: trust ≈ being able to forecast behaviour from environmental conditions. Consistent behaviour lets users stop monitoring — you can predict the outcome without watching, which is precisely how automation's promise of peace of mind gets honoured. Once behaviour wobbles, monitoring must resume and the labour-saving promise collapses — and users cannot schedule the resumed monitoring for "when it matters", because they never know when that is. They must stay online indefinitely.

The technical sources of inconsistency deserve naming, because each is stealthy:

• Chatter around thresholds: the condition hovers at the boundary and triggering becomes a coin toss;
• Timing races: multiple trigger sources arriving together in nondeterministic order;
• Hidden internal state: cooldowns, caches, and rate limits are invisible, so identical visible inputs meet different internal states;
• Latency variance: cloud-dependent processing times vary, so the system seems to act "at whim".

Studying it

• Trust calibration research: Lee and See's 2004 review systematised the trust–reliability relationship — trust should be calibrated to demonstrated capability, and behavioural consistency is the observable proxy. Its famous conclusion: a system that is mediocre but stable can be relied upon more than one that is occasionally brilliant.
• Automation surprise: Sarter and Woods's 1995 cockpit studies documented pilots' struggles with unexpected automation behaviour — when behaviour cannot be attributed, operators cannot maintain an accurate picture of the system's mode.
• Method: measure same-condition behavioural variance from long-deployment trigger logs; manipulate consistency levels experimentally and track trust scales and monitoring behaviour (glances, checking frequency).

Where it stops holding

• Consistent is not rigid. The test is always "were the conditions the same?" — misdiagnosing condition-following change as inconsistency yields the false prescription "never change". A learning system's legitimate evolution (the household's routine changed; the system followed) is not the inconsistency meant here; its real hazard is silent evolution, which is another entry in this group.
• Inference systems wobble by nature. Confidence around a decision threshold fluctuates; full determinism is unattainable. The engineering backstop is hysteresis and debouncing, which crush the randomness of the critical band. What consistency demands is eliminating unexplained variance, not eliminating probabilistic inference.
• Consistency is defined over user-observable conditions. Identical internal sensor readings with different environmental presentation (a light sensor half-covered by a curtain) looks like inconsistency to the user — the yardstick is what the user can see.

Applying it

• Add hysteresis to trigger decisions: cross a one-sided threshold to trigger, retreat a real distance to reset — chatter at the boundary disappears.
• Make hidden internal state (cooldowns, caches, rate limits) either explicit or behaviourally uniform; never let the same visible conditions meet different hidden states.
• Institute variance attribution: every "same conditions, different behaviour" sample must be attributable to some explainable factor (different mode, different goal, learning update); variance with no attribution is a defect.
• How to check: automatically extract same-condition/different-behaviour sample pairs from trigger logs and attribute each one; zero unexplained variance is the pass mark.

Related

• Same group: Z3.05.2 Unpredictable automation costs more mental effort than manual operation · Z3.05.3 Predictability comes from transparent rules, not from memorised exceptions · Z3.05.4 Behavioural drift in learning automation keeps eroding predictability
• Nearby: Z5.02 Rule conflicts · Z5.06 Time windows and debouncing
• Search terms: behavioural consistency · trust calibration · automation surprise · hysteresis