Behavioural drift in learning automation keeps eroding predictability

What it is

Learning automation — systems that keep learning from user behaviour — carry a predictability erosion all their own: behavioural drift. Their "rules" change over time; what was learned this week may be overturned by new data next week; the user changes no setting, and the system's behaviour shifts anyway. The cornerstone of predictability is "same conditions, same behaviour", and the learning mechanism is precisely what cannot guarantee it: every silent update interrupts the model the user had just built.

Drift differs from random failure in a way worth remembering: random failure is understood as "it broke for a moment" and the model survives; drift leaves users unable to tell "it broke" from "it changed" — the attribution system itself destabilises, and the trust damage runs deeper than any occasional failure.

Why it happens

Three sources, each stealthier than the last:

• New data reshapes the model: recent observations weigh more, so changed routines change behaviour — that is the design intent of learning, but from the user's viewpoint it is visually indistinguishable from a fault.
• Training objectives misalign with user objectives: the system optimises measurable targets — energy, engagement, session length — while the user wants comfort and peace of mind; when objectives misalign, "the better it learns, the further it drifts".
• Silence: learning happens backstage with no surface representation at all. Users see behaviour change and can find no reason for it.

The third is the amplifier. Yang and Newman's 2013 in-home study of the Nest learning thermostat found systematic misconceptions of the learning mechanism (believing it recorded audio, treating the leaf indicator as a reward to be deliberately maintained); when learning is invisible, users develop folk theories to fill the explanatory vacuum — naive, often wrong, yet genuinely steering their behaviour. Wrong models produce wrong predictions; wrong predictions spend trust.

Studying it

• Yang and Newman (CHI 2013): first-hand documentation of misconceptions and folk theories around a learning product in real homes — the canonical record of what user cognition looks like when learning is opaque.
• Folk-theory research: users' naive explanations of how algorithmic systems work is an established object of study (folk theories of social media feeds show the same pattern: an explanatory vacuum is always filled by naive theory, and once hardened it is stubbornly hard to correct).
• Method: drift audits — over long deployments, automatically detect change points in behaviour distributions (under no-setting-change conditions), aligned on a timeline with surprise reports from experience sampling, quantifying "detection latency of silent change" and "post-detection attribution accuracy".

Where it stops holding

• Drift is not intrinsically bad. Adapting to new needs is the point of learning; the problem is silence and uncontrollability, not change. The conclusion is not "don't learn" — it is that drift must be visible, reversible, and pausable.
• Small drifts may sit inside the tolerance band. Users neither notice nor mind a gently adjusted temperature preference; the damage comes from drift that exits the band, or runs against the user's wishes (an energy-saving objective riding roughshod over comfort). Do not read "no complaints" as "no harm" — accumulation inside the band is also grinding trust down.
• One-off changes in rule-based systems are a different entry. Firmware updates altering behaviour are discrete, version-anchored events belonging to the device lifecycle; this entry is about continuous, version-less creep.

Applying it

• Change notices: when learning changes behaviour, announce it with the triggering evidence and an undo — "from this week, pre-heating starts 20 minutes earlier, based on your usual 18:40 arrival over the last three weeks; wrong? tap to restore".
• Freezable learning: users can pin current behaviour and forbid further learning — trading adaptation for predictability is the user's call to make.
• Visible training objective: state what the system optimises (energy / comfort / whose preference), and which objective wins when they conflict — don't make users infer the objective function from surprises.
• How to check: run a drift audit — extract behaviour-change events under no-setting-change conditions from the logs and verify each carries a notice and an undo path; silent drift must count zero.

Related

• Same group: Z3.05.1 Identical situations should produce identical automation behaviour · Z3.05.2 Unpredictable automation costs more mental effort than manual operation · Z3.05.3 Predictability comes from transparent rules, not from memorised exceptions
• Nearby: Z3.07.2 Automation keeps adjusting its strategy from observed behaviour · Z3.04.3 Overrides should be learned by the system
• Search terms: behavioural drift · folk theories · learning thermostat · model update transparency