Mutual adaptation can entrench a wrong initial assumption

What it is

The dark side of mutual shaping: the system forms an assumption from initial observations and acts on it; the user, seeing the system's behaviour, adjusts; the adjusted behaviour then "confirms" the assumption — a wrong initial assumption gets entrenched in a loop. The system thinks the data is speaking; the data is echoing.

This is not a rare fault but the default tendency of the adaptive structure: feedback loops are inherently self-confirming — the very act of acting on an assumption manufactures evidence for it. The tell-tale phrase is "but the data clearly shows" — yes; the data was shaped by that assumption in the first place.

Why it happens

Structure of the loop: assumption → output (the system acts on it) → the user adapts to the output (human adaptivity is automatic) → new observations agree with the assumption → confidence rises.

A domestic example: someone happened to be home on leave during the first week, and the system learned "afternoons usually occupied" → it maintains afternoon heat and light accordingly → arriving home always finds it warm and lit, so nobody ever switches anything on themselves → the data further shows a "stable afternoon pattern". The system's claim "afternoons are usually occupied" has become a self-fulfilling prophecy — it manufactured the world that keeps the claim true.

The fragility of initial data amplifies the problem: the first weeks after installation are precisely when behaviour is least typical — novelty, experimentation, deliberate testing, post-renovation tidying — and that is exactly the window the learning system samples most earnestly. The least typical data initialises the most important assumptions, and the feedback loop then hardens them layer by layer.

Why entrenchment goes undetected: the entrenched result is often "not bad" — the system is serviceable, the user copes, nothing hurts enough to trigger correction. Compared with a wrong assumption that makes the system outright unusable (corrected and re-learned immediately), the "serviceable error" has no alarm at all and survives the longest; by the time it surfaces, several layers of user habit have grown around it.

Studying it

• Algorithmic feedback-loop research: self-reinforcement in recommender systems is the isomorphic phenomenon and has systematic study — users only see what was pushed, and their behaviour in turn reinforces the pushing model, amplifying initial bias and shrinking diversity. Its conclusion (loops amplify the errors of the initial distribution) transfers directly to assumption entrenchment in ambient automation.
• Cold-start sensitivity experiments: manipulate the content of the initial training window (a typical week versus an anomalous one) and watch long-run behaviour diverge — the impact of initial-window contamination on long-run strategy is a repeatedly demonstrated effect.
• Method: counterfactual control periods — switch learning off, run a fixed strategy for a while, and compare user behaviour across the two periods; assumption audits — compare the system's current core assumptions against independently collected ground truth (separately gathered questionnaires or logs) and measure the bias.

Where it stops holding

• Not all stability comes from entrenchment. True preferences also produce stable, consistent behaviour. A testable discriminator: does behaviour follow when the system's output is changed — true preference follows (turn off pre-heating and the user complains it's cold); entrenchment does not (turn off pre-heating and the user never notices, because it was never needed). This criterion turns "detecting entrenchment" from divination into experiment.
• The fault lies not with learning itself but with how initial confidence is assigned. Entrenchment happens when initial assumptions are granted high confidence with no decay mechanism; assumptions with decay and re-estimation naturally yield to new evidence.
• Multi-user systems are at far higher risk. A single user whose assumption is wrong will correct it directly; a shared system's assumption models the "average person" — matching nobody exactly but everybody tolerably, so the corrective pressure is diluted to nothing.

Applying it

• Down-weight the initial learning period: explicitly mark the first N days as "calibration" — observations excluded from the long-term model (or near-zero weight), quarantining the least typical window away from the most important parameters.
• Periodically de-confident assumptions: at intervals, lower each assumption's confidence and re-estimate from fresh observations — giving wrong initial assumptions a natural exit instead of unlimited tenure.
• An explicit assumption-correction channel: "you think I arrive when? Wednesdays I'm always late" — correcting the assumption directly is far faster than waiting for behavioural data to slowly counter-weight, and it plugs in the user's meta-knowledge (I know where the system is wrong).
• How to check: assumption-versus-reality audits — sample the system's current core assumptions (routines, preferences, room usage) against independent measurement; every item showing "bias present, system confidence high" is an entrenchment candidate to be dealt with.

Related

• Same group: Z3.07.1 Users adjust their behaviour to fit how automation judges them · Z3.07.2 Automation keeps adjusting its strategy from observed behaviour · Z3.07.4 Long co-evolution needs periodic review, not unchecked drift
• Nearby: Z3.05.4 Behavioural drift in learning automation keeps eroding predictability · L2 Recommendation and filtering in intelligent systems
• Search terms: self-fulfilling prophecy · feedback loop · cold start · confirmation bias in models