Feedback loops amplify small initial disparities

What it is

A feedback loop arises when a model's predictions change the world, and the changed world becomes the next round of training data: predictive patrol sends officers to predicted neighborhoods, stops and records accumulate there, and the next model "learns" those neighborhoods are riskier — a small initial bias amplified round over round. Such systems are performative: they do not describe reality but participate in manufacturing the reality they describe. Small initial differences (uneven data collection, early-user skew) grow exponentially into structural gaps through the loop, with no malicious link anywhere.

Why it happens

Amplification runs through three pathways. Visibility: predictions allocate resources and attention (patrols, moderation, recommended exposure); the predicted objects accrue more "events," and events fuel the next training round — popular content becomes more popular, heavily moderated groups accumulate more violation records. Selection: outputs shape people's behavior and retention (rejected applicants get financially worse, down-ranked creators leave), so the surviving sample's composition reflects the output and the remaining data supports the existing model. Self-fulfilling: predictions, once known, change behavior (applicants labeled high-risk stop repairing their credit — they cannot borrow anyway) — the prediction manufactures its own truth. All three share one mathematical structure: with loop gain above one, bias grows exponentially; at one, bias is locked in; below one, it self-corrects. Commercial systems deliberately push gain high (recommendations should close the interest loop, risk control should tighten suspicious segments), turning amplification into a feature.

Studying it

Loops cannot be identified from static datasets; time must enter. Longitudinal tracking: measure group metrics across time slices and test whether gaps widen over rounds. Simulation: agent-based models or simulators initialized from real data manipulate initial bias and loop gain under controlled conditions to observe long-run distributions — the classic predictive-policing simulations showed small initial biases snowballing this way. Intervention experiments: randomly exempt some units from model decisions (keep human or random controls), compare subsequent data distributions of exempt versus non-exempt units, and directly estimate the loop's causal contribution. Methodological cautions: simulation conclusions depend on assumed gain parameters and need calibration on real data before extrapolating to a product; longitudinal data is confounded by concurrent market and policy shifts, requiring comparison series.

Where it stops holding

Not every correlation forms a loop: one-shot decisions (scoring a single exam) have no next training round and no amplification; loops require a channel where outputs flow back as data. Loops are not always harmful: benign loops exist (good content earning exposure that attracts similar creators); the problem is specifically loops that amplify disadvantage. Interrupting an identified loop has real costs — lowering gain usually sacrifices short-term metrics (recommendation relevance, risk-control hit rates), and the organization must be willing to bear them, which is already beyond a technical matter.

Applying it

• Draw the data-reflow diagram: mark every path of "model output → world event → training data" and identify which outputs become their own training labels.
• Maintain bypass control groups for high-risk loops: keep a share of units unaffected by model decisions (random exemption or human handling) as a baseline for measuring loop distortion.
• Reset priors periodically: freeze an initial dataset as the long-term reference, monitor current-distribution drift against it in direction and speed; drift concentrated in one group is the loop's alarm.
• Verify: compare group gaps between exempt and non-exempt units over several cycles — if the non-exempt gap persists larger, amplification is confirmed and that decision's loop gain needs lowering (randomization, lower decision frequency, caps).

Related

• Same group: P4.09.4 Disparate impact is judged by outcome distributions, not intent · P4.09.1 Dropping sensitive attributes does not equal equal outcomes
• Adjacent: L6.06 Long-term effects of recommender systems · P4.03.3 Stratified evaluation
• Search terms: feedback loop · runaway feedback · performativity of prediction