The system only observes items it showed; unshown items never receive positive feedback

What it is

Clicks, completions, adds-to-cart can only happen on items that already appeared on screen. An unranked item, however fitting, leaves no positive. Exposure bias means observed positives are conditioned on the display policy. In the data, unshown items cannot be told apart from “the user dislikes them” — not disliked, not allowed to show.

The display–behaviour loop says the two sides write each other. What is nailed here is the missingness structure of labels: the positive set is bound by what was shown.

Why it happens

Recommender logs are missing not at random: missingness is decided by the last ranking. If a learner treats non-click as negative, it collapses “never seen” and “seen and rejected” into one class. The next ranking is even less willing to let unshown items out, and the positive channel stays open only to the already shown. Unshown items then die in the data, not in preference.

Position sharpens this: even among shown items, later rows are less likely to be seen, so positives concentrate further on the head. “Only positive signals” is therefore not the whole problem — even skips are conditioned on exposure.

Studying it

Methods come from exposure bias and MNAR: collect a slice of policy-independent observations in random or uniform probe slots, and contrast them with policy logs. Independent variables: probe share, whether non-click is treated as negative or as missing. Dependent variables: chance that unshown items enter later exposure, gap between relevance estimated on probe data and on policy data.

Offline replay cannot see true labels of unshown items. Report the share of items that are liked in the probe slot and look negative in the policy log. That is the volume of exposure bias. Online evaluation and the cost of exploration are later problems.

Where it stops holding

When the user searches or opens an inbound link, display is set by the query and exposure bias moves to retrieval, not to the recommendation policy. When the pool is so small that almost every item rotates in, the bias eases. This entry only treats “positives must be conditioned on display.” It does not treat users performing on purpose, and it does not treat offline metrics favouring the logging policy.

Applying it

• Treat non-click as missing by default, not as negative. Take negatives only from explicit dismiss, “not interested,” or an open that bounces immediately.
• Keep a stream of exposure that does not follow the current rank, so items that have never appeared can still leave a positive.
• Check: find items with high clicks in the probe slot and almost no exposure under the main policy. If the main model scores them low, exposure bias is already in the score. Force-expose them for a round and see whether positives appear.

Related

• Same group: L6.09.2 Early accidental clicks get amplified into durable profile features · L6.09.3 Users adjust behaviour to get the recommendations they want, so behavioural data lose representativeness · L6.09.4 Breaking the loop needs active exploration; the cost is immediate and the benefit is delayed · L6.09.5 Offline evaluation on historical logs systematically favours the policy that produced those logs
• Nearby: L6.04 Feedback Loops · L6.08 Filter Bubbles and Diversity · L6.13 Negative Feedback Channels for Recommendations
• Search terms: exposure bias · MNAR · missing not at random