Offline evaluation on historical logs systematically favours the policy that produced those logs

What it is

Take last week’s impressions and clicks, compute a new model’s recall, precision, nDCG — and the high scores tend to belong to models that look more like last week’s ranker. Logging-policy bias in offline evaluation means positives in the historical log were exposed by the current policy. Offline metrics reward reproducing that policy and punish giving exposure to items the log never saw — even when those items would do better online.

Exposure bias says positives are conditioned on display. Here the evaluator has eaten the same conditioned data and will mis-judge a new policy.

Why it happens

The offline protocol assumes clicks in the log are labels of item quality that can be replayed on any new ranking. Clicks were only observed on the old ranking’s display set. If the new policy promotes items the old one never showed, the log offers no positive and nDCG calls it wrong; if the new policy pushes the old head further up, positives light up and nDCG calls it right. “Better exploitation” is rewarded; “necessary exploration” is punished.

Inverse-propensity weighting, doubly robust estimation, and interleaving exist to lift evaluation off the logging policy. Skip them and ship on the offline table, and the product will systematically refuse models that open the loop — not because they are worse online, because they look worse in their own log.

Studying it

The recsys evaluation contrast: the same candidate models, scored with pure offline nDCG, with IPS / doubly robust, and with online interleaving or A/B. Ask whether the three rankings agree, especially the offline rank of explore-heavy and diversity-heavy models. Independent variables: whether the log contains random probes, whether propensities are calibrated. Dependent variables: Kendall correlation of offline rank with online rank, rate at which explore-type models are wrongly killed offline.

Logs without probe slots make IPS high-variance too. Report propensity overlap, not only one weighted score. Describe the method this far — do not invent “paper X lifted the metric by N percent.”

Where it stops holding

When the candidate set is fully covered by the old policy (a tiny library that nearly rotates), offline bias is light. Pure content retrieval, a task independent of display policy, is on firmer offline ground. This entry is “scoring yourself on your own display log will bias toward yourself.” It does not treat users performing, and it does not restate exploration’s experiential cost.

Applying it

• An offline table must not alone decide a ship. Explore-type and diversity-type changes go through interleaving or small-traffic online even if the offline score drops.
• Keep propensity and probe flags in the log. Default evaluation to a counterfactual method; keep raw nDCG as diagnosis only.
• Check: take a diversity model known to cover better online with a slight click drop, and look at its rank on a pure offline table. If it sinks while “more like the old policy” sits on top, offline evaluation is already campaigning for the loop. Re-run the same model in an interleave.

Related

• Same group: L6.09.1 The system only observes items it showed; unshown items never receive positive feedback · L6.09.2 Early accidental clicks get amplified into durable profile features · L6.09.3 Users adjust behaviour to get the recommendations they want, so behavioural data lose representativeness · L6.09.4 Breaking the loop needs active exploration; the cost is immediate and the benefit is delayed
• Nearby: L6.04 Feedback Loops · L6.08 Filter Bubbles and Diversity · L6.02 Filter Bubbles
• Search terms: offline evaluation bias · logging policy · off-policy evaluation