Periodically test whether the proxy relationship still holds

What it is

The relationship between a proxy and a goal is not calibrated once and valid forever. Redesigns, population shifts, and incentive changes rewrite the co-variation, so a proxy relationship audit is required: using an observation closer to the goal, check whether the current proxy still indicates the same thing. At launch, “completion click” and “actually done” may still match; later the completion click is pre-checked, and the relationship breaks. Periodicity is not a taste for retesting. It is because the relationship expires while nobody is watching.

Why it happens

A proxy relationship is an empirical regularity for a period, a population, and an interface. Change the interface and the indicator’s meaning changes: “share” once meant the content was worth sending; once share is tied to a lottery, the indicator becomes lottery entry. Change the population and the same behavior changes meaning: long dwell for a new user may be being lost; for an old user it may be work. Change the incentive and shortcuts appear, replacing the old correlation with a new strategy. With no audit cycle, those conditions are assumed frozen. The audit must use a criterion that does not enter daily rewards, or the criterion is optimized with the proxy and the check loses its contrast. An expired proxy that continues to drive iteration sends the team further along the wrong indicator while they believe they are calibrated.

Studying it

Pre-specify the cycle and the criterion source (small-sample task tests, diaries, unrewarded later behavior), and on the same users estimate correlation, calibration curve, and misclassification between proxy and criterion. Stratify by version and population to see whether expiry is global or local. When the relationship falls below a pre-defined threshold, mark the proxy failed; do not stall with “watch another quarter.” A negative control also helps: introduce a change that should break the relationship; if the audit still reports it intact, the criterion or the method is itself invalid.

Where it stops holding

Audits cost; cycle length can follow how hard the proxy is pressured—dense for proxies in the plan, sparse for watch-only ones. Extremely stable physical or compliance counts do not need the same cadence as experience proxies. Small-sample criteria are noisy; one drop is not enough to retire, look at trend and strata. With no criterion, the audit cannot run; stop calling the number a proxy and call it an uncalibrated count.

Applying it

• Register a criterion, a cycle, and a failure threshold for every proxy that enters decisions, and write them into the measurement plan.
• When due, issue a conclusion with only three allowed values: holds, holds locally, failed.
• A failed proxy leaves success announcements; it may remain on a watch layer, renamed so it is not still read as a goal indicator.
• Run an unscheduled audit immediately after a major redesign, an incentive change, or a shift in the main population, rather than waiting for the cycle.

Related

• Same group: Q6.05.1 A measure that becomes a target ceases to be a good measure · Q6.05.2 Proxy metrics leave a gap to the true goal
• Adjacent: Q6.09 Metric gaming and proxy distortion · Q6.12 Continuous tracking and alerting
• Search terms: proxy relationship audit · proxy validation · criterion calibration