Proxy metrics leave a gap to the true goal
Aliases: proxy metric · construct gap · coverage gap
What it is
What experience actually needs to judge—understanding, trust, fluency, lack of regret—is rarely countable directly, so opens, dwell, ratings, and clicks stand in as proxies. The proxy–goal gap is the systematically uncaptured difference between what the proxy records and the goal. Dwell proxies “involvement”; the gap is staring, being stuck, and autoplay. Ratings proxy “willingness to return”; the gap is polite praise and the fact that only people at affective extremes score. The gap exists before any pressure is applied; it is not created by becoming a target.
Why it happens
Operationalization always loses information: choosing an observable indicator abandons the parts of the goal that indicator does not cover. Gaps have direction. Some proxies are too narrow, covering one face of the goal (clicks for “the feature was discovered,” ignoring clicked-but-not-understood). Some are too wide, counting unrelated process (session length for “the draft was finished,” including time spent on a loading screen). There is also a time gap: currently collectable behavior corresponds to a shallow layer of the goal; the deeper layer shows later. The gap is what lets “the metric moved” and “the goal moved” happen separately. Design work that only closes the metric stops on this side of the gap, with the goal still on the other.
Studying it
Draw a coverage map for each proxy: which parts of the goal are covered, which fall explicitly outside, and find an independent observation for the outside (interview, task success, later regret). On the same users, collect the proxy and a criterion closer to the goal, and report unexplained variance, not only a significant correlation. Deliberately introduce a change that should move the proxy and, by design, not the goal (meaningless waiting), and see whether the proxy rises alone; if it does, the gap is already being used by the interface.
Where it stops holding
Some goals can almost only be proxied, such as long-horizon trust. The task then is not to kill the gap but to label it and refuse to read the proxy as the goal. A small gap does not mean the proxy can be pressured: selection pressure will widen even a small gap. Conversely, a wide-gap proxy can still serve as early reconnaissance, provided it does not enter the success announcement. Averaging several proxies does not fill the gap; it mixes several gaps into one number.
Applying it
- Next to each proxy, write one sentence naming the part of the goal it does not cover; that limit is mandatory reading.
- Give the uncovered part at least one check that does not enter the dashboard (task test, diary, coded support).
- Ban “the correlation is decent” as evidence that proxy equals goal; correlation is co-movement, not complete coverage.
- When a product change mainly acts on a known gap (inflating dwell, inflating clicks), demote that proxy; it must not announce goal attainment.