Local A/B optima can hide structural problems

What it is

Repeatedly comparing button copy, color, or order on a wrong information structure or a wrong task model can still produce a stable “winner.” That winner is a local optimum: the metric is higher in a small neighborhood of the current structure, while the structural error—an unfindable entry, a mis-split task, a missing capability—never entered the candidate set and was therefore never falsified. Local wins also supply a reason to stop searching: the metric is up, so the structural problem looks handled. The hiding is not a statistical breakdown. The search radius is too short.

Why it happens

Each experiment takes a small step along the gradient of the already shipped interface. Completion and click metrics are sensitive to copy and contrast, and insensitive to “the user is not in this task at all”—people who are not there never join the success path of the sample, and cannot move the contrast on that path. After a winner ships, the next round starts from the new locality: hill climbing. The ridge can be high while the whole hill sits in the wrong valley: registration may be optimized to its best conversion while the structural problem is that the core task cannot be done logged out. Organizational narrative then writes a sequence of small wins as usability progress, and the structural defect leaves the agenda.

Studying it

Align experimental wins with an independent list of task-level failures. If the main-path failure location is unchanged for several quarters while experiment metrics keep rising, mark local masking. Code the layer each experiment moved (pixel, component, flow, task model) and check whether structural items on the failure list never became an arm. A paper or low-fidelity prototype can put a structural alternative in front of a small sample without first becoming a full arm; if the main failure vanishes there, while live local experiments never touched it, masking is demonstrated.

Where it stops holding

When the structure has already been validated by independent research and what remains is wording and defaults, local experiments are refinement, not hiding. A missing underlying capability shared across business lines will not show up in one product’s A/B tests; that needs a cross-product task audit, not more button experiments. A local optimum can be genuinely better on the short metric; denying the small win is pointless. What should be denied is “small win equals structure solved.”

Applying it

• On every experiment request, map to the current structural failure list. If the listed item is not in the candidates, the experiment may not close that failure.
• After three consecutive rounds that only move copy or color while the main failure location stays put, freeze that class of test and produce a structural option.
• When shipping a local winner, write “untreated structural problem remains X” in the report so the victory summary does not overwrite the failure list.
• Run a small-sample prototype that leaves the current structure (a different task decomposition or entry). If the main failure disappears only there, move resources off the hill climb and onto the structure.

Related

• Same group: Q3.04.1 Random assignment is what licenses a causal reading · Q3.04.2 A/B tests compare built variants; they do not invent new ones · Q3.04.4 A short-horizon lift can be long-horizon harm
• Adjacent: Q2.06 Usability testing · Q3.05 Multivariate testing
• Search terms: local optimum · hill climbing · structural usability