The primary metric is locked before the experiment starts, not chosen afterward

What it is

The primary metric is the one outcome this experiment will use to ship or stop. It has to be frozen before traffic or participants enter treatment: definition, window, denominator, success rule. Picking, after the experiment, whichever of conversion, time, and ratings looks best as “what we actually cared about” is outcome switching. Switching writes the already-seen win into the rule, and the nominal error rate is void. Restricting attention to a primary can be one strategy against many-metric inflation; this leaf is the moment of the lock—before start, not after the table.

Why it happens

If the decision rule can wait for results and then choose a metric, the choice leans toward whichever happened to clear a line. Among several related endpoints, one will look good under noise; the chosen magnitude and p have been selected and can no longer be read as a single test. The timestamp is checkable: did the plan, the dashboard config, and the instrumentation freeze precede the outcome window. Saying “we always cared most about retention” while retention appears in the documents only at the weekly meeting means the lock never happened. Changing the primary mid-flight is a new confirmatory study; data from the old window cannot give the new metric unselected evidence. Locking does not forbid collecting secondaries; it forbids promoting a secondary after the fact.

Studying it

Write the primary’s full operationalization into a plan timestamped before the split: event name, attribution window, exclusions, per person or per session. The config repo or experiment platform should export the history of that field, not a memory. Analysis reads the frozen field first, then opens the rest. If the definition must change (instrumentation was wrong), record a deviation: the new definition either uses only yet-unexposed data, or the whole run drops to exploratory. Audit: does the primary in the conclusion match the launch record? On a mismatch, downgrade the conclusion. Do not confuse “primary” with “the chart we talked about last.”

Where it stops holding

A sequential design may stop under a prewritten rule; what stops is the sample, not a newly chosen endpoint. When the primary is a composite (complete, then no complaint), the composite rule is frozen too; the composite may not be split after the fact to keep one piece. If a platform failure makes the original primary unusable, a backup named beforehand may step in; naming one on the spot is still switching. Exploratory work has no primary; do not add one at the end of exploration and pretend the run was confirmatory.

Applying it

• Before the split switch is thrown, the primary field must already be frozen on the experiment platform; the freeze record goes in the ticket.
• Line one of the weekly pack cites the frozen definition. If it disagrees with the live dashboard, fix the definition first; do not change the conclusion.
• Do not offer a dropdown in the results meeting to “set which one is primary.”
• Check: compare the launch-day config snapshot with the conclusions page. If the metric or the window differs, relabel the conclusion exploratory; it is not a ship decision.

Related

• Same group: Q3.22.1 Testing many metrics raises false positives · Q3.22.2 Correct the threshold or name a primary in advance · Q3.22.3 Reporting the significant slices is data peeking
• Adjacent: Q3.21 Statistical significance and effect size · Q3.04 A/B testing
• Search terms: primary metric · outcome switching · pre-specification