Telling novelty from learning takes a split by people; the blended curve confuses them

What it is

Novelty and learning can run at the same time in different people: newcomers and light users are pulled in by the new object, skilled users slow down while they relearn. Add everyone into one series and the rise and the fall cancel, or only the shape of the larger group remains. A team then reads “flat overall” or “better overall” while both mechanisms are running. Separating them requires plotting by pre-release experience, frequency, or whether someone saw the old version—not guessing from a mixed trajectory.

Why it happens

The aggregate is a weighted sum of group trajectories. The heaviest group writes the visible shape. A redesign week that also brings a wave of new users will lift the total and hide veterans’ trough; a base made mostly of skilled users will drop first and look as if novelty never happened. Mixing also fabricates false turning points: the day veterans start recovering may be the day newcomers cool off, and the total looks like fall-then-rise learning when it is two groups out of phase. Without a split, a shape diagnosis has no one to attach to.

Studying it

Lock grouping variables before looking at the total, using states observable before the release: whether a critical task was completed on the old version, activity in a recent window, whether the install is new. Estimate each group’s post-release path, and report whether group weights themselves move over time. Test whether shapes oppose or sit out of phase across groups. If newcomers only rise then fall and veterans only fall then rise, both mechanisms hold and the blended curve is not mechanism evidence. Do not group on behavior that exists only after the release (people who “love the new feature”); that writes the outcome into the grouping rule.

Where it stops holding

Very fine splits make each line noisy; pre-limit the number of primary contrast groups. Experience labels misfire: reinstalls look new, shared devices mark a skilled user as a novice. Some products have almost nobody who never saw the old version; splits still help (frequent versus infrequent) but cannot manufacture a pure novelty control. The blended curve remains useful for capacity and planning. It cannot carry a causal story about which mechanism ran.

Applying it

• Default a redesign report to three charts: brand-new users, infrequent veterans, frequent veterans. Put the blended chart last and mark the weights.
• If the three charts disagree in direction, do not issue a single sentence that “users liked” or “users disliked” the change.
• When a campaign overlaps the redesign, put campaign-acquired newcomers in their own group so acquisition is not read as novelty.
• Check: drop the largest group and see whether the blended shape flips. If it flips, what was being read was composition, not one mechanism.

Related

• Same group: Q3.14.1 Early metric movement after a change can be transitory · Q3.14.2 Learning costs temporarily suppress metrics for existing users · Q3.14.3 Observation windows must be long enough · Q3.14.4 Novelty typically rises then falls; learning typically falls then rises · Q3.14.6 Engagement metrics pick up novelty more than task success · Q3.14.7 Major redesigns void historical baselines
• Adjacent: Q3.12 Funnel and retention analysis · Q1.08 Sample size
• Search terms: segmented trajectories · aggregation masking · user tenure split