Novelty Effect

The novelty effect is the temporary lift a change gets because it is new rather than because it is better. Existing users notice that something moved, click it to find out what it does, and the metric rises for a week. The behavior is real, the measurement is real, and the conclusion drawn from it is wrong, because what was measured was attention paid to a difference and not value delivered by a feature. Its mirror image is change aversion, sometimes called the primacy effect: an audience with established habits is temporarily slowed by any rearrangement, so a genuine improvement reads as a regression for as long as people are relearning where things are. The two together mean a short test on an established product can be biased in either direction, and neither bias is visible in the headline number. Detection is mostly about time and cohorts. Plot the effect by days since exposure rather than pooled over the whole period; a novelty effect decays toward zero while a durable improvement holds a roughly flat gap. Split the result into users who had never seen the old version and users who had; a new-user cohort has no old habit to be curious about or to unlearn, so it estimates the steady-state effect far better than the overall average does. And where a change is large enough to justify it, hold the experiment open past the point of statistical significance, because significance answers whether an effect exists and says nothing about whether it will last. The honest limitation is that separating the two costs weeks that a team often does not want to spend, which is why the effect is so frequently named after launch rather than before it.