Statistical Significance

Statistical significance is a statement about how easily an observed difference could have arisen by chance if there were no real difference at all. The conventional test computes a p-value — the probability of seeing a result at least this extreme under that assumption — and calls the result significant when it falls below a threshold set in advance, usually 0.05. It is a guard against being fooled by noise, and it is routinely misread. Three misreadings are worth naming. A p-value is not the probability that your change works, and one above the threshold is not evidence that it does not — most often it means the sample was too small to tell. Significance is not size: with enough traffic, a difference far too small to matter commercially will clear any threshold, which is why an effect estimate with a confidence interval is more useful to a decision than a yes/no verdict. And significance applies to the test you planned, not the one you found — checking daily and stopping when the line crosses, or comparing twenty segments and reporting the one that cleared, produces false positives at a far higher rate than the threshold implies. The practical discipline is to fix the primary metric, the sample size and the duration before starting; to report the estimated effect and its interval rather than a bare verdict; and to treat segment-level findings as hypotheses for a future test rather than results. Repeated tests on the same data need an explicit correction, and a result that is significant but tiny is a reason to look at the cost of shipping it.