Peeking, Novelty, and Seasonality

A checkout experiment reports a +1.8 percentage-point conversion lift on day 4, with p = 0.03, so the team stops the test and declares a win. By day 14, the lift has vanished. The original p = 0.03 did not support the shipping claim because the team changed the analysis plan after seeing each daily result.

The interviewer asks, “The team checks an A/B test every day and stops when the result is significant. What is wrong with that plan?”

A weak answer is, “Nothing is wrong if the p-value is below 0.05.” That answer treats day 4 as though the team committed to testing only on day 4. In reality, the team gave random variation several chances to cross the threshold. A fixed-horizon p-value and confidence interval assume the stopping rule was fixed before results were examined.

A stronger answer is, “Repeatedly checking a fixed-horizon test and stopping at the first significant result increases the chance of declaring an effect that is not real. I would set the stopping rule before launch, then interpret the result only under that rule.” The inference contract includes the planned sample size or end date, the primary metric, and the rule for deciding when evidence is sufficient.

If an interviewer pushes back with “But we need to monitor the experiment while it runs,” we should separate operational monitoring from unplanned significance-based stopping. We can monitor assignment integrity, exposure logging, and guardrails. If the team also wants to make efficacy decisions while monitoring, it needs an analysis method designed for sequential decisions.

A concise answer is: “The test was not wrong because we looked at data. It became invalid when the team used repeated looks to decide when to stop without a rule that controlled the resulting false-positive risk.”

Repeated looks create extra chances for noise

An A/A test assigns users to two identical experiences, so its true treatment effect is zero. Under a ...