Search⌘ K
AI Features
Home
Courses
Data Science Bootcamp

Pitfalls: Peeking, Novelty Effects, and Multiple Comparisons

Explore key pitfalls in A/B testing such as peeking, novelty effects, and multiple comparisons that can inflate false-positive rates and mislead conclusions. Learn how to predefine analysis windows, fix stopping rules, and properly control multiple testing to ensure reliable, valid experimental results. This lesson helps you design, analyze, and interpret A/B tests to avoid common validity errors and make sound business decisions based on trustworthy data.

We'll cover the following...

A dashboard reports p < 0.05 for a checkout treatment on day 3, so the result looks ready to ship. By day 14, the conversion lift has disappeared. The hypothesis test may have been calculated correctly, yet the conclusion was still wrong because the decision process changed the evidence being used.

A correctly sized experiment needs more than enough users. Repeatedly checking results, treating an early behavioral spike as a lasting effect, or searching across many outcomes can create a convincing but false win. These are validity problems in the experiment plan and readout process, not arithmetic mistakes in a p-value calculation.

Peeking changes the chance of a false win

When the true conversion rate is 0.05 in both variants, a single test at the planned end of a 14-day experiment has a small chance of producing p < 0.05 by random variation. Peeking means checking that same experiment repeatedly and ending it as soon as a favorable result appears.

The wrong intuition is that each daily check uses the same 0.05 threshold, so the overall experiment still has the same false-positive risk. Each check creates another opportunity for ordinary noise to cross the threshold. Stopping at the first crossing keeps the favorable accident and discards later evidence that might reverse it.

The simulation compares two policies across many experiments with no real treatment effect. One policy tests only after day 14. The other tests every day and labels the experiment a win at the first result below 0.05.

Python
import math
import random
def normal_cdf(x: float) -> float:
    return 0.5 * (1.0 + math.erf(x / math.sqrt(2.0)))
def two_proportion_ztest_pvalue(success_a: int, n_a: int, success_b: int, n_b: int) -> float:
    if n_a == 0 or n_b == 0:
        raise ValueError("Sample sizes must be positive")
    p_pool = (success_a + success_b) / (n_a + n_b)
    se = math.sqrt(p_pool * (1.0 - p_pool) * (1.0 / n_a + 1.0 / n_b))
    if se == 0.0:
        return 1.0
    z = (success_b / n_b - success_a / n_a) / se
    return 2.0 * (1.0 - normal_cdf(abs(z)))
def simulate_ab_null(
    n_sims: int = 2000,
    days: int = 14,
    n_daily: int = 1000,
    p: float = 0.05,
    alpha: float = 0.05,
    seed: int = 7,
) -> tuple[float, float]:
    rng = random.Random(seed)
    false_positive_fixed_final = 0
    false_positive_daily_peek = 0
    for _ in range(n_sims):
        cum_conv_a = 0
        cum_conv_b = 0
        cum_n_a = 0
        cum_n_b = 0
        peek_win = False
        for day in range(1, days + 1):
            conv_a = sum(1 for _ in range(n_daily) if rng.random() < p)
            conv_b = sum(1 for _ in range(n_daily) if rng.random() < p)
            cum_conv_a += conv_a
            cum_conv_b += conv_b
            cum_n_a += n_daily
            cum_n_b += n_daily
            if not peek_win:
                pval_day = two_proportion_ztest_pvalue(cum_conv_a, cum_n_a, cum_conv_b, cum_n_b)
                if pval_day < alpha:
                    peek_win = True
        pval_final = two_proportion_ztest_pvalue(cum_conv_a, cum_n_a, cum_conv_b, cum_n_b)
        if pval_final < alpha:
            false_positive_fixed_final += 1
        if peek_win:
            false_positive_daily_peek += 1
    return false_positive_fixed_final / n_sims, false_positive_daily_peek / n_sims
def main() -> None:
    fixed_final_rate, daily_peek_rate = simulate_ab_null()
    print("False-positive rates under the null (p_A = p_B = 0.05)")
    print(f"Strategy A (test once at Day 14): {fixed_final_rate:.3f}")
    print(f"Strategy B (peek daily, stop at first p < 0.05): {daily_peek_rate:.3f}")
if __name__ == "__main__":
    main()

  • Lines 9–19: The code computes a standard two-proportion z-test p-value using the pooled conversion rate under the null; returning 1.0 when the standard error is zero avoids division-by-zero when both groups have identical all-0 or all-1 outcomes.

  • Lines 43–55: The code implements the “peeking” policy by re-testing the cumulative data each day and locking in a win the first time p < 0.05, which is what inflates the overall false-positive rate.

  • Lines 57–63: The code separates the two decision rules: Strategy A only counts significance at Day 14, while Strategy B counts any earlier crossing (even if the final Day-14 p-value is not significant). ...