Sample Size and Power Calculations
Explore how to calculate sample size and statistical power for A-B testing experiments. Understand concepts like the minimum detectable effect, baseline conversion rates, significance levels, and allocation ratios. Learn to translate sample size into test duration given traffic constraints, and defend experiment design choices against calendar and business requirements. This lesson helps you design feasible, valid conversion experiments with proper statistical rigor.
The interviewer asks, “We have 3,000 eligible checkout users per day and can run a test for two weeks. How would you choose the sample size?”
A conversion experiment cannot meet a fixed calendar deadline through optimism alone. Before launch, we must decide the smallest conversion change worth detecting and calculate whether eligible traffic can reveal that change within the window.
For a checkout experiment, suppose historical conversion is 0.10, or 10%. The team considers an increase to 0.11 worth shipping because it adds one conversion per 100 eligible users. That one-percentage-point change is the minimum detectable effect (MDE), the smallest effect the experiment is designed to reliably distinguish from noise.
A two-group proportion calculation needs five inputs fixed before results exist:
Baseline conversion rate, such as
0.10, from historical data or an explicit planning assumption.MDE, stated as the treatment conversion rate implied by the business threshold, such as
0.11versus0.10.Significance level α, the tolerated false-positive probability.
Power, the probability of detecting the planned effect if that effect is real.
Allocation ratio, the planned treatment-to-control sample-size ratio.
Each input has a cost. A smaller MDE requires more users because random variation can hide a smaller gap. Higher power requires more users because the test must miss the planned effect less often. A lower α also raises the requirement because the evidence threshold becomes stricter. Equal allocation uses traffic efficiently for a simple two-arm comparison. A 90/10 split generally needs more total traffic than a 50/50 split to obtain the same precision.
The baseline rate also matters, but not through one universal rule. We must state whether the MDE is an absolute change or a relative change, then recompute the effect size. A drop from 10% to 5% with a fixed relative lift makes the absolute gap smaller, usually making the test less feasible. With a fixed percentage-point gap, the answer depends on ...