Power Analysis and Sample Size
Explore how to calculate the required sample size for A/B tests based on baseline conversion rates, minimum detectable effect, and power. Understand the tradeoffs between sample size, effect size, and test duration for making informed product decisions. Learn to interpret results carefully, avoid biased significance, and plan experiments that fit operational constraints.
A conversion result can only support a product decision if the experiment has enough users to detect the lift that matters before its deadline. For a checkout change with a two-week window, planning starts with a concrete question: how many eligible users must each variant receive to detect a small improvement in purchase conversion?
Power is the probability that a test detects a real effect of a chosen size at a chosen significance level. Minimum detectable effect (MDE) is the smallest true lift worth acting on. If increasing conversion from 5% to 5.1% would not justify shipping or maintaining the change, that 0.1 percentage-point lift should not drive the plan. If a 0.5 percentage-point lift would justify the change, the test should be sized to detect that lift.
The significance level, alpha, controls the Type I error rate chosen before the test. We will use alpha=0.05 and target power of 0.8 as planning examples, not universal requirements. The central tradeoff is fixed: a smaller MDE requires more users because random sampling noise can more easily hide a small real lift.
Calculate users per variant
With the decision-relevant lift defined, the next step converts the baseline conversion rate and MDE into a required number of users for variant A and variant B. The calculation assumes independent user outcomes, user-level random assignment, equal-sized variants, and ...