Hypotheses, Metrics, and Randomization
Explore how to design A/B tests by defining clear hypotheses, choosing suitable primary and guardrail metrics, and establishing reliable randomization methods. Understand the importance of specifying decision targets, sample sizes, and success criteria before experimentation to ensure valid and actionable results. This lesson helps you create defensible testing plans that align metrics with product goals and handle risks like interference and leakage.
The interviewer asks, “We changed checkout to load faster. How would you test whether it improved conversion?” Behind that prompt sits a vaguer business question, “Should we ship a faster checkout?”, and that is not yet an experiment question. Before comparing any numbers, we need to name the decision, the affected population, and the period the decision covers. For example, the decision might be whether to release a checkout page that loads faster for eligible signed-in shoppers in the United States over the next two weeks.
The decision target is the estimand, the precise effect we want to learn from the experiment. A useful estimand for this prompt is the difference in completed-purchase conversion rates between the faster checkout and the current checkout among eligible signed-in shoppers who are assigned a variant during the two-week experiment.
That sentence excludes implementation details such as clicks, page-load events, or a particular statistical test. Those details may help measure the estimand, but they are not the decision target. If the product question concerns completed purchases, an estimand about checkout_button_clicks answers a different question.
A defensible pre-launch specification begins with three blanks filled in:
The decision is whether to release the faster checkout.
The population is eligible signed-in shoppers in the United States.
The time window is the planned experiment period, not whichever days later look favorable.
Once we can state that sentence, we can choose a hypothesis, a success criterion, metrics, and an assignment plan that all answer the same question.
Pre-register the claim before results exist
The estimand becomes actionable when we state a hypothesis and a success criterion before assignment begins. For the faster checkout, the directional hypothesis could be that the faster checkout increases purchase conversion among eligible assigned shoppers.
The primary metric is the one pre-specified outcome that determines whether the experiment achieved its intended product value. Here, it could be user-level purchase conversion:
A practical decision also needs a minimum effect worth shipping. Suppose the product team considers an absolute ...