Designing A/B Tests and Choosing Metrics
Explore how to design reliable A/B tests by defining clear hypotheses, selecting appropriate primary and guardrail metrics, and choosing the correct randomization unit. Understand how to protect experiment integrity and specify tests to make confident, data-driven decisions about business changes.
A checkout redesign must increase completed purchases without making checkout slower or increasing refunds. A before-and-after chart cannot answer that question reliably. Friday traffic may contain more returning customers than Monday traffic, a promotion may change who visits the site, and seasonality may change purchase behavior even when checkout stays unchanged.
An A/B test assigns eligible traffic at random to variant A, the control, or variant B, the treatment. Both variants run during the same period, so they face similar traffic patterns on average. Random assignment does not guarantee that every small group matches exactly. It makes the groups comparable on average, which gives a difference between A and B a more credible connection to the checkout change.
The unit of analysis is the entity whose outcome we measure. For purchase conversion, a common unit of analysis is an eligible user. Each eligible user contributes one outcome, such as whether that user completed at least one purchase during the test window. A session or an order can also be a unit of analysis, but the choice ...