Practice Set: A/B Testing

These six problems add no new methods. They practice producing a complete answer under changing constraints: clarify the decision, state the estimand and units, choose metrics, justify the sample size or readout, check validity, make a conditional decision, and update the plan under follow-ups. Each proposed fix needs two parts: the scorecard change we expect if the diagnosis is correct and what a missing change would tell us. For each problem, read the scenario and question, answer aloud in two minutes, open the hints only if you are stuck, and then compare your answer with the solution.

Problem 1: Pricing page design

Scenario: A pricing page now shows the annual plan first instead of the monthly plan, making it the default selection. We have 21 days, limited traffic, and a concern that customers may feel misled. How would we decide whether to keep the change? The decision is whether to launch it to all new visitors within three weeks.

Question: Should we launch the annual-first pricing page to all new visitors within three weeks, and how would we make that decision?

Solution

A defensible answer begins by clarifying whether the goal is completed paid subscriptions, annual-plan revenue, or lower cancellation risk. Suppose the decision concerns completed subscriptions among eligible new visitors who reach the pricing page during the 21-day test. The estimand is the treatment-control difference in completed subscription rate for that population under assignment.

Assign at the user level so repeat visits retain the same version. Use completed subscription rate as the primary metric, not annual-plan clicks or default-plan selections. Those intermediate actions can rise while customers abandon payment. Use cancellation within the available observation window and payment-failure rate as guardrail metrics. If immediate cancellation data are too immature for a launch decision, say so, and use an explicit staged rollout rather than implying the guardrail was measured.

The required denominator depends on the claim. For page performance, analyze eligible users who actually reached the pricing page, provided exposure logging is reliable. Do not divide completed subscriptions by every assigned user when a substantial share never had the opportunity to see either page. Check treatment and control assignment counts, exposure rates, and missing purchase events before reading lift.

The two constraints in the prompt each change the plan. Limited traffic means checking, before launch, whether 21 days of eligible pricing-page visitors can detect the smallest subscription lift worth shipping. If they cannot, say so up front rather than reading an underpowered result as no effect. The concern that customers may feel misled is a guardrail, not a footnote. Pre-specify early cancellations, refund requests, and support contacts about unexpected annual charges so that a default that wins subscriptions by surprising people shows up as harm.

The likely failure mode is that the new default increases annual-plan selection while lowering completed subscriptions or raising rapid cancellations. If the design is genuinely easier to understand, treatment should improve completed subscriptions without a guardrail regression. If plan selection rises but completed subscriptions do not, the default changed a proxy rather than customer value.

Problem 2: Notification frequency

Scenario: A team randomizes notification sends. A user can receive treatment at breakfast and control at lunch. The readout ...