Question: Interpreting Mixed Results of A/B Tests
Explore how to interpret mixed results from A/B tests in product management. Understand how to evaluate trade-offs between conversion rates and average order value using a structured framework. Gain skills to analyze root causes, segment user behavior, and make evidence-based decisions focused on net revenue impact.
We'll cover the following...
Interview question
You ran an A/B test on a new feature that simplifies the checkout process. It decreased conversion rate by 3% but increased Average Order Value (AOV) by 8%. How do you interpret this?
In this lesson, you’ll learn how to interpret mixed experiment results on an e-commerce platform using a disciplined, analytical ACT-3 mindset. You’ll practice evaluating:
Whether the primary metric regression is acceptable
Whether the counter-metric lift is meaningful
Whether the experiment should ship, be revised, or be re-run
The goal is to demonstrate experimental judgment focusing on net revenue impact rather than single metric movements.
Scenario overview
Your interviewer asks:
Your new, simpler checkout flow reduced the number of people who complete a purchase by 3%, while increasing the average value of each successful order by 8%. What should we do?
This question tests whether you can:
Analyze trade-offs between volume (conversion) and value (AOV).
Interpret metrics through net revenue, not just individual metric changes.
Identify the underlying drivers behind both changes. ...