Question: Interpreting Mixed Results of A/B Tests
Explore how to interpret mixed A/B test outcomes with a strategic framework that balances trade-offs between conversion rate and average order value. Learn to evaluate net revenue impact, identify user cohort effects, and decide whether to ship, revise, or re-run experiments using clear, metric-driven reasoning.
We'll cover the following...
Interview question
You ran an A/B test on a new feature that simplifies the checkout process. It decreased conversion rate by 3% but increased Average Order Value (AOV) by 8%. How do you interpret this?
In this lesson, you’ll learn how to interpret mixed experiment results on an e-commerce platform using a disciplined, analytical ACT-3 mindset. You’ll practice evaluating:
Whether the primary metric regression is acceptable
Whether the counter-metric lift is meaningful
Whether the experiment should ship, be revised, or be re-run
The goal is to demonstrate experimental judgment focusing on net revenue impact rather than single metric movements.
Scenario overview
Your interviewer asks:
Your new, simpler checkout flow reduced the number of people who complete a purchase by 3%, while increasing the average value of each successful order by 8%. What should we do?
This question tests whether you can:
Analyze trade-offs between volume (conversion) and value (AOV).
Interpret metrics through net revenue, not just individual metric changes.
Identify the underlying drivers behind both changes. ...