Simpson's Paradox and Multiple Comparisons
Explore Simpson's paradox and multiple comparisons in A/B testing, learning to detect when aggregate results contradict subgroup data due to weighting or sampling issues. Understand how over-segmentation inflates false positives, and practice methods to validate experiment findings, ensuring accurate and defensible decisions in data science interviews.
The interviewer hands over a checkout readout and asks, “Would we ship treatment based on this result?”
Overall: control converts
240/2,000 = 12.0%; treatment converts340/2,000 = 17.0%.Mobile: control converts
160/1,600 = 10.0%; treatment converts36/400 = 9.0%.Desktop: control converts
80/400 = 20.0%; treatment converts304/1,600 = 19.0%.
Treatment appears to win overall, yet it loses on both platforms. A weak answer says that 17.0% exceeds 12.0%, so treatment improved conversion. That answer reads the aggregate without checking whether the aggregate represents comparable populations.
The stronger answer begins with the contradiction. “Treatment is lower on mobile and ...