Sampling and Confidence Intervals
Explore the principles of confidence intervals and sampling distributions to understand how statistical estimates vary across samples. Learn to correctly interpret confidence intervals, avoid common misconceptions, and apply concepts like the central limit theorem and standard error. This lesson helps you evaluate estimate reliability, check assumptions, and communicate uncertainty effectively in data science contexts.
Variant B has an observed conversion rate of 12.4%, with a 95% confidence interval of 11.8% to 13.0%. The interviewer asks, "How should you interpret this confidence interval?" A common misinterpretation is that the true conversion rate has a 95% probability of falling between 11.8% and 13.0%. This interpretation incorrectly assigns a probability to the fixed population parameter after the confidence interval has been calculated.
In frequentist statistics, the true population conversion rate is treated as a fixed but unknown parameter. Randomness comes from the sampling process, which produces different estimates and confidence intervals across repeated samples.
A confidence interval is a range produced by a procedure designed to cover a fixed population parameter at a stated long-run rate. If we repeatedly sampled comparable groups of users, calculated a 95% confidence interval each time, and used the same valid procedure, about 95% of those intervals would contain the true conversion rate.
The incorrect interpretation confuses the variability of confidence intervals across repeated samples with the coverage of a specific calculated interval. Once the 95% confidence interval of 11.8% to 13.0% has been calculated, it either contains the true population conversion rate or it does not. Because the true population conversion rate is unknown, we cannot determine whether this particular interval contains it.
A stronger answer starts by stating what repeated sampling would show:
With this sampling method, 95% of intervals constructed across repeated samples would contain the true conversion rate. This interval gives a range of plausible values for the true conversion rate, but it does not assign a 95% probability to the fixed true rate.
A sharper follow-up is, “Roughly how many users does that interval imply?” A margin of 0.6 percentage points at 95% means a standard error of about 0.6 / 1.96 ≈ 0.31 points. For a proportion,