Hypothesis Testing and p-Values
Explore how to correctly interpret p-values and hypothesis tests within data science interviews. Learn to define null and alternative hypotheses, distinguish statistical significance from business decisions, and evaluate risks from Type I and II errors. Understand test selection criteria, proper error interpretation, and apply these to practical interview scenarios, preparing you to explain results clearly and make informed data-based decisions.
An interviewer asks, "An experiment evaluating the new checkout flow reports a p-value of 0.08. What can you conclude at a significance level of 0.05?" An incorrect interpretation is, "The new checkout flow has no effect because the result is not statistically significant." This interpretation treats statistical significance as proof of effectiveness and overlooks the estimated effect size, its uncertainty, and the risks associated with the launch decision.
With a preselected 0.05, we do not reject the null hypothesis. Does that mean the feature failed? No. Does it mean we should automatically ship it? Also no.
An effective response begins by defining the null and alternative hypotheses. The null hypothesis states that there is no difference in the target metric between the existing and new checkout flows, such as a difference of zero in mean order value. The alternative hypothesis specifies either a difference between the checkout flows (two-sided test) or an improvement in a predefined direction (one-sided test). The test direction must be selected before examining the results.
For p = 0.08 means the observed result does not meet the chosen rule for rejecting the null. It does not establish that the null is ...