Practice Set: Applied Business Statistics

These six problems prepare you to evaluate statistical evidence and communicate findings in product discussions. A product manager asks, “The conversion rate changed after a checkout update. Is the observed difference statistically significant, and what can we conclude?” Before selecting a statistical method, examine the available data, study design, and question being evaluated. Determine whether the evidence supports statistical inference or only a descriptive comparison.

For each problem, define the statistical question, select an appropriate method, validate the result, and explain its business implications and limitations. The six problems cover confidence interval interpretation, p-values, statistical test selection, regression output interpretation, survivorship bias, and the distinction between association and causation. Each problem includes two hints and a worked solution that demonstrates how to explain the reasoning during an interview. Attempt each problem independently before reviewing Hint 1, Hint 2, and the worked solution, in that order.

Problem 1: A meaningful estimate with an interval crossing zero

Scenario: A retention analysis estimates that a new account-recovery email changed $30$ -day retention by $+0.6$ percentage points. Its $95\%$ confidence interval runs from $-0.3$ to $+1.5$ percentage points.

Question: How should we interpret this interval, and should product keep the new email?

Solution

The interval includes zero, so the data are compatible with no retention change. We should not say there is a $95\%$ chance that the true effect lies inside this completed interval. The correct statement is that the procedure would contain the fixed true effect in $95\%$ of repeated samples, assuming the sampling conditions hold.

The point estimate still matters. A $+0.6$ percentage-point increase may be commercially valuable, but the interval also permits a $-0.3$ percentage-point decrease. If sending the email has negligible cost and downside risk is low, product could keep the change while collecting more independent observations. If sending the email creates support burden or regulatory risk, we should hold the rollout until the uncertainty narrows.

A quick check is directional. Because zero falls between $-0.3$ and ...