Reading a Results Readout

The interviewer hands over the scorecard below and asks, “What do you recommend?” The primary metric’s difference is +0.3 percentage points, or +6% relative lift, with a 95% confidence interval from +0.11 to +0.49 percentage points. The scorecard also reports assignment counts, exposure rates and two guardrails.

Before interpreting the lift, we check whether these rows describe comparable populations and the same unit. The earlier sizing plan assumed eligible users, so the readout must use eligible users as its denominator rather than raw visits, sessions, or every assigned account. We then inspect the assigned sample counts, exposure rates, and outcome completeness. A precise interval cannot repair a broken assignment or a treatment-only logging path.

The first fields to read are the assignment counts, the denominator definition, the conversion counts, the primary-metric rates, and any guardrail metrics. Only after those checks support the table do we interpret the observed 0.3 percentage-point difference.

State lift in units that support a decision

With the scorecard interpretable, the useful business statement is that treatment produced 0.3 additional conversions per 100 eligible users, compared with control. That is the absolute lift, the treatment rate minus the control rate, expressed here in percentage points.

The +6% figure is the relative lift because it divides the absolute lift by the control rate.

Both calculations are correct, but they answer different communication needs. Relative lift makes change easier to compare across baselines. Absolute lift tells us how much customer behavior actually changed. If baseline conversion were 0.1% and treatment reached 0.2%, relative lift would be 100% while the operational gain would still be one extra conversion per 1,000 eligible users. A large relative percentage can therefore ...