Search⌘ K
AI Features
Home
Courses
Data Science Bootcamp

Statistical Significance in A/B Tests

Explore how to assess the effectiveness of A B tests by calculating conversion rates, absolute and relative lifts, and constructing confidence intervals. Learn to interpret p values and understand the implications of statistical significance for making informed decisions with experimental data.

We'll cover the following...

A checkout experiment recorded 520 purchases from 10,000 users in variant A and 575 purchases from 10,000 users in variant B. The raw counts differ, but the decision depends on whether the conversion increase is credible evidence of improvement or plausible sampling noise.

For conversion, each user either purchases or does not purchase. The conversion rate for each variant is the number of conversions divided by the number of assigned users. We measure absolute lift as the difference between variant B’s conversion rate and variant A’s conversion rate.

We measure relative lift as that difference compared with variant A’s conversion rate.

The following calculation uses the complete result table and reports both point estimates.

Python
import pandas as pd
def main() -> None:
    results = pd.DataFrame(
        {
            "variant": ["A", "B"],
            "users": [10_000, 10_000],
            "conversions": [520, 575],
        }
    )
    results["conversion_rate"] = results["conversions"] / results["users"]
    p_a = float(results.loc[results["variant"] == "A", "conversion_rate"].iloc[0])
    p_b = float(results.loc[results["variant"] == "B", "conversion_rate"].iloc[0])
    absolute_lift = p_b - p_a
    relative_lift = absolute_lift / p_a
    display_table = results.copy()
    display_table["conversion_rate"] = (display_table["conversion_rate"] * 100).map(lambda x: f"{x:.2f}%")
    print(display_table.to_string(index=False))
    print()
    print(f"Variant A conversion rate: {p_a * 100:.2f}%")
    print(f"Variant B conversion rate: {p_b * 100:.2f}%")
    print(f"Absolute lift (B - A): {absolute_lift * 100:+.2f} percentage points")
    print(f"Relative lift: {relative_lift * 100:.2f}%")
if __name__ == "__main__":
    main()

  • Lines 5–13: The code builds the complete A/B results table (users and conversions) and derives each variant’s conversion rate as conversions / users.

  • Lines 15–19: The code pulls the two conversion rates into scalars so the point-estimate lifts can be computed as absolute lift p_B - p_A and relative lift (p_B - p_A) / p_A.

  • Lines 21–22: The code keeps the underlying rates as decimals for math, but formats a separate display copy so the printed table is inspectable with 5.20% and 5.75%.

  • Lines 26–29: The code prints the requested point estimates: 5.20% vs. 5.75%, absolute lift +0.55 percentage points, and relative lift 10.58%.

This code shows variant A converting at 5.20% and variant B converting at 5.75%. The absolute lift is 0.55 percentage points, while the relative lift is about 10.6%. Relative lift can sound large when the starting conversion rate is small, so a launch decision should retain the absolute lift as well.

Put uncertainty around the conversion lift

Because the point estimate came from one finite group of assigned users, the 0.55 percentage-point lift is not guaranteed to equal the lift we would see in another valid run. A confidence interval gives a range of values compatible with the observed data and the calculation method across repeated samples.

For two independent conversion rates, the standard normal approximation estimates the uncertainty in pB−pAp_B-p_A ...