Search⌘ K
AI Features
Home
Courses
Data Science Bootcamp

Gradient Boosting Machines

Explore how gradient boosting machines work by sequentially training weak learners to predict and correct residual errors from previous models. Understand the role of residuals, learning rate, and loss functions in refining predictions for regression problems. This lesson builds a foundation to apply and validate gradient boosting models effectively using boosted tree algorithms.

We'll cover the following...

A delivery-time model that predicts every order will take 32 minutes ignores a useful pattern. Orders traveling 2 kilometers may arrive in 20 minutes, while orders traveling 12 kilometers may take 44 minutes. The mean prediction gives every row a starting point, but its errors are not random noise. They contain information that a later model can correct.

A residual is the actual outcome minus the current prediction. If an order actually takes 44 minutes and the model predicts 32, its residual is 44−32=1244 - 32 = 12 minutes. A positive residual means the model predicted too low. A negative residual means the model predicted too high.

The code below begins with the mean delivery-time prediction and calculates the residual for each order. It also summarizes the remaining error after that simple baseline.

Python
import pandas as pd
def main() -> None:
    df = pd.DataFrame(
        {
            "distance_km": [2, 4, 6, 8, 10, 12],
            "actual_minutes": [20, 24, 30, 35, 39, 44],
        }
    )
    baseline_mean = df["actual_minutes"].mean()
    df["predicted_minutes"] = baseline_mean
    df["residual"] = df["actual_minutes"] - df["predicted_minutes"]
    mean_residual = df["residual"].mean()
    mae = df["residual"].abs().mean()
    mse = (df["residual"] ** 2).mean()
    print("Delivery-time baseline and residuals")
    print(df.to_string(index=False))
    print("\nSummary error metrics")
    print(f"Baseline mean prediction (minutes): {baseline_mean:.1f}")
    print(f"Mean residual (minutes): {mean_residual:.1f}")
    print(f"Mean absolute residual (minutes): {mae:.3f}")
    print(f"Mean squared error (minutes^2): {mse:.3f}")
if __name__ == "__main__":
    main()
    ...