Key Hyperparameters: Learning Rate, Depth, and Estimators
Explore the impact of key hyperparameters—learning rate, max depth, and number of estimators—on gradient boosting models. Understand how to balance these settings using validation RMSE, early stopping, and train-validation gaps to build models that generalize well. This lesson guides you through disciplined tuning practices for improved model selection and performance.
A boosted model can fit its training data well while making different predictions on new records. For example, two configurations may produce similar training root mean squared error (RMSE), yet one has validation RMSE of 0.55 and the other reaches 0.63. The lower validation RMSE is the useful result because the validation partition represents records the model did not fit.
Three settings create most of this tension. learning_rate scales each new tree’s correction. n_estimators sets the maximum number of trees the model may add. max_depth limits how many splits each tree may make. We change these settings against one fixed validation split, not against the held-out test set, so the comparison measures the configuration rather than a changing sample.
The wrong intuition is that more trees or deeper trees must improve a model. More capacity can reduce training error while teaching the ensemble details that do not repeat outside the training partition. Early stopping gives the model a limit by monitoring validation performance and retaining the iteration that performed best there.
One change: A tuning result is interpretable only when the dataset split, metric, and all settings except the one under study remain fixed.
Balance learning rate and estimator budget
Because ...