The Random Forest Algorithm
Explore the random forest algorithm and how it enhances bagging by introducing feature randomness. Understand how varying feature subsets at each split reduces correlation among trees, leading to better variance reduction. Learn to fairly compare classifiers using ROC AUC and discover key parameter effects like n_estimators, max_features, and min_samples_leaf to tune model performance on classification tasks.
Bagging reduces variance by fitting trees on different bootstrap samples, but the trees can still make similar decisions. When tenure_months strongly predicts employee attrition, many bagged trees choose tenure_months near the root. Their errors then overlap, so voting across them provides less benefit than voting across genuinely different trees.
A random forest is a bagged collection of decision trees that adds a second source of randomness. At every split, each tree considers only a random subset of the available features. The tree still chooses the best split from the features it received, but it cannot always choose the globally strongest predictor.
This feature restriction makes trees less correlated. Less-correlated tree errors cancel more effectively when the forest combines their positive-class probabilities. Random forests therefore target the same main problem as bagging, variance from unstable trees, while improving the diversity of the trees being averaged.
Feature randomness creates different trees
When every tree can examine all features, tenure_months and overtime_hours may repeatedly win the earliest splits because both separate attrition outcomes well. Bootstrap sampling changes the rows slightly, but the same strong predictors can still dominate the paths.
The setting max_features controls how ...