Out-of-Bag Evaluation and Feature Importance
Explore how out-of-bag (OOB) evaluation provides an internal validation method for random forests by using training samples excluded from each tree. Understand how OOB scores offer insights into model accuracy without needing the final test set. Learn to interpret impurity-based feature importance to identify which variables contribute most to splitting decisions in the forest. This lesson helps you grasp how to evaluate model generalization and explain the influence of features, emphasizing association rather than causation.
A random forest can produce a validation signal during training without using the final test set. Each tree trains on a bootstrap sample, so some training rows are absent from that tree’s sample. Those omitted rows are out-of-bag (OOB) samples for that tree.
An OOB prediction for a loan application uses only the trees that did not train on that application. An OOB score combines those held-out predictions across the training rows. OOB evaluation exists only because bootstrap sampling draws rows with replacement. A row may appear more than once in one tree’s training sample, while another row may not appear at all.
OOB evaluation gives a convenient approximation of validation performance while we compare a few random forest settings. It does not replace the held-out test set. We keep the test set untouched until we need a final, independent check of the selected model.
Tracing OOB predictions
For one row, a tree either trained on that row or left it out. Only a tree that left the row out may contribute an OOB prediction for it. If row 6 is OOB for Tree 1 and Tree 3 but in-bag for Tree 2, its OOB prediction aggregates Tree 1 and Tree 3 only.
This diagram shows that OOB predictions are assembled row by row, not from every tree. More trees usually give each training row more OOB predictions, making the ...