Bootstrap Aggregating (Bagging)
Explore how bootstrap aggregating, known as bagging, addresses prediction instability in deep decision trees by training multiple models on bootstrap samples and combining their outputs through averaging or majority voting. Understand the impact on variance reduction and its application in chargeback prediction, along with practical evaluation using ROC AUC metrics to compare single trees with bagged ensembles.
A deep decision tree can make unstable chargeback decisions when small changes in its training rows produce different splits. After a legitimate transaction and a disputed transaction enter or leave a small noisy sample, the tree may send the same new payment down a different path. This instability is variance: predictions change substantially across plausible training samples.
Bagging addresses that problem by fitting many models rather than trusting one tree. A bootstrap sample draws training rows with replacement, so a row such as transaction 17 may appear twice while transaction 42 does not appear at all. Each model is a base learner. Aggregation combines their predictions, usually by majority vote for classification or by a mean for regression.
The target remains unchanged. For chargeback prediction, every tree still learns whether chargeback=1 from the same input columns. Bagging changes the training rows each tree sees, then reduces the influence of any one tree’s unstable decision.
Averaging noisy predictions
Averaging works because independent prediction errors can offset one another. Suppose ten models estimate the cost of an insurance claim as values between $1,700 and $2,350. Any one estimate may be high or low because its ...