The Bias-Variance Tradeoff
Explore the concepts of bias and variance in machine learning models to understand how they influence prediction errors. Learn to identify when models are overfitting or underfitting by examining error stability across different training samples. This lesson helps you choose the right model complexity by interpreting error patterns and introduces methods like regularization and resampling for effective model evaluation.
We'll cover the following...
A degree-1 curve and a degree-10 curve can achieve similar average test RMSE, yet the degree-10 curve may make sharply different predictions after a small change to its training rows. Training fit alone cannot explain that difference. We also need to ask whether the fitted model learned a stable pattern or reacted to the particular noise in one sample.
Bias is prediction error caused by assumptions that are too simple for the pattern in the data. A straight line fitted to a curved delivery-time relationship has high bias because it misses the curve in a repeatable way. The predictions may be stable, but they remain wrong in the same regions.
Variance is prediction error caused by a model changing substantially when its training sample changes. A highly flexible polynomial can bend toward different noisy observations in each sample. Its predictions may look excellent on one training set, then shift on new routes because a different set of observations produced a different curve.
The bias-variance tradeoff describes the tension between those errors. Reducing model complexity often makes predictions more stable and reduces variance, but may ...