Fundamentals of Machine Learning: A Pythonic Introduction/

...

/

Regularization

Regularization

Learn what regularization is and how it affects validation and training error.

We'll cover the following...

Regularization is a technique to reduce the variance of a model. One such way is restricting the parameters to a subset of the parameter space. Reduction in variance turns out to be a prevention of overfitting.

Why not choose a simple model?

Starting with a model that’s too simple and gradually increasing its complexity by monitoring its performance on the testing data is one solution. Regularization does the reverse, that is, starting with a complex model and decreasing its complexity.

Press + to interact

The answer to this question of why not choose a simple model has to do more with implementation than theory. Regularization is more systematically implementable compared to increasing the model complexity gradually. Furthermore, different regularization methods offer different ways to reduce the variance of the model, where one way might be better than the other for a task at hand.

Shrinkage method

In shrinkage-based regularization, the parameters are restricted to stay close to zero (shrink to zero). One way is to apply this restriction explicitly while minimizing the loss, that is, minimize L(fw(x),y)L(f_{\bold w}(\bold x),\bold y) subject to α1<w1<β1,α2<w2<β2,…,αk<wk<βk\alpha_1 < w_1 < \beta_1, \quad \alpha_2 < w_2 < \beta_2, \quad \dots, \quad \alpha_k < w_k < \beta_k, where LL is a loss function and w=(w1,w2,…,wk)\bold w=(w_1,w_2,\dots,w_k) ...