Regularization (Lasso, Ridge, and ElasticNet Regression)

Regularization

We use the term overfitting to describe a situation where a model performs well on the training dataset but fails to generalize to unseen or test data. This condition is also associated with high variance. Overfitting on the training data can be illustrated as:

$J(w) \approx 0$

In other words, our predicted values are so close to the actual values, that the cost goes to zero and the model has memorized everything.

How high variance (overfitting) can be reduced

• The first strategy is to look for more training data so that the data has more variety in it.

• Regularization, which will be the focus of this part of the lesson, is also used to tackle overfitting.

• Employ good feature selection techniques.

• There are also some specific deep learning techniques for reducing the high variance.

Now, we will look into how various regularizations are used to overcome overfitting.

Ridge regression

The following steps demonstrate how the cost function is modified in Ridge regression, also known as L2-regularization.

$J(w)$ = $\frac{1}{2m}[\sum_{i=1}^{m}(\hat{y}^i-y^i)^2 + \lambda \sum_{j=1}^{n}w_j^2]$ ...