Regularization

Regularization is a technique to reduce the variance of a model. One such way is restricting the parameters to a subset of the parameter space. Reduction in variance turns out to be a prevention of overfitting.

Why not choose a simple model?

Starting with a model that’s too simple and gradually increasing its complexity by monitoring its performance on the testing data is one solution. Regularization does the reverse, that is, starting with a complex model and decreasing its complexity.

The answer to this question of why not choose a simple model has to do more with implementation than theory. Regularization is more systematically implementable compared to increasing the model complexity gradually. Furthermore, different regularization methods offer different ways to reduce the variance of the model, where one way might be better than the other for a task at hand.

Let’s explore one approach to regularization called the shrinkage method, which “shrinks” model parameters to reduce overfitting.

Shrinkage method

Shrinkage-based regularization keeps model parameters (weights) close to zero. This reduces the impact of any single parameter and prevents the model from fitting the noise in data.

Instead of setting exact upper/lower limits for each parameter, we can combine parameter shrinkage and loss minimization into a single goal:

$$\min_{\bold w}\{L(f_{\bold w}(\bold x),\bold y) + R(\bold w)\}$$

• $L(f_{\mathbf{w}}(\mathbf{x}), \mathbf{y}) =$ loss (how well the model fits the data)
• $R(\bold w) =$ shrinkage function (penalty for large weights)
• $\bold w =$ vector of model parameters

Shrinking the parameters helps prevent overfitting while still trying to fit the data well.

Note: The goal is to minimize the loss $L$ while shrinking the parameters $\bold w$ as much as possible.

Shrinkage functions

The regularization term $R(\mathbf{w})$ (the shrinkage function) is calculated using mathematical norms of the weight vector ($\mathbf{w}$). The two most popular choices are the L2 norm and the L1 norm.

• $L2$ norm of a vector $\mathbf{w}$ is denoted as $\|\mathbf{w}\|_2$.
• $L1$ norm is denoted as $\|\mathbf{w}\|_1$.

Let’s examine the L2 norm.

Implementation of L2 norm

The L2 norm measures the straight-line distance (Euclidean distance) of the weight vector from the origin.

Mathematically, given the weight vector $\mathbf{w} = \begin{bmatrix} w_{0} \\ w_{1} \\ \vdots \\ w_{n} \end{bmatrix}$ ...