Ridge and Lasso Regression

In the previous lesson, we learned that regularization helps control the bias–variance trade-off and prevents overfitting by adding a penalty on large weights. Now we will look at the two most widely used regularization methods:

• Ridge (L2 regularization)

• Lasso (L1 regularization)

Both techniques shrink the model’s weights, but the type of penalty they use leads to very different results. Understanding this difference is essential because the Lasso can eliminate features entirely, while the Ridge cannot, and this comes purely from geometry.

Ridge and Lasso objectives

Both Ridge and Lasso regression are special forms of regularized linear regression. They use the simplest model type (linear model) and the standard way to measure error (squared loss), differing only in their regularization penalty.

The core model and loss function

Before introducing the penalty, we must define the model that makes a prediction and the loss function that measures the error.

Linear model ($f_{\mathbf{w}}$)

A linear model assumes the output ($\hat{y}_i$, the prediction) is a simple, weighted sum of the inputs ($x_i$). The goal is to find the best set of weights ($\mathbf{w}$) that connect the inputs to the output.

• We have $n$ training examples, $D = \{(\mathbf{x}_i, y_i) \mid 1 \le i \le n\}$. Each input $\mathbf{x}_i$ has $d$ features.
• The model expression:

$$f_{\mathbf{w}}(\mathbf{x}_i) = w_0 + w_1 x_{i1} + w_2 x_{i2} + \cdots + w_d x_{id}$$

• $w_0$ is the intercept (or bias).
• $w_1$ to $w_d$ are the slopes or feature weights.

To simplify the math, we often combine $w_0$ with the other weights by adding a constant $1$ to the start of the feature vector: $\hat{\mathbf{x}}_i = (1, x_{i1}, \dots, x_{id})$ ...