Generalized Linear Regression

We’ve previously learned that while standard linear models are powerful, many real-world relationships are non-linear. The generalized linear model (GLM) framework solves this by introducing a basis function ($\phi(\mathbf{x})$) that transforms the input features into a higher-dimensional space, allowing a linear model to fit a complex, non-linear curve to the data. In this lesson, we move from conceptual understanding to practical implementation by exploring closed-form solutions for training generalized linear models.

Single target

The input features ${x}_i \in {R}^d$ are vectors where each data point has $d$ distinct, real-valued features (e.g., size, age). The target variable $y_i \in {R}$ is a single, continuous, real-valued number (e.g., house price) that the model aims to predict, defining this as a single-target regression problem. The model $f_{{w}}(f{x}) = {w}^T\phi({x})$ is a generalized linear model (GLM). It achieves non-linear modeling by first applying a basis function $\phi({x})$ (the mapping) to transform the input features, and then making the prediction via a linear dot product with the learned parameters $\mathbf{w}$.

The function $f_{{w}}({x}) = {w}^T\phi({x})$ successfully defines the structure of our generalized linear model (GLM) for any given input ${x}$. However, this model structure is useless until we determine the ideal values for the parameter vector ${w}$. These parameters must be chosen so that the model’s predictions best match the true target values in our training dataset $D$.

To quantify how well a given set of parameters ${w}$ performs, we use a loss function $L({w})$. This function measures the total error between the model’s predictions and the actual observed values across all $n$ data points. To find the ${w}$ that provides the best fit, we must find the ${w}$ that minimizes this loss.

The optimal parameters ${w}^*$ can be determined by minimizing a regularized squared loss as follows:

$$\bold w^*=\argmin_{\bold w}\bigg\{\sum_{i=1}^n (\bold w^T\phi(\bold x_i)-y_i)^2 + \lambda \bold w^T\bold w\bigg\}$$

Here, $\sum_{i=1}^n (\mathbf{w}^T\phi(\mathbf{x}_i)-y_i)^2$ is the squared error (or data loss) term, and $\lambda \mathbf{w}^T\mathbf{w}$ is the L2 regularization term. Their sum, $\sum_{i=1}^n (\mathbf{w}^T\phi(\mathbf{x}_i)-y_i)^2 + \lambda \mathbf{w}^T\mathbf{w}$ ...