Kernel Linear Regression

Single target example

It’s possible to reformulate generalized linear regression to incorporate the kernel trick. For example, the loss function $L(\bold w)$ for generalised linear regression with a single target is as follows:

$$L(\bold w)= \|\phi(X) \bold w-\bold y\|_2^2 + \lambda \|\bold w\|_2^2$$

Note:

$$\bold w^T\bold w = \|\bold w\|_2^2$$

Setting the derivative of the loss with respect to $\bold w$ to $\bold 0$ results in the following:

$$\begin{align*} & \phi(X)^T(\phi(X)\bold w-\bold y)+\lambda \bold w = \bold 0 \\ & \bold w = -\frac{1}{\lambda}\phi(X)^T(\phi(X)\bold w-\bold y) \\ & \bold w = \phi(X)^T\bold a \\ \end{align*}$$

Here, $\bold a=-\frac{1}{\lambda}(\phi(X)\bold w-\bold y)$.

Reparameterization

We can now parametrize the loss function with parameter vector ...