Gradients of Matrices

Gradient with respect to matrices

Many times, machine learning objectives, such as minimizing the loss function of a linear regression, can be written using matrices and vectors, making them compact and easy to understand. Therefore, to make computations easier, it is worthwhile to understand how gradients are computed when matrices are involved.

The gradient of matrices with respect to vectors (or matrices) can also be computed like the Jacobian of vector-valued functions. The Jacobian can be thought of as a multi-dimensional tensor that is a collection of partial derivatives. For example, the gradient for a $m \times n$ matrix $A$ with respect to the $p \times q$ matrix $B$ will be a $(m \times n) \times (p \times q)$ Jacobian whose entries will be given as follows:

To understand better, let’s consider a function $f = Ax$, where $f \in \R^m$, $A \in \R^{m \times n}$ , and $x \in \R^n$. Here, we want to calculate the gradient $df/dA$. To do so, we will start by determining the dimensions of the Jacobian, as shown below:

By definition, the gradient is a collection of partial derivatives and, as a result, $df/dA$ can be written as follows:

Each $f_i$ can be explicitly written as $f_i = \sum_{j=1}^n A_{ij}x_j$ ...