Limitations of Gradient Descent

We have seen how well gradient descent works in the case of convex optimization because of the presence of a single global optimal solution. We will now look at some of the limitations of gradient descent and address them in this chapter.

Intractability

Consider a machine learning problem where we want to minimize the discrepancy between the model prediction $f_\theta(x_i)$ and the ground-truth label $y_i$, as follows:

Here, $\mathcal{L}$ is an arbitrary loss function, such as cross-entropy, to measure the discrepancy between the predicted and the ground-truth value. The gradient descent update for the objective above at any time $t$ can be written as follows:

To compute the gradient $\nabla_\theta J(\theta)$ ...