Steepest Descent

The method of steepest descent

So far, in gradient descent, we use the step size $\alpha$ to control the amount of descent we like to perform at a particular point. Choosing a large step size makes the algorithm unstable, whereas choosing a small step size requires more iterations to reach convergence.

Steepest descent works by finding the optimal step size for the gradient descent. For a convex function $f(x)$, the steepest descent update at a time $t>0$ can be written as follows:

The optimal step size $\alpha_t$ at the time $t$ can then be found by optimizing the following objective:

where $\phi_t(\alpha) = f(x_{t-1} - \alpha_t(\nabla_x f(x_{t-1}))^T)$ ...