Search⌘ K
AI Features
Home
Courses
Optimization for Machine Learning with NumPy and SciPy

Steepest Descent

Explore how the steepest descent method improves gradient descent by optimally selecting step sizes to accelerate convergence. This lesson guides you through implementing steepest descent on a quadratic convex function, demonstrating faster optimization compared to fixed step gradient descent. You'll learn to apply grid search to find optimal step sizes and visualize the path to an efficient solution using NumPy and Matplotlib.

We'll cover the following...

The method of steepest descent

So far, in gradient descent, we use the step size α\alpha to control the amount of descent we like to perform at a particular point. Choosing a large step size makes the algorithm unstable, whereas choosing a small step size requires more iterations to reach convergence.

Steepest descent works by finding the optimal step size for the gradient descent. For a convex function f(x)f(x), the steepest descent update at a time t>0t>0 ...