Newton’s Method

The second-order optimization algorithms

Newton’s methods are a class of optimization algorithms that leverage second-order information, such as Hessians, to achieve faster and more efficient convergence. In contrast, the gradient descent algorithms, like the Nesterov momentum, depend solely on the first-order gradient information.

The idea of Newton’s method is to utilize the curvature information present in Hessians to get a more accurate approximation of the function near the optimum.

Recall the two-degree Taylor series expansion of our objective $f(x)$ around a point $x_t$ (at the time $t$)​ as follows:

where $\nabla_x f(x_t)$ and $H(x_t)$ represents the gradient and the Hessian of the function $f(x)$ at $x = x_t$ ...