Gradient-Solving Approach

Gradient-solving method

The gradient-solving method is a popular method to find the optimal solution of convex functions by solving for values where the gradient of the function is zero.

To understand better, consider the two-degree Taylor polynomial approximation of a convex function $f$ around a point $x$.

where $\nabla_xf(x)$ and $H(x)$ are the gradient and Hessian of the function $f$ at the point $x$.

Assume that $\nabla_x f(x) = 0$, which reduces the equation above to

Because $f$ is a convex function, its Hessian $H(x)$ is semipositive definite for any $x$, i.e., $h^T H(x) h \geq 0$. This implies that $f(x+h) \geq f(x)$ ...