Search⌘ K
AI Features
Home
Courses
Optimization for Machine Learning with NumPy and SciPy

Gradient-Solving Approach

Explore the gradient-solving method to identify optimal points in convex functions by solving for zero gradients. Understand how this approach guarantees global minima and apply it to solve least squares linear regression problems. Gain practical skills to compute optimal parameters using algebraic solutions and visualize results, enhancing your ability to implement convex optimization techniques.

We'll cover the following...

Gradient-solving method

The gradient-solving method is a popular method to find the optimal solution of convex functions by solving for values where the gradient of the function is zero.

To understand better, consider the two-degree Taylor polynomial approximation of a convex function ff around a point xx.

where ∇xf(x)\nabla_xf(x) and H(x)H(x) are the gradient and Hessian of the function ff at the point xx.

Assume that ∇xf(x)=0\nabla_x f(x) = 0, which reduces the equation above to

Because ff is a convex function, its Hessian H(x)H(x) is semipositive definite for any xx, i.e., hTH(x)h≥0h^T H(x) h \geq 0. This implies that f(x+h)≥f(x)f(x+h) \geq f(x) or the function ff increases in all directions around xx, making xx ...