We'll cover the following

We learned about the binary search algorithm. Among the benefits of this algorithm is the possibility of optimizing a function even if we don’t know its formula. On the other hand, this method requires the function to be monotonic and the restriction (if any) should divide the x-axis into two separable and continuous regions. We can’t rely on binary search to solve all optimization problems because these conditions are too restrictive, and it’s not clear how to extend the method to solve multivariate problems.

In this lesson, we’re going to learn another method that requires different conditions and can be naturally extended to solve problems with many variables. This method is called the gradient descent algorithm. We’ll learn what gradient descent is, its advantages and disadvantages, when to use it, and how to implement it using Python.

Gradient descent’s applicability is widespread in many fields like robotics and video game development. But undoubtedly, the main application of gradient descent (and its variants) is in machine learning. Many algorithms, like linear regression and neural networks, can learn because of a gradient descent variant that carries on the process of automatic learning (which is just solving an optimization problem).