๐ฉ Problems with Gradient Descent and the Fix
Explore common challenges with gradient descent such as overshooting, local minima, and small gradients. Understand how to apply learning rate adjustments and momentum to improve model training and achieve better convergence in perceptron models using NumPy.
Gradient descent is the most optimal solution but it is not perfect. Take a look at its issues and see how we can fix them. This will allow us to improve our model and overcome these issues.
Problem 1: When the gradients are too large
A steeper slope tends to overshoot. Instead of converging, we begin to diverge further away from where we started. This overshooting leads to divergence.
Solution: Add learning rate
Multiply the gradient with the learning rate. When doing this, do not overshoot and the model will converge.
...