Optimization and Gradient Descent
Explore how optimization and gradient descent work to minimize loss functions in neural networks. Learn to calculate gradients, understand the role of the learning rate, and apply the update rule to adjust weights iteratively. This lesson introduces key concepts essential for training effective neural network models using frameworks like PyTorch.
In our 2D example, the loss function can be thought of as a parabolic-shaped function that reaches its minimum on a certain pair of and ...
To find these weights, the core idea is to simply follow the slope of the curve. Although we don’t know the actual shape of the loss, we can calculate the slope in a point and then move towards the downhill direction.
You can think of the loss function as a mountain. The current loss gives us information about the ...