The Case for Backpropagation
Explore the concept of backpropagation and how it solves the challenge of computing gradients in neural networks. Understand its role in training models by calculating loss derivatives efficiently, and gain practical insight by implementing this algorithm to improve network accuracy beyond simpler models.
We'll cover the following...
Recap and follow up of our neural network
Before we jump into the backpropagation, let’s have a quick review of what we have learned about neural networks so far.
In the previous chapter, we wrote a functioning neural network, or at least half of it. The network’s prediction code is done: it passes data through the model, and produces labels. However, that process also requires a set of weights, and we haven’t written the code that finds those weights yet. We’ll do it in this chapter by implementing the train() function of our neural network.
In the early years of neural networks, training was tough. AI experts even questioned whether they could be trained at all. The answer came in the early 1970s, when researchers found a way to calculate the gradient of a network with an algorithm called backpropagation or backprop.
There is a chance that we’ll never have to implement backpropagation in a real-life project. Modern ML libraries already come with their own ready-made implementations. However, it is still important that we get an intuitive sense of ...