Search⌘ K
AI Features
Home
Courses
Deep Learning with PyTorch Step-by-Step: Part I - Fundamentals

Autograd

Explore how PyTorch's autograd simplifies automatic differentiation by computing gradients of tensors involved in model training. Learn to use the backward method to calculate gradients from loss and zero_ to reset them, enabling effective parameter updates in your regression models.

We'll cover the following...

Introduction to autograd

Autograd is PyTorch’s automatic differentiation package. Thanks to it, we do not need to worry about partial derivatives, chain rules, or anything like it.

The backward method

So, how do we tell PyTorch to do its thing and compute all gradients? That is the role of the backward() method. It will compute gradients for all (requiring gradient) tensors involved in the computation of a given variable.

Do you remember the starting point for computing the gradients? It was the loss, as we computed its partial derivatives w.r.t. our parameters. Hence, we need to invoke the backward() method from the corresponding Python variable: loss.backward().

Python 3.5
# Step 1 - Computes our model's predicted output - forward pass
yhat = b + w * x_train_tensor
# Step 2 - computes the loss
# We are using ALL data points, so this is BATCH gradient 
# descent. How wrong is our model? That's the error! 
error = (yhat - y_train_tensor)
# It is a regression, so it computes mean squared error (MSE)
loss = (error ** 2).mean()
# Step 3 - computes gradients for both "b" and "w" parameters
# No more manual computation of gradients! 
# b_grad = 2 * error.mean()
# w_grad = 2 * (x_tensor * error).mean()   
loss.backward()                           # 1)
...

The following refer to the steps (old and new) of the process occurring in the code above:

  1. New “step 3 -
...