Batch Normalization and Dropout

Explore how batch normalization normalizes feature channels to accelerate training and stabilize gradients, and how dropout randomly drops units to prevent overfitting. Learn their advantages, disadvantages, and implementation within convolutional neural networks to enhance model robustness.

We'll cover the following...

- Batch normalization
- Notations
- Advantages and disadvantages of using batch normalization
- Dropout

Batch normalization

If you open any introductory machine learning textbook, you will find the idea of input scaling. It is undesirable to train a model with gradient descent with non-normalized input features.

Let’s start with an intuitive example to understand why we want normalization inside any model.

Suppose you have an input feature $x1$ in the range [0,10000] and another feature $x2$ in the range [0,1]. Any linear combination would ignore $x2$ : $x1*w1 + x2*w2 \approx x1$ , since our weights are initialized in a very tiny range like [-1,1].

We encounter the same issues inside the layers of deep neural networks. In this lesson, we will propagate this idea inside the NN.

If we think out of the box, any intermediate layer is conceptually the same as the input layer; it accepts features and transforms them.

Notations

Throughout this lesson, $N$ will be the batch size, $H$ will refer to the height, $W$ to the width, and $C$ to the feature channels. The greek letter μ() refers to mean and the greek letter σ() refers to standard deviation.

The batch features are denoted by $x$ with a shape of [N, C, H, W].

x

1.Learn Deep Learning

2.Neural Networks

3.Training Neural Networks

4.Convolutional Neural Networks

5.Recurrent Neural Networks

6.Autoencoders

7.Generative Adversarial Networks

8.Attention and Transformers

9.Graph Neural Networks

10.Conclusion

Assessment

Batch Normalization and Dropout

Batch normalization

Notations