Training with an EMA (Exponential Moving Average)

Explore how to apply the Exponential Moving Average technique during image classification model training in PyTorch. Understand implementing EMA with decay rates, its benefits for model performance, and managing memory with CPU options, enabling you to fine-tune models like ResNet50 effectively.

We'll cover the following...

Training with an EMA
Track an EMA on the CPU

The PyTorch Image Model framework supports an exponential moving average (EMA), which maintains moving averages of the trained variables by employing an exponential decay.

The implementation of an EMA is as follows:

Add shadow copies of trained weights during initialization.
Compute a moving average of the trained weights at each training step. It uses exponential decay for the computation.

weights = decay * weights + (1 - decay) * weights_{new}

1.Introduction

2.Basic Concepts

3.Augmentation

4.Loss

5.Training

6.Model Conversion

Project

7.Deployment

8.Appendix

9.Conclusion

Training with an EMA (Exponential Moving Average)