The SimSiam Algorithm

Learn how to implement another widely used distillation algorithm, SimSiam.

SimSiam vs. BYOL

Like BYOL, Simple Siamese Representation Learning (SimSiam) is another distillation-based self-supervised algorithm that works without a momentum encoder. The main idea of SimSiam is to use a stop-grad operation to stop the flow of gradients through the teacher encoder. This ensures learning asymmetry as only the student branch (its encoder and predictor) gets updated during training.

The Siamese networks have been shown to develop better representations with this very simple idea. The figure below illustrates the concept of SimSiam.

Get hands-on with 1200+ tech skills courses.