Transformers Building Blocks
Understand transformer building blocks including short residual skip connections and layer normalization. Learn how these mechanisms enable top-down processing and stable training in NLP models, improving your grasp of transformer architectures.
We'll cover the following...
Short residual skip connections
In language, there is a significant notion of a wider understanding of the world and our ability to combine ideas. Humans extensively utilize these top-down influences (our expectations) to combine words in different contexts.
In a very rough manner, skip connections give a transformer a tiny ability to allow the representations of different levels of processing to interact.
With the forming of multiple paths, we can “pass” our higher-level understanding of the last layers to the previous layers. This allows us to re-modulate how we understand the input. Again, this is the same idea as human top-down understanding, which is nothing more than expectations.
Layer normalization
Let’s open the Layer Norm black box.
In Layer Normalization (LN), the mean and variance are computed across channels and spatial dims.
In language, each word is a vector. Since we are dealing with vectors, we only have one spatial dimension.
...