Multi-Head Attention and Transformer Blocks
Explore how multi-head attention divides input into specialized heads to capture distinct linguistic relationships without interference. Understand the role of residual connections and layer normalization in stabilizing training. Delve into position-wise feed-forward networks as primary knowledge stores. This lesson explains how these components combine to form transformer blocks foundational to models like GPT and LLaMA, preparing you to leverage their embeddings in advanced retrieval and generation tasks.
We'll cover the following...
When a language model processes the sentence “The bank by the river collapsed after the flood,” it needs to resolve several relationships at once. The word “bank” relates syntactically to “collapsed” as its subject, but it also relates semantically to “river” and “flood,” which together disambiguate its meaning from a financial institution. A single attention mechanism compresses all of these competing signals into one set of Query, Key, and Value projections. The result is a forced average across different types of linguistic relationships, and the model loses the ability to cleanly separate syntax from semantics from coreference.
This is the core limitation that multi-head attention was designed to solve. By splitting the attention computation into multiple parallel “heads,” each operating on a smaller slice of the representation, the model can dedicate different heads to different kinds of relationships without interference. This lesson walks through how multi-head attention works mechanically, then shows how it combines with feed-forward layers, residual connections, and layer normalization to form the transformer block, the fundamental repeating unit behind GPT, BERT, LLaMA, and every modern LLM.
How multi-head attention works
Instead of computing a single attention function over the full model dimension
Projection into subspaces
The input embedding for each token is projected