From RNNs to Transformers: A Brief History

With the feedforward neural network foundations established in the previous lesson, including weights, biases, activation functions, and depth, a natural question emerges. What happens when the task requires understanding ordered sequences of tokens rather than fixed-size inputs? Language is inherently sequential and context-dependent. The meaning of a word shifts based on the words surrounding it. A feedforward network treats each input independently and has no built-in notion of order, which makes it insufficient for tasks like machine translation, summarization, and text generation.

The field spent decades building recurrent architectures to solve this exact problem, only to discover fundamental bottlenecks that a radically different design would overcome in 2017. That design is the transformer, and it is the reason modern LLM-powered services exist. Platforms like Amazon SageMaker with Hugging Face integration can train and serve massive language models efficiently on GPU instances precisely because transformers solved the scaling limitations of their recurrent predecessors. This lesson traces that evolutionary arc, from RNNs through LSTMs to the transformer breakthrough, so you have the historical and technical context needed to study the encoder in the next lesson.

Recurrent neural networks and the idea of memory

Unlike feedforward networks, a recurrent neural network (RNN)A neural network architecture that processes sequential data by maintaining a hidden state that is updated at each time step, giving the network a form of memory over previous inputs. introduces a loop that passes a hidden state from one time step to the next. At each time step t, the RNN takes the current input token $x_t$ ...