Reconstructing Context with Sequence Models

We’ve seen how techniques like TF-IDF and GloVe help computers understand the relationships between words. Think of them as LEGO bricks, each representing a word. They’re useful for spotting which words often appear together. The problem is that even advanced tools like GloVe always assign the same brick to a word. So “bank” looks identical whether we mean a financial institution or the side of a river. These are static embeddings.

But language is more than a bag of bricks. Meaning comes from the sequence and context, how earlier words shape the ones that follow. To capture the entire story, we need models that not only store words but also remember their order and connections.

That’s where sequence models come in. They’re like LEGO sets that not only give you the pieces but also keep track of how you assemble them, preserving the structure of the story.

In this lesson, we will explore sequence models that capture order and context. We will understand convolutional neural networks (CNNs) for local patterns, recurrent neural networks (RNNs) for memory, and long short-term memory networks (LSTMs) for overcoming RNN limitations. Clear analogies and simple math will show how these models support modern generative AI.

Why sequence models?

Imagine building a sentence out of LEGO bricks. Earlier methods, such as TF-IDF and GloVe, provided us with colorful bricks that capture word relationships, but each brick was fixed. So the word “bank” looked the same whether in a financial context or by a river.

Language, however, depends on order and context. “I went to the bank to deposit money” ...