Reconstructing Context with Sequence Models

We’ve seen how techniques like TF-IDF and GloVe help computers understand word relationships. Think of them as LEGO bricks, each representing a word. They’re useful for spotting which words often appear together. The problem is that even advanced tools like GloVe always assign the same brick to a word. So “bank” looks identical whether we mean a financial institution or the side of a river. These are static embeddings.

But language is more than a bag of bricks. Meaning comes from the sequence and context, how earlier words shape the ones that follow. To capture the whole story, we need models that don’t just store words but also remember their order and connections.

That’s where sequence models come in. They’re like LEGO sets that not only give you the pieces but also keep track of how you assemble them, preserving the structure of the story.

In this lesson, we will explore sequence models that capture order and context. We will look at convolutional neural networks (CNNs) for local patterns, recurrent neural networks (RNNs) for memory, and long short-term memory networks (LSTMs) for overcoming RNN limitations. Clear analogies and simple math will show how these models support modern generative AI.

Why sequence models?

Imagine building a sentence out of LEGO bricks. Earlier methods like TF-IDF and GloVe gave us colorful bricks that capture word relationships, but each brick was fixed. So the word “bank” looked the same whether in a financial context or by a river.

Language, however, depends on order and context. “I went to the bank to deposit money” ...