How Do Models Learn?

Have you ever wondered how these foundation models become so intelligent in the first place? They aren’t born understanding language or recognizing images, right? Instead, they go through an initial phase called pretraining—AI’s equivalent of foundational education. Let’s dive deep into how this foundational education happens and why it matters.

We’ll briefly introduce the landscape of pretraining methods for modern AI and see how models like GPT rely on heavy training to understand language. First, let’s step back and explore how to train a foundation model for images, text, audio, or a combination of all three. Think of it like hiring three robot chefs to work in your restaurant kitchen:

• The first robot attended culinary school, carefully following labeled recipes with step-by-step instructions.

• The second robot never had formal instruction; instead, it studied countless cookbooks to find common cooking patterns.

• The third robot had no instructions. It experimented by cooking randomly, tasting the results, and learning what worked best.

These robots perfectly represent AI’s three main pretraining paradigms: supervised learning, unsupervised learning, and self-supervised learning. Let’s understand how exactly these models learn.

What does it mean to train a model?

When we say we’re “training a model,” we mean teaching a computer to recognize patterns from data. A model starts off knowing nothing (random parameters), and as it sees more examples, it refines its internal “brain”—the weights and biases that define its understanding. ...