Embeddings: The Engine of Semantic Search
Take a practical deep-dive into embeddings, the numerical vectors that represent the meaning of text.
We'll cover the following...
In our last lesson, we explored the art of communication with LLMs through prompt engineering. We can now control a model’s persona, rules, and output format. However, to build more advanced applications, we need to go beyond simply interacting with the model. We need to build systems around it. The first step in that journey is to understand the most important tool for representing the meaning of text: embeddings.
This lesson will be our practical deep dive. We will move from the theory of embeddings we touched on earlier to a hands-on exploration of how they work and why they are the engine behind modern AI applications like semantic search.
A quick recap
Before we dive into the code, let’s build a mental model of the transformation from text to a meaning vector. We’ll do this by breaking the process down into three key questions.
Question 1: What is a token?
A single character in a piece of text.
The smallest unit of meaning the model works with, which can be a word, part of a word, or punctuation.
A special keyword that tells the model what to do.
Choose the correct option below.
The power of comparing meaning
Why do we need to compare meanings? This is because embeddings translate the fuzzy, ambiguous world of human language into the precise world of mathematics. Once we have these meaning vectors, we can perform a simple calculation called cosine similarity to get a numerical score of how related two pieces of text are.
Imagine two arrows starting from the same ...