Search⌘ K
AI Features
Home
Courses
Generative AI Essentials

Vectorizing Language

Explore the fundamentals of language vectorization and word embeddings to understand how modern NLP captures word meanings and relationships. Learn about the evolution from traditional frequency-based methods to dense embeddings, including Word2Vec and GloVe, and their role in advancing generative AI. This lesson equips you with knowledge of sparse versus dense embeddings and the impact on language understanding in AI systems.

We'll cover the following...

Traditional NLP methods, such as rule-based systems, Bag of Words, TF-IDF, and n-grams, represent text by counting word occurrences. This works for basic tasks such as classification or prediction, but it treats words as isolated tokens with no sense of meaning or connection.

For example, “cat” and “feline” are seen as completely unrelated, even though they describe the same animal. Likewise, words like “great,” “terrific,” and “awesome” are not recognized as expressing similar sentiments. Frequency-based methods can only note co-occurrences, not capture true relationships.

Traditional methods treat words as independent units, such as assigning each guest at an event a unique badge number. The system can track who is present, but knows nothing about relationships between guests. In the same way, frequency-based models count words but miss their connections. They also create huge, sparse representations that require lots of data yet struggle to generalize.

What are word embeddings?

Counting words was not enough, so researchers developed word embeddings to capture meaning. Instead of treating each word as an isolated unit, embeddings represent words as vectors in a continuous space where similar words appear closer together. For example, “king” and “queen” will be close in the vector space, and the relationship between “man” and “woman” parallels that of “king” and “queen.”

These vectors are dense, meaning they use far fewer dimensions than Bag of Words or TF-IDF, yet carry much richer information. Each dimension reflects some linguistic feature, such as topic, sentiment, or grammatical role. This compact, meaningful representation allows models to generalize better, learn patterns faster, and apply knowledge across different contexts.

Word embeddings have become the bridge between simple frequency-based methods and modern deep learning, powering breakthroughs such as Word2Vec and GloVe.

How have word embeddings changed NLP?

Word embeddings transformed NLP by moving beyond word counts to dense vectors that capture meaning and relationships. They allowed models to understand similarity, context, and nuance in ways earlier methods could not.

In the next sections, we will examine specific approaches, such as Word2Vec, CBOW, Skip-gram, and GloVe, to see how these embeddings are learned.

What is Word2Vec?

Word2Vec, introduced by Tomas Mikolov’s team at Google in 2013, was a major leap in how machines understand language. Instead of simply counting how often words appear, it learns word meanings from the contexts in which they occur. This creates dense vector representations where similar words are positioned close together.

Word2Vec showed that embeddings could capture not only similarity but also relationships between words. A famous example is king – man + woman ≈ queen, where arithmetic on vectors reflects real semantic patterns.

In the next sections, we will explore the two main approaches Word2Vec uses to learn these embeddings: CBOW and Skip-Gram.

Continuous Bag of Words (CBOW)

It predicts the missing word in a sequence based on its surrounding context. For example, in the sentence “The cat sat on the ___,” the model looks at [“The,” “cat,” “sat,” “on”] to guess the word “mat.” By repeating this process across millions of sentences, CBOW learns embeddings that capture the meaning of words from their context.

Continuous bag of words
Continuous bag of words

Skip-gram

Unlike CBOW, which predicts the center word from its surrounding context, skip-gram does the reverse—it predicts the surrounding context words based on the center word. For example, in the sentence “The cat sat on the mat,” if “sat” is chosen as the center word, the model will try to predict its context words ["The," "cat," "on," "the"]. Each word, including the center word “sat,” is represented as an embedding—a vector in a high-dimensional space.

The embedding of the center word is passed through a neural network, which outputs probabilities for the most likely context words in the vocabulary. By learning to predict these context words, Skip-gram builds embeddings where words with similar contexts (like “mat” and “rug”) end up closer in the vector space.

Skip-gram
Skip-gram

Optional: Click to view implementations.

Both CBOW and skip-gram adjust embeddings so that words appearing in similar contexts share similar vectors. Similarity is often measured with cosine similarity, which compares the angle between two vectors. Words like “cat” and “feline” ultimately point in nearly the same direction, making them highly similar.

Limitations of Word2Vec

While Word2Vec was a breakthrough, it also came with challenges:

  • Large vocabularies are inefficient: Predicting probabilities for every word in the vocabulary is expensive. Word2Vec solves this with negative sampling, where the model only checks a few “incorrect” word pairs at a time. This shortcut speeds up training, but it still highlights the difficulty of scaling to very large vocabularies.

  • Embeddings are static: A word always has the same vector, no matter the context. For example, “bank” represents both “river bank” and “money bank” in the same way, even though the meanings are different. This limitation makes it hard for Word2Vec to fully capture language nuance.

Because of these issues, Word2Vec works best for capturing local patterns, like neighbors in a small neighborhood, but struggles with broader, context-dependent meaning. This gap led researchers to explore new methods such as GloVe, which used global statistics of word co-occurrence to improve embeddings.

What is GloVe?

While Word2Vec learns from local context, GloVe (Global Vectors) looks at the bigger picture. It builds embeddings from a global co-occurrence matrix that tracks how often words appear together across an entire corpus.

Think of it like a huge spreadsheet where each cell records how often two words occur side by side. By compressing this matrix into fewer dimensions, GloVe preserves the most important patterns. This allows it to uncover broader relationships, such as linking “coffee” to “tea”, that might be missed by focusing only on sentence-level context.

The result is word vectors that capture both global and local context, producing embeddings that reflect semantic similarity and analogies such as king – man + woman ≈ queen.

GloVe example
GloVe example

Strengths and limitations of GloVe

GloVe improved on Word2Vec by taking a global view of language, but it still came with trade-offs.

  • Global perspective: GloVe captures word relationships across the entire corpus. For example, it links “coffee” and “tea” even if they rarely appear in the same sentence.

  • Limitation-static vectors: A word like “tea” always has the same vector, whether it appears in a café menu or a metaphor. This gap led to context-sensitive models.

  • Bias in embeddings: Word vectors often reflect stereotypes from training data. A well-known case was doctor – man + woman ≈ nurse, which highlighted the need for debiasing methods.

Below is a simple, self-contained Python demonstration of a GloVe-like model using only NumPy.

Keep in mind that this implementation is highly simplified compared to production-grade GloVe. It’s provided for illustration only—you’re not expected to master every detail. If you’re curious, feel free to explore further, but it’s perfectly fine to focus on the high-level ideas for now!

Python 3.10.4
import numpy as np
def weighting_func(x, x_max=100, alpha=0.75):
    """
    Compute the weighting function for a co-occurrence count.
    If x < x_max, return (x / x_max)^alpha; otherwise return 1.
    """
    return (x / x_max) ** alpha if x < x_max else 1
# ---------------------------------------------------
# Step 1: Define the Corpus
# ---------------------------------------------------
corpus = [
    "I like deep learning",
    "I like NLP",
    "I enjoy flying"
]
print("Original Corpus:")
for sentence in corpus:
    print(" -", sentence)
# ---------------------------------------------------
# Step 2: Preprocess the Corpus
# Lowercase and tokenize each sentence.
# ---------------------------------------------------
sentences = [sentence.lower().split() for sentence in corpus]
print("\nTokenized Sentences:")
for sentence in sentences:
    print(" -", sentence)
# ---------------------------------------------------
# Step 3: Build the Vocabulary and Mappings
# ---------------------------------------------------
vocab = set()
for sentence in sentences:
    for word in sentence:
        vocab.add(word)
vocab = list(vocab)
word2idx = {word: idx for idx, word in enumerate(vocab)}
idx2word = {idx: word for idx, word in enumerate(vocab)}
print("\nVocabulary (word to index mapping):")
for word, idx in word2idx.items():
    print(f" {word}: {idx}")
# ---------------------------------------------------
# Step 4: Build the Co-occurrence Matrix
# We'll use a window size of 1 for simplicity.
# ---------------------------------------------------
vocab_size = len(vocab)
X = np.zeros((vocab_size, vocab_size))
window_size = 1
for sentence in sentences:
    sentence_length = len(sentence)
    for i, word in enumerate(sentence):
        word_idx = word2idx[word]
        # Define the window boundaries
        start = max(0, i - window_size)
        end = min(sentence_length, i + window_size + 1)
        for j in range(start, end):
            if i == j:
                continue  # Skip the word itself
            context_word = sentence[j]
            context_idx = word2idx[context_word]
            X[word_idx, context_idx] += 1
print("\nCo-occurrence Matrix (X):")
print(X)
# ---------------------------------------------------
# Step 5: Initialize GloVe Parameters
# ---------------------------------------------------
embedding_dim = 10      # Dimension of the embeddings
learning_rate = 0.05
epochs = 100
# Initialize word and context embeddings randomly
W = np.random.rand(vocab_size, embedding_dim)
W_context = np.random.rand(vocab_size, embedding_dim)
# Initialize bias terms for words and context words
b = np.random.rand(vocab_size)
b_context = np.random.rand(vocab_size)
# ---------------------------------------------------
# Step 6: Train the GloVe Model
# ---------------------------------------------------
# We minimize the cost: f(X_ij) * (w_i^T w_j~ + b_i + b_j~ - log(X_ij))^2
for epoch in range(epochs):
    total_cost = 0
    # Iterate over all nonzero co-occurrence entries
    for i in range(vocab_size):
        for j in range(vocab_size):
            if X[i, j] > 0:
                # Compute weighting for this co-occurrence
                weight = weighting_func(X[i, j])
                # Calculate the difference between prediction and log count
                diff = np.dot(W[i], W_context[j]) + b[i] + b_context[j] - np.log(X[i, j])
                cost = weight * (diff ** 2)
                total_cost += cost
                # Compute gradient (the factor 2 comes from the derivative of the square)
                grad = 2 * weight * diff
                # Update the parameters using gradient descent
                W[i] -= learning_rate * grad * W_context[j]
                W_context[j] -= learning_rate * grad * W[i]
                b[i] -= learning_rate * grad
                b_context[j] -= learning_rate * grad
    if (epoch + 1) % 10 == 0:
        print(f"Epoch {epoch + 1}/{epochs}, Total Cost: {total_cost:.4f}")
# Combine word and context embeddings as the final representation
final_embeddings = W + W_context
print("\nLearned GloVe Embeddings:")
for word, idx in word2idx.items():
    print(f" {word}: {final_embeddings[idx]}")

By combining the local insights of methods like Word2Vec with the global statistics of GloVe, NLP took a major leap forward. With GloVe, word embeddings became not just a way to process text but a powerful tool for capturing the hidden structure of language itself.

What are sparse and dense embeddings?

Before we explore embeddings further, it helps to compare how text was represented in older methods to modern approaches.

  • Sparse embeddings: Traditional methods like Bag of Words and TF-IDF create sparse vectors, where each dimension represents a word in the vocabulary. Most entries are zero since a sentence only contains a few of those words. For example, with the vocabulary [“love,” “hate,” “cats,” “dogs,” “ai”], the sentence “I love cats” becomes [1, 0, 1, 0, 0]. These vectors grow very large as the vocabulary expands, making them inefficient.

Sparse vectors
Sparse vectors

  • Dense embeddings: Word embeddings, such as Word2Vec and GloVe, produce dense vectors, typically with 100–300 dimensions, where every entry contains meaningful information. For example, “cat” and “feline” appear close together in this space because their vectors are similar.

Dense vectors
Dense vectors

Dense embeddings have two key advantages over sparse ones:

  • Efficiency: They are compact, fixed-size representations, so they are easier to compute and store than giant sparse vectors.

  • Semantic relationships: Capture meaning and structure, allowing models to understand patterns like: king : queen :: man : woman.

Why it matters: Moving from sparse to dense embeddings not only improves efficiency but also unlocks richer language understanding, paving the way for modern neural NLP models and today’s generative AI.

How have these techniques accelerated GenAI?

The journey of generative AI began with simple methods, such as counting words and co-occurrences, which provided machines with their first means of handling language. Word embeddings then added a richer layer, allowing computers to see not just words, but also relationships and meaning.

Generative AI builds directly on this foundation. Neural networks refine these embeddings by learning context across massive datasets, enabling today’s large language models to generate fluent, human-like text.

By understanding how these early techniques evolved, you can see how each step brought us closer to the systems we use today.