Building Context with Neurons

We’ve seen how word embeddings like Word2Vec and GloVe represent words as dense vectors, capturing their basic meaning and relationships better than frequency-based methods. This was a huge step forward in natural language processing.

But embeddings alone are static. They tell us what a word usually means, not how its meaning shifts in context. For example, “The movie was fantastic” is clearly positive, while “The price was fantastic” could imply something very different. The word “fantastic” is the same, yet the interpretation changes with context.

To move beyond static meaning, we need models that can learn how words interact within sentences. This is where neural networks enter the picture.

Let’s test your knowledge. In the widget below, type out your answer to the following question:

Word embeddings are learned from existing data. How do you think a model handles brand-new slang or words that didn’t appear in its training data, and what challenges might that create?

In this lesson, we will trace the origins of neural networks, from simple perceptrons to the deep architectures that underlie today’s large-scale generative AI models. We’ll explore how these networks learn and process embeddings and why they are so effective at tasks that once seemed impossible for machines.

How did neural networks come to be?

The idea of neural networks began in the 1940s and 1950s, when Warren McCulloch and Walter Pitts proposed simple mathematical models of neurons. These early attempts mimicked the brain’s basic function: taking inputs, processing them, and producing outputs.

In the late 1950s, the Perceptron became one of the first practical models. It could separate data with a straight line, like dividing pepperoni from mushrooms on a pizza if they’re neatly split. However, if the toppings are mixed together, a single straight cut won’t work. Likewise, the Perceptron struggled with problems that required more complex decision boundaries, which limited its usefulness.

For decades, neural networks remained obscure due to limited theory and computing power. The breakthrough came in the 1980s with the popularization of backpropagation, a method for adjusting weights based on errors. This allowed multi-layer networks, or “deep” networks, to learn non-linear patterns far beyond the single-layer Perceptron.

As computing power and training techniques improved in the 1990s and 2000s, neural networks grew deeper and more capable, powering advances in speech recognition, image classification, and natural language processing. Today, they are the engines behind nearly every major AI achievement, including modern generative AI.

What is a neuron?

At the core of a neural network is the artificial neuron, inspired by how biological neurons work. Think of it as a small decision-making unit.

A neuron takes multiple inputs, each with a weight that sets its importance, like a volume knob that turns some inputs up and others down. It adds these weighted inputs together, then adjusts the result with a bias, similar to adding a fixed amount of seasoning to a dish, no matter the ingredients.

Mathematically, if we denote the inputs as $x_1$,$x_2$,…,$x_n$and the corresponding weights as $w_1$,$w_2$,…$w_n$​, the neuron calculates:

where $b$ is the bias. This sum, $z$, is then passed through an activation function that decides how much the signal should continue.

Imagine you’re at a candy store deciding how many candies to buy. You weigh factors like price, crowd size, and how much you love the candy (the weights). You add them up, maybe with a bonus if it’s a special day (the bias). Finally, you set a rule, for example, “if my score is above 10, I’ll buy the candies.” That decision step is the activation function.

How do neural networks learn?

A single neuron is simple, but when many are connected in layers, they form a network that can learn complex patterns. Data enters through the input layer (for example, word embeddings), moves through hidden layers where it is transformed, and finally reaches the output layer, which produces a prediction.

The hidden layers are where the learning happens. Think of them as workstations in a factory: the first layer extracts basic features, later layers combine them into more abstract ideas, and deeper layers capture concepts like grammar, sentiment, or context. By stacking layers, the network builds a hierarchy of understanding.

But how does the network know if its prediction is right or wrong? This is where the loss function comes in. The loss function measures the difference between the network’s prediction and the correct answer. A small loss means the prediction is close; a large loss means the network is far off. You can think of it like a scorecard after each attempt.

Once we have the loss, the network needs a way to improve. That’s where two key techniques come in:

1. Gradient descent: Imagine standing on a hilltop in the dark, trying to reach the lowest valley. You can feel the slope under your feet and take small steps downhill. That’s what gradient descent does: it adjusts the network’s weights step by step to reduce the loss.

2. Backpropagation: To determine which direction to move in, the network must determine which weights contributed most to the error. Backpropagation sends the error backward through the layers, calculating how much each weight needs to change.

When training neural networks, two common challenges are overfitting and underfitting.

• Overfitting happens when the model memorizes noise and details in the training data instead of learning general patterns. It performs well on the training set but poorly on new, unseen data. Techniques like regularization (adding a penalty for overly complex models) and dropout (randomly turning off neurons during training) help reduce overfitting by encouraging the network to learn more robust features.

• Underfitting is the opposite problem. Here, the model is too simple and fails to capture important patterns in the data. This leads to poor performance on both training and test sets. Increasing model complexity or improving training can help address underfitting.

How are word embeddings related to neural networks?

Word embeddings turn words into dense vectors that capture their basic meaning, but they are static. They don’t show how words interact in a specific context. Neural networks solve this by taking embeddings as input and refining them through hidden layers to capture relationships and nuances.

For example, in “The movie was fantastic!”, embeddings for “movie” and “fantastic” represent general meaning and positivity. On their own, they don’t show the connection. As they pass through hidden layers, the network first detects structure, then sentiment, and finally learns that “fantastic” modifies “movie” to imply a positive review.

This transformation from static embeddings to context-aware representations is what makes modern NLP powerful. Neural networks, with millions or even billions of adjustable parameters, scale this process up, enabling applications such as sentiment analysis, translation, and even creative tasks like text, image, and music generation.

Why neural networks aren’t enough?

Neural networks are the engine of modern generative AI. They take raw inputs like word embeddings and, through many layers of processing, transform them into context-rich representations that capture meaning, sentiment, and nuance. This is what allows them to translate sentences, recognize images, or generate fluent text.

They achieve this by training on vast amounts of data and adjusting their parameters with backpropagation and gradient descent. Over time, they learn patterns and structures that make generative AI possible. But despite their power, neural networks have important limitations. These challenges set the stage for the next breakthroughs in NLP.

Feed-forward networks and their limits

Feed-forward networks are the simplest neural networks. Data flows one way from input to hidden layers to output, with no loops or feedback. They work well for static tasks, such as image classification, but process each input independently, disregarding word order. For example, “The cat chased the mouse” and “The mouse chased the cat” would look the same to such a model.

To handle sequences like language, specialized models such as RNNs, LSTMs, and Transformers were developed. These architectures add memory and attention, allowing networks to capture context over time.

Even so, neural networks can still struggle with tasks that demand explicit reasoning. This has led to hybrid models that blend neural learning with symbolic logic, combining pattern recognition with precise rule-based reasoning.