Search⌘ K
AI Features
Home
Courses
Applied AI Engineer: Build, Evaluate, and Operate Production LLM

How LLMs Work

Explore how large language models process input by breaking text into tokens and transforming them into embeddings. Understand how attention mechanisms influence which prior tokens shape the next words, and how autoregressive generation produces text step-by-step. This lesson helps you grasp why slight prompt adjustments change model outputs and teaches strategies to control variability for consistent results in real-world AI applications.

We'll cover the following...

Understanding why AI outputs change can feel unpredictable, but it comes down to a clear step-by-step process:

  • How text is broken down into tokens.

  • How context is processed.

  • How the model picks its next words.

In this lesson, we’ll look at what happens behind the scenes when you send a prompt. You'll see how small changes in your wording shift token boundaries and alter how the model processes your text. Rather than guessing at prompt language, you'll learn how to structure and position your instructions strategically so you can control variations and get consistent, reliable results.

Two prompts can ask for the same thing and still produce different completions.

Prompt A
Write a 2-sentence explanation of why the sky looks blue. Avoid jargon.
Representative output
The sky looks blue because air molecules scatter blue light from the sun more than other colors. That scattered blue light reaches your eyes from all directions.
Prompt B
Write a 2-sentence explanation of why the sky looks blue. Avoid jargon. Keep it simple.
Representative output
The sky looks blue because sunlight hits tiny bits in the air that spread blue light around more than other colors. Your eyes see that extra blue light coming from the whole sky.
Comparison of two similar prompts

Nothing semantic changed much, but adding a few words shifted the completion. Shipping and operating LLM applications means treating these shifts as an engineering problem, not a surprise.

This traces what data moves through one request and where variability enters.

Tokens are the input, not words

Once we send text, the model call does not receive characters or words. The request gets converted into tokens, which are short text pieces chosen from a fixed vocabulary, and the context window counts tokens, not sentences.

Token boundaries matter because they change both length and granularity. A small phrasing tweak can split a common word into multiple tokens, pushing later content closer to the context limit or changing which fragments appear together as units.

Illustrative tokenization (not exact)
"unbelievable" -> ["un", "believ", "able"]
"it's simple"  -> ["it", "'", "s", " simple"]
Tokenization

Rule

For any budget or limit, we measure tokens with our tokenizer tooling. Do not estimate by word count.

Embeddings turn tokens into vectors

After tokenization, each token id is mapped to a numeric vector used by the rest of the model. These embeddings are arranged so that tokens used in similar contexts tend to land near each other in vector space, which makes similar continuations more likely even when the surface text differs.

Two embedding notions show up in practice. A token embedding is the base vector looked up for a token id, while a contextual embedding is the vector after the model has mixed in surrounding tokens, so the same token can lead to different next-token tendencies depending on its neighbours.

That shared vector space becomes an engineering interface later. Similarity search for retrieval compares vectors, so whatever the model treats as close in this space will also tend to be retrieved as related.

Attention changes which prior tokens matter

With embeddings in place, the next step is attention, which reweights how much each earlier token influences the next token. A useful analogy is a mixing board where each slider boosts or reduces the influence of a prior token for the current position.

In each transformer block, attention produces weights over prior positions, then uses those weights to combine their vectors. Because those weights differ by layer and head, some details persist strongly while others get diluted, especially when many tokens compete in the same window.

Tokens flow through the next-token generation pipeline
Tokens flow through the next-token generation pipeline

When an output ignores an instruction, suspect instruction placement and competition in the context before blaming capability. Move the instruction later, make it shorter, or remove nearby conflicting text and check whether behavior changes.

Autoregressive generation is a loop

Once the model computes a next-token distribution, generation repeats the same cycle. The system assembles the current context tokens, runs a forward pass to produce logits, converts logits into probabilities, samples the next token, appends it to the context, and repeats until a stop condition triggers.

Non-determinism comes from sampling. Even with the same prompt and settings, selecting from a probability distribution can yield different tokens, and early differences compound because each sampled token becomes part of the next step’s context.

Some setups can be effectively deterministic if every source of randomness is fixed, such as a fixed seed and identical decoding settings, but many deployments do not guarantee that end-to-end. When debugging, treat each prompt-response pair as one sample and look for behavior classes, not exact strings.

Explaining the changed behavior

The outputs for Prompt A and Prompt B differed because the pipeline changed. The extra phrase likely altered tokenization, which changed the token sequence length and the local neighbourhoods the model processes. That change then flowed into attention, which redistributed weight across competing instructions and content.

After that, sampling selected one high-probability continuation over another. Both completions can be consistent with the request because the model is choosing one path through many plausible next tokens.

When behavior matters, use a repeatable checklist.

  • Measure tokens for every prompt and injected context before blaming the model.

  • Place critical instructions where they face less competition in the context, often near the end of the input and in a compact form.

  • Capture the full request context, including system text, retrieved passages, and decoding settings, before evaluating quality or latency.

In many applications, the primary quality degraders are context competition in attention and context length pressure from tokenization. Redesign by trimming and reordering context, then re-test with token counts and multiple samples per prompt.