Tokens, Context Windows, and Generation
Learn how to budget tokens in an LLM request: fitting the system prompt, history, and retrieved content within the context window while reserving room for output, separating stored history from sent context, understanding prefill and decode latency, and using truncation, summarization, prompt caching, streaming, and cancellation to control latency and cost.
A token is the unit an LLM uses to measure both input and output. Tokens don’t map one-to-one to words or characters: a short string of code or JSON can use more tokens than a longer paragraph of prose. The context window is the maximum number of tokens one request can hold, and for most models it covers the full prompt plus the model’s reply.
Token counts directly affect answer quality, latency, and cost. More relevant context can improve an answer, but every extra token adds processing work and leaves less room for output.
In this lesson, we will budget the tokens in one request, separate stored history from sent context, see where generation time goes, and apply controls that keep long conversations fast and affordable. How content is retrieved and ranked comes in the next lesson; here, any selected content is simply input competing for space.
What fits in one request
A request is a budgeting problem where every chosen input plus the reserved output must fit within one fixed limit. In a typical chat application, the token consumers are:
System instructions: Stable behavior, tone, and safety rules for the assistant.
Current user message: The task the model must answer now.
Prior conversation turns: Earlier exchanges that give the current turn its context.
Retrieved passages: Optional external snippets the application chooses to include.
Reserved output tokens: Capacity held back so the model has room to answer.
For example, with a 32k-token context window, an application might send a 1k-token system prompt, a 500-token user message, 10k tokens of history, and 18.5k tokens of retrieved content, while reserving 2k tokens for the reply. That adds up to exactly 32,000 tokens. In practice, designs keep some margin below the limit, because token estimates can be off and ...