Tokens: The Unit That Drives Cost, Latency, and Context Limits

A triage workflow can produce different results even when the prompt remains unchanged. Send a short support email, and the workflow may return the expected JSON fields quickly. Send a long email thread with extensive quoted history, and the response may take longer, omit required JSON keys, or be truncated before completion. These failures can result from token usage, context limits, and output-length constraints. In this lesson, we will measure the token footprint of each request. We’ll use an approximate token counter to compare a short email with a long thread while keeping the triage prompt unchanged.

The key thing to notice is that the quoted thread text often dominates the input size, so the combined input plus output total grows fast even though the instruction prompt stayed the same.

Tokens are the accounting unit per run

With the size difference visible, the next step is naming the unit that grows. Tokens are the chunks of text a model processes and counts for both limits and billing, so a longer email thread usually consumes more tokens than a short email even if both have similar meaning.

For a support inbox workflow, each triage is one run with three separate quantities that we can reason about independently.

• Input tokens are everything we send, including the triage instruction prompt plus the customer email or thread.

• Output tokens are everything the model generates back, like labels, a summary, and a reply draft.

• Total tokens per run are the combined count, which is what typically matters for cost and for whether the full request fits within the model’s context limit.

Because the same workflow runs many times per day, a stable prompt can still produce unstable latency and spend if the input tokens vary widely across messages.

Rule: Estimate input size and output size separately before changing prompts or settings.

Why word counts do not control limits

When token limits matter, character and word counts are no longer reliable size limits. Two 500-word emails can produce different token counts. One may contain short words and simple punctuation, while the other includes long IDs, dense formatting, and repeated quoted headers. Exact counts also depend on the tokenizer, so a word-count target cannot guarantee that a request stays within the model’s context limit. When reducing token usage, identify the largest contributors first and trim them before optimizing smaller sections:

• If an email includes a long quoted thread, drop or summarize the quoted history first.

• If the prompt pastes a large policy or knowledge snippet, replace it with a short checklist or only the relevant section.

• If instructions are verbose, keep only the fields you actually store in the triage system.

When operating a workflow in production, your best defense against erratic performance is establishing a strict token budget. By estimating your input size and tracking observed behaviors like slow runs or truncated drafts you can easily identify when it is time to proactively trim quoted thread text or overly verbose instructions. But what actually happens when you fail to trim that excess text and the payload gets too large? Next up, we will look at exactly how massive token inputs can silently break your triage system.