Token Math and Unit Economics
Understand how to calculate token-based costs for large language model inference, evaluating cost per request, session, and user. Explore architectural levers like prefix caching, model tiering, and context management to optimize expenses while balancing quality and latency in GenAI system design.
Two AI assistants can produce nearly identical answers, yet one can quietly cost 3 times as much to run. In a System Design interview, that cost difference can turn a recommended rollout into an architectural failure.
Token math connects system architecture with financial sustainability. By estimating token volume across prompts, retrieved documents, and outputs, engineers calculate the cost of a single request or an active monthly user. The objective is not billing-level perfection, but a practical BOTEC that balances answer quality, latency, and infrastructure cost.
In this lesson, we build on the performance mechanics established in the Latency, Throughput, and Cost lesson to price individual requests, project monthly user expenses, and apply optimization levers like prefix caching and dynamic routing to keep production systems affordable.
Why token economics matters
In inference workloads, compute expenditure recurs on every transaction. While model training represents an upfront capital expenditure, prompt length, context expansion, and completion verbosity scale linearly with production traffic. A minor inefficiency across millions of requests compounds into substantial infrastructure expenditure.
In System Design interviews, candidates evaluate unit economics across four granularities.
Cost per request: Captures the full financial footprint of a single invocation, including input tokens, completion tokens, retry overhead, and guardrail verification passes.
Cost per session: Aggregates multiple conversational turns, accounting for context expansion as chat histories accumulate.
Cost per active user per month: Projects aggregate monthly consumption per user, enabling direct comparison against subscription pricing and operational budgets.
Gross margin contribution: Evaluates infrastructure serving expenses against per-user revenue or platform business value.
A production request pipeline contains multiple token sources that must be accounted for systematically.
System instructions: Developer instructions that establish model persona, output constraints, and safety guidelines.
Retrieved reference context: External chunks retrieved by semantic search and injected into the prompt window.
Conversational history: Prior conversational turns carried forward to maintain multi-turn ...
Cost per request
A simple way to think about cost per request is to price every token the system sends or receives, then add side calls. In plain English, hosted cost equals uncached input tokens times the input rate, plus output tokens times the output rate, plus cached input tokens times the cached rate, plus extra request paths such as moderation, retries, or fallback models.
A canonical BOTEC formula
To maintain consistency across lessons, the financial footprint of an individual inference request uses the standardized cost breakdown: