Serving Models within Latency and Cost Budgets
Learn how to serve generative models within latency and cost budgets: sizing capacity from prefill and decode token demand, placing routing, admission, and model-selection controls on the request path, tuning batching within GPU and KV-cache memory limits, reusing work with response and prefix caches, and degrading gracefully under overload.
Model serving is the part of a system that exposes a trained model through an inference API and turns incoming demand into responses. Its job is to keep the API responsive and affordable as load changes. This lesson focuses on online serving for generation workloads such as chat and text completion.
Serving decisions balance three linked groups of budgets:
Latency: Time to first token (TTFT), when streaming begins, and total response latency, when the full answer is complete.
Capacity: Request rate coming in, throughput going out (best measured in tokens per second), and GPU memory, which limits how many requests can run at once.
Cost: Cost per request or per token, which decides whether the design is affordable.
These budgets move together. Raising concurrency increases throughput but can add queueing and memory pressure, and larger batches lower cost per token but can delay the first token.
The illustration below shows how these trade-offs pull against each other.
In this lesson, we will size capacity from token demand, place control points along the request path, tune batching within memory limits, and keep the system usable during overload.
From demand to capacity
Sizing starts by translating requests into token work. A generation request has two phases, in which prefill processes the input and builds attention state, and decode generates output one token at a time. Request count alone hides this difference, so capacity estimates should track both.
For example, consider a chat feature on a 7B-parameter model with this workload:
Traffic: 20 requests per second, with bursts to 60.
Tokens per ...