Latency, Throughput, and Cost

In the last lesson, we explored quality as a nonfunctional requirement in GenAI systems, covering how to measure it and how it shapes architecture. However, improving quality often adds processing overhead, affecting response time and cost.

In this lesson, we will explore latency, throughput, and cost, including Time to First Token (TTFT), Time Per Output Token (TPOT), request concurrency, and GPU utilization. You will learn how to balance these constraints and define measurable performance requirements for GenAI systems.

Decomposing latency across generative workloads

In traditional distributed systems, latency measures round-trip network and compute time as a single scalar duration. In generative workloads, measuring end-to-end latency alone conceals how users perceive system responsiveness. Because generative models produce text sequentially, system designers break latency into two distinct operational phases.

Time to First Token

TTFT represents the elapsed duration from the moment a user submits a query until the inference runtime emits the initial generated token.

This phase corresponds to the prompt ingestion or prefill stage. During prefill, the Graphics Processing Unit (GPU) ingests the entire prompt sequence in parallel. The accelerator saturates its matrix calculation tensor cores, computing projected Key-Value representations for all input tokens simultaneously. TTFT is dominated by prompt length, retrieval latency, network hops, and queueing delays when the GPU cluster operates near peak capacity.

Time Per Output Token

TPOT, also known as inter-token latency, measures the average duration required to generate each subsequent token during the decoding phase.

During decode, generation is sequential and autoregressive. To produce one additional token, the GPU must stream all model weight parameters and past Key-Value cache entries from High-Bandwidth Memory (HBM) into active compute registers. Because each forward pass generates only a single token per sequence, compute cores sit idle while waiting on memory transfers, making decoding memory-bandwidth bound. A TPOT of about 75 ms or less keeps generation ahead of a fast reader at roughly 10 words per second, creating the perception of fluid interaction.

Formulating end-to-end latency

Total turnaround time represents the cumulative duration needed to complete the entire generation.

In this formulation, $N_{\text{out}}$ represents the total count of emitted completion tokens. While TTFT governs user-perceived responsiveness in interactive applications, total latency governs downstream programmatic consumers, such as structured JSON parsers or database write pipelines, which cannot execute until the final stop token is reached.

The diagram below illustrates how an inference request divides across these latency stages.

Several factors determine how long each of these latency stages takes.

Primary drivers of generative latency

Generative latency is governed by physical hardware constraints and request characteristics. Platform architects analyze five structural variables when tuning response times.

Prompt sequence length

Prompt length directly dictates prefill compute overhead. While self-attention projections process prompt tokens in parallel, computational complexity scales quadratically $O(S^2)$ ...