Latency Budgets and Degraded Modes
Explore how to define and enforce latency budgets in AI architectures, balancing cost and performance. Understand how to design explicit degraded modes with governance policies that maintain safety and reliability when latency constraints are exceeded.
The unit economics model prices out a set of execution paths. Those same paths determine how long a request takes. Unlike cost, latency can’t get fixed after the fact with a bigger budget. Context size, routing, retries, and agent step limits define latency and cost together, so treat those limits as an architecture contract. Start a defensible contract from an end-to-end target from stakeholders, then convert it into a bounded segment that policy can enforce.
End-to-end latency as a budgeted timeline
Apply a latency target to the whole request path, not just one model generation. During peak hours, segment the timeline into request admission and policy checks, retrieval and optional reranking, model generation, tool calls into systems of record, and any optional agent step that repeats that loop.
Give every segment a control that expands or caps it, and make that control architectural, not incidental. Routing determines which model family is eligible and whether a fast path exists. A containment envelope sets a hard bound, maximum context, ...