Search⌘ K
AI Features

Hard Caps, Truncation Rules, and an Output Reserve

Explore how to implement deterministic token budgeting and truncation rules to ensure API requests fit within model limits. Understand the importance of output reserves, prioritizing content drops, and fail-closed errors to maintain reliable and predictable backend service behavior. This lesson equips you to handle input overflows before model calls, improving overall system stability.

A documented token budget does not prevent an oversized request from reaching the model. It only defines the intended limits. The current implementation does not yet validate the assembled prompt against those limits, remove or truncate content that exceeds the budget, or preserve the token budget reserved for model output. This lesson implements those budget checks and establishes where token limits are enforced before the request is sent to the model.

Consider our running example, the Support Ticket Assistant API. A request can fail in a way that looks random. One ticket passes perfectly. Then, a user adds a single extra paragraph to their ticket, or our database attaches one more retrieved snippet. This pushes the assembled prompt over the model’s input limit. The request gets rejected, or the answer cuts off mid-sentence because there was no room left to generate.

The fix belongs at the assembly boundary in our code. Enforcement has to happen before any model call, and it has to be deterministic so the same inputs always produce the same assembled artifact or the same fail-closed error.

The assembly boundary is where overflow becomes a bug

With five sections in play (system instructions, ...