Search⌘ K
AI Features

Deciding When to Retry: A Policy Built on the Error Taxonomy

Explore how to implement a retry policy for LLM APIs based on a detailed error taxonomy. Understand when retries are safe using idempotency keys, error codes, and side-effect considerations. Learn techniques like exponential backoff and circuit breakers to manage retries within budgets and maintain API reliability.

With idempotency in place, a retry can no longer duplicate work or produce a second, inconsistent result. But idempotency only answers whether a retry is safe; it says nothing about when one should actually happen. A gateway still needs a rule for which failures are worth retrying at all, and which ones will just fail the same way twice. This lesson builds that rule: a retry policy driven directly by the error taxonomy we already defined.

A Support Ticket Assistant call can fail in two ways that look similar in a dashboard but require opposite handling. A time-out means the request exceeded its time budget with no response, so the client has no result and a retry may succeed. A SCHEMA_VALIDATION_FAILED means the gateway received a response that did not match the contract, so retrying the same input against the same prompt and model is likely to produce the same invalid shape again.

Retries are a tradeoff. They add latency and load in exchange for another chance at success, and without an idempotency guarantee, a retry can duplicate the underlying work entirely. This endpoint uses a client-generated Idempotency-Key header scoped to the tenant_id, so the same key plus the same request returns the same stored response envelope rather than re-running the model.

A retry decision is a matrix

With the taxonomy in place, the next problem is separating safe retries from unsafe ones using only request metadata and the error envelope. The decision should depend on four fields a client or API gateway can check quickly: the error code, the envelope retriable flag, whether an idempotency key was present on the original request, and whether the request could trigger side effects.

The rule ...