When the Call Fails
Explore how to manage AI model call failures in backend systems by distinguishing failure types such as timeouts, rate limits, and transient errors. Learn to implement bounded retries with exponential backoff and design fallback mechanisms, ensuring your backend handles AI call errors predictably without crashing.
Previous lessons in this chapter assumed that the model request completed successfully and focused on how to handle the response. That assumption does not hold in production. Network connections fail. Providers can become unavailable or return transient errors. We can exceed provider rate limits. This lesson focuses on handling these failures explicitly rather than catching everything with a broad except Exception. Different failure modes require different handling strategies.
Not all failures are the same failure
This lesson covers failures that happen before we ever get a response to evaluate; the call itself didn’t complete. That's different from malformed output or low confidence, which is a separate concern covered later. Three categories show up across virtually every provider:
Time-outs: The request took too long, and our own client gave up waiting. This can mean the provider is slow right now, or that the request was unusually large.
Rate limits: We've sent more requests (or more tokens) than our account is allowed to send within a given time window. The provider is telling us to slow down, not that anything is broken.
Transient errors: Temporary server-side issues on the provider's end, often signaled by a 5xx-style HTTP status, that have nothing to do with what we sent and are likely to succeed if retried a moment later.
This distinction matters because only some of these are worth retrying, and none should be retried the same way forever. A timeout or a transient error is usually ...