Search⌘ K
AI Features
Home
Courses
AI for Backend Engineers

Guardrails on Spend and Volume

Explore how to implement effective guardrails on AI backend systems by setting per-user request limits, global daily budgets, and immediate disable mechanisms. Understand how these safeguards prevent excessive model calls, manage system costs, and maintain reliability during traffic spikes or failures, without relying on model validation or prompting strategies.

We'll cover the following...

Every safeguard this course has built so far assumes the model is being called a reasonable number of times by reasonable people. That assumption can fail even when nothing is actually broken: a single user retrying the same request in a loop, a traffic spike nobody planned for, or a bug in an entirely unrelated part of the system that happens to call this endpoint far more often than intended. None of the earlier safeguards catch this, since validation, confidence checks, and injection resistance all assume the call is happening at all. This lesson is about deciding, ahead of time, how many calls are allowed to happen in the first place.

Most of this lesson needs almost no setup, since a rate limiter, a budget tracker, and a kill switch are all plain Python with no dependency on a model, a prompt, or a schema. The pipeline from earlier chapters only shows up once, at the end, when all three guardrails come together around a real call.

A per-user request cap

The most direct limit is a simple count: how many requests a single user is allowed to make within a given window of time.

Python
import time
class RateLimiter:
    def __init__(self, max_requests: int, window_seconds: int):
        self.max_requests = max_requests
        self.window_seconds = window_seconds
        self.request_times: dict[str, list[float]] = {}
    def allow_request(self, user_id: str) -> bool:
        now = time.monotonic()
        cutoff = now - self.window_seconds
        history = self.request_times.setdefault(user_id, [])
        history[:] = [timestamp for timestamp in history if timestamp > cutoff]
        if len(history) >= self.max_requests:
            return False
        history.append(now)
        return True
limiter = RateLimiter(max_requests=3, window_seconds=60)
for attempt in range(1, 6):
    allowed = limiter.allow_request(user_id="user-42")
    print(f"Attempt {attempt}: {'allowed' if allowed else 'blocked'}")

  • Lines 5–8: max_requests and window_seconds are configured once, when the limiter is created. request_times maps ...