Search⌘ K
AI Features
Home
Courses
AI for Backend Engineers

Guardrails on Spend and Volume

Understand how to implement safeguards like rate limiting, daily usage budgets, and kill switches to prevent excessive AI model calls. This lesson helps you manage backend AI feature volume proactively to ensure reliability and control costs.

We'll cover the following...

Every safeguard this course has built so far assumes the model is being called a reasonable number of times by reasonable people. That assumption can fail even when nothing is actually broken: a single user retrying the same request in a loop, a traffic spike nobody planned for, or a bug in an entirely unrelated part of the system that happens to call this endpoint far more often than intended. None of the earlier safeguards catch this, since validation, confidence checks, and injection resistance all assume the call is happening at all. This lesson is about deciding, ahead of time, how many calls are allowed to happen in the first place.

Most of this lesson needs almost no setup, since a rate limiter, a budget tracker, and a kill switch are all plain Python with no dependency on a model, a prompt, or a schema. The pipeline from earlier chapters only shows up once, at the end, when all three guardrails come together around a real call.

A per-user request cap

The most direct limit is a simple count: how many requests a single user is allowed to make within a given window of time.

Python
import time
class RateLimiter:
    def __init__(self, max_requests: int, window_seconds: int):
        self.max_requests = max_requests
        self.window_seconds = window_seconds
        self.request_times: dict[str, list[float]] = {}
    def allow_request(self, user_id: str) -> bool:
        now = time.monotonic()
        cutoff = now - self.window_seconds
        history = self.request_times.setdefault(user_id, [])
        history[:] = [timestamp for timestamp in history if timestamp > cutoff]
        if len(history) >= self.max_requests:
            return False
        history.append(now)
        return True
limiter = RateLimiter(max_requests=3, window_seconds=60)
for attempt in range(1, 6):
    allowed = limiter.allow_request(user_id="user-42")
    print(f"Attempt {attempt}: {'allowed' if allowed else 'blocked'}")

  • Lines 5–8: max_requests and window_seconds are configured once, when the limiter is created. request_times maps each user ...