Search⌘ K
AI Features
Home
Courses
Grokking the Generative AI System Design Interview

Design for Cost at 1M Queries a Day

Design a GenAI serving system that answers 1 million queries a day for under $0.02 each without sacrificing latency or quality. It routes traffic through exact and semantic caches and a cheap-versus-frontier model cascade, caps retrieval and output tokens, and keeps spend in check with per-route monitoring and degradation policies.

We'll cover the following...

One million GenAI queries per day turn small inefficiencies into budget incidents. Adding 200 extra prompt tokens, letting output length drift, or routing every question to the strongest model moves the monthly bill before anyone notices any quality gain. In this setting, cost is not a finance afterthought. It is a design constraint that sits next to latency and answer quality.

This lesson treats the interview problem as a production design exercise for a conversational RAG assistant under a hard unit-cost ceiling. We assume three concrete targets: cost at or below $0.02 per query, P95 time to first tokenHow long until the first token of the answer appears. (TTFT) at or below 500 ms, with streaming, and retrieval and answer quality that stay within an acceptable range, such as recall@5The share of questions whose needed evidence is in the top 5 chunks. near 0.90 and no visible answer collapse on hard questions.

Interviewers ask this because strong candidates must reason quantitatively across pricing, tokens, traffic mix, caching, and serving architecture. Drawing boxes is not enough. You need to show how the request path changes cost.

Note: The strongest answer usually starts with a naive estimate, then reduces blended costAverage cost per query across all paths, weighted by traffic share. through routing and reuse instead of hand-waving about "optimizing prompts."

We will build the expensive baseline first, identify the dominant multipliers, then redesign the path with caching, model tiering, retrieval budgets, output limits, and batch processing. That structure also sets up the next lesson on delivering this clearly in a 45-minute interview.

System requirements

Let's pin down what the assistant must do and the budgets it must meet before estimating cost.

Functional requirements

  • Conversational answers: The system should answer questions with retrieval-augmented generation and stream responses.

  • Reuse: The system should serve repeated and near-duplicate questions from exact and semantic caches.

  • Model routing: The system should route each cache miss to a cheap or frontier model based on predicted difficulty, with escalation on low confidence.

  • Request budgets: The system should cap retrieval depth, output length, agent steps, and per-user usage.

  • Offline processing: The system should run re-embedding, summarization, and evaluation as batch jobs outside the serving path.

Nonfunctional requirements

  • Cost: The system should stay at or below $0.02 per query, about $20,000 per day at 1 million queries.

  • Latency: P95 time to first token should stay at or below 500 ms with streaming.

  • Quality: Retrieval recall@5 should stay near 0.90, with no visible answer collapse on hard questions.

  • Availability: Cache or vector-store failures should degrade gracefully instead of failing requests.

  • ...