Search⌘ K
AI Features
Home
Courses
Grokking the Generative AI System Design Interview

Design an LLM-Powered Search Engine

Design an LLM-powered search engine that rewrites queries, retrieves and ranks web evidence, and generates grounded answers with citations. Balance live search with cached content to keep responses fresh, fast, reliable, and cost-efficient at high query volumes.

We'll cover the following...

A user asks, "What changed in the new EU AI policy this morning?" and now the system has four seconds to search the web, read enough evidence, write a useful answer, and show citations that support each claim. That request sounds like chat, but the engineering problem looks more like search under tight freshness, latency, and trust constraints.

This lesson designs an LLM-powered search engine that answers open-web questions with grounded, cited responses. The request enters a pipeline that understands the query, rewrites it when needed, retrieves candidate pages, ranks trustworthy evidence, synthesizes an answer, and evaluates whether the answer remained supported by sources.

From a system design angle, the requirements are clear. The answer must be fresh for fast-moving topics, fast enough for interactive use, cheap enough to serve at scale, and trustworthy enough that users can inspect the evidence path. We keep the scope to web retrieval only, not enterprise documents or multisource federation. That narrower scope lets us build a strong single-source architecture first, which sets up the next lesson on unified retrieval across many backends.

System requirements

Let's turn these goals into concrete functional and nonfunctional requirements.

Functional requirements

  • Query understanding: The system should resolve conversational references and rewrite or decompose questions into retrieval-friendly subqueries.

  • Fresh retrieval: The system should retrieve open-web evidence, using live search for time-sensitive queries and cached or indexed content for evergreen ones.

  • Evidence ranking: The system should deduplicate candidates and rank them by relevance, recency, and source trust.

  • Grounded answers: The system should answer from the selected evidence with sentence-level citations.

  • Verification: The system should verify each claim against its cited passage and downgrade or remove unsupported claims.

Nonfunctional requirements

  • Latency: The p95 answer should complete within about 4 seconds, with streaming.

  • Freshness: Answers to time-sensitive questions should reflect current sources.

  • Cost: Fan-out, page fetches, and model calls should stay within a per-query budget at 100K queries per day.

  • Trust: Users should be able to inspect the evidence behind every factual claim.

  • Reliability: Provider or fetch failures should degrade to partial, cached, or indexed evidence rather than failing the request.

Resource estimation

Let's start with a back-of-the-envelope estimate before choosing components. For an answer engine, fan-out matters more than raw user QPS because one user query often expands into several retrieval operations.

Let's assume 100K user queries per day. If the rewriter produces three subqueries on average, the system executes 300K searches per day. If each search returns 10 candidate ...