Search⌘ K
AI Features
Home
Courses
Grokking the Generative AI System Design Interview

Design a Unified Query Engine Across Data Sources

Design a unified query engine that runs a single query across emails, chats, documents, and calendars and returns ranked results or cited answers. Use connectors for ingestion, hybrid retrieval, and permission checks so only authorized, fresh information is returned within latency limits at scale.

We'll cover the following...

A product manager asks, "What changed for the Phoenix launch after Tuesday's review?" The evidence lives everywhere. The decision note sits in a document, the final approval came through email, the scheduling change lives in calendar invites, and the objection thread happened in chat. The system must answer in a few hundred milliseconds, show citations, and never reveal a private message or restricted attachment.

That request defines the unified query engine. It is an enterprise retrieval system that executes one user query across heterogeneous sources while preserving source-native permissions and freshness expectations.

We can treat this lesson as retrieval infrastructure for later agentic systems. The next lesson on deep research agents will assume this backbone already exists because multi-step agents fail quickly when retrieval, governance, and citations are weak.

The design tension is straightforward. You want broad coverage, low latency, fresh data, and exact access control, but each of those goals pushes the architecture in a different direction.

We will walk through requirements, storage and serving estimates, architecture choices, connector design, permission enforcement, query execution, answer generation, and evaluation. From there, the retrieval backbone becomes concrete enough to support more advanced reasoning systems.

System requirements

Before sizing anything, fix what the engine must do and how well it must do it. The rest of the lesson is checked against these requirements.

Functional requirements

  • Unified search: The system should search email, chat, documents, and calendar with one query.

  • Permission enforcement: The system should show users only what they can access in the source system.

  • Grounded answers: The system should return ranked results or a synthesized answer with citations.

  • Safe abstention: The system should abstain or ask a follow-up when evidence is weak.

Nonfunctional requirements

  • Latency: a few hundred milliseconds at p95, with per-source deadlines.

  • Freshness: under 1 minute for email and chat, under 5 minutes for documents and calendar. Permission revocations apply faster than new content appears.

  • Scale: 50K users, about 1.375B items, and 150 peak queries per second.

  • Availability: a slow or failed source returns partial, labeled results.

  • Security: encrypted, region-aware data and audit logs of the evidence behind each answer.

Resource estimation

Let's estimate the storage, query-serving capacity, and ingestion bandwidth this system needs, starting with storage and the index.

Storage and index estimation

Let's start with a representative enterprise of 50,000 users. We use rough per-user counts such as 5,000 emails, 20,000 chat messages, 2,000 documents, and 500 calendar events. That gives about 250 million emails, 1 billion chat messages, 100 million documents, and 25 million calendar items, or roughly 1.375 billion searchable objects.

The byte distribution is uneven. Documents dominate storage because extracted text from slides, PDFs, and attachments is much larger than a single email or message. Email and chat dominate update frequency, which drives ingestion throughput, checkpoint churn, and shard refresh behavior.

Let's assume average searchable text sizes of 2 KB for email, 0.5 KB for chat, 50 KB for documents, and 1 KB for calendar. The index also stores metadata, ACLACL (Access Control List): per-item permission data stating who can view an item, applied as a filter at query time so users only see results they're entitled to. payloads, tokenized inverted indexes, and replicas. Index size usually lands at 1 to 5 times raw searchable text, especially when storing snippets, term positions, and optional vectors.

A good estimate separates four budgets rather than one giant number.

  • Corpus storage: This stores normalized text, metadata, ACL references, and optional snippets for retrieval.

  • Hot shards: This keeps active index partitions sized for low-latency querying and frequent refreshes.

  • Replicas: This duplicates searchable state for availability and read scaling.

  • Connector state: This stores checkpoints, webhooks, change tokens, and retry queues.

Freshness targets are set ...