The Building-Block Approach to GenAI Design
Explore the foundational building-block approach to GenAI system design, focusing on decomposing complex AI architectures into reusable modules like retrieval, context management, model serving, evaluation, and orchestration. Understand how to manage trade-offs in latency, cost, data privacy, and system scalability while using this framework to design effective and compliant AI systems.
A common interview problem statement sounds simple until the constraints arrive. Design an enterprise assistant that answers employee questions over private documents, cites sources, stays under a tight p95 latency target, and never leaks data across teams. In the first two minutes, strong candidates stop thinking in terms of one giant AI system and start decomposing the architecture into reusable parts.
That is the building-block approach. We treat a GenAI system as a small set of modules that recur across products, then place them in the request path and reason about their trade-offs. This course uses that approach as its backbone because the same blocks reappear in internal knowledge assistants, support copilots, research agents, and multimodal workflows.
At a high level, GenAI system design can be organized into four building-block families, supported by traditional infrastructure components.
Retrieval and context: Find relevant information and prepare it for the model.
Model serving: Run the model, manage inference, and return generated output.
Evaluation and guardrails: Check output quality, safety, and policy compliance.
Agentic orchestration: Plan multistep tasks and coordinate tool calls.
Traditional infrastructure, such as caches, databases, queues, and load balancers, supports these families across the system.
The full block map
The full architecture map shows the four GenAI building-block families alongside the traditional infrastructure that supports them. The four families handle GenAI-specific responsibilities, while infrastructure components provide shared capabilities such as caching, security, observability, and traffic management.
A useful high-level map includes the following components:
Retrieval and context layer: This layer locates candidate knowledge, filters it, reranks it, and assembles prompt-ready context.
Model serving layer: This layer routes requests through a model gateway, selects models, manages inference, handles retries and failover, and tracks token and latency budgets.
Evaluation and guardrails plane: This layer checks output quality, enforces safety policies, and supports offline evaluation and audits.
Agentic orchestration and tools layer: This layer plans multistep tasks, invokes external systems, and maintains intermediate state when needed.
Cross-cutting infrastructure: This includes observability, security, cost controls, caching, and latency management across all layers.
A