The AI Design Problems

The previous lessons examined how AI has changed System Design, engineering roles, and interview expectations. This chapter applies these concepts to practical System Design problems. The next four chapters cover the design of ChatGPT, a data infrastructure platform, an LLM-powered customer support assistant, and an AI coding assistant. Each design follows a consistent approach to architectural components and evaluates trade-offs among inference cost, latency, and output quality.

What each problem covers

Each of these highlights a different pattern in how AI systems are built and designed, so together they give you a well-rounded view of AI-enabled design:

• Generation: Producing content from a model at scale → Design ChatGPT

• The data foundation: The data layer that ML systems are built on → Design a Data Infrastructure system

• Retrieval-grounded assistants: Pairing a model with your own data to stay accurate → LLM-Powered Customer Support Bot

• Agentic (tool-using assistants): Models that act, not just answer → AI-Powered Code Assistant

Here's a closer look at each:

Design ChatGPT

A text-to-text generation system that interprets open-ended, ambiguous input and responds in context. We’ll define the core requirements and design the large-scale inference and serving path, where latency, throughput, and cost per request dominate the trade-offs.

Design a Data Infrastructure System

The foundational platform beneath every ML system: data ingestion, processing, feature storage, and low-latency serving across the ML life cycle. We’ll look at why general-purpose warehouses fall short (training–serving skew, feature reuse, reproducibility) and design a platform for high throughput and scale.

Design an LLM-Powered Customer Support Bot

A retrieval-augmented assistant that grounds every answer in company knowledge rather than the model’s static training data, cutting hallucinations while holding context across a conversation. We’ll define requirements and design the retrieval-plus-generation path, including grounding and graceful fallback.

Design an AI-Powered Code Assistant

A context-aware, tool-using assistant embedded in the developer workflow that must understand large codebases, retrieve the right context, and often act in multiple steps. We’ll design the context/retrieval, inference, and integration path, with a sharp eye on inline latency and cost.