The AI Design Problems
AI has transformed system design, leading to the development of various AI-driven systems. This section focuses on designing four distinct systems: ChatGPT, a data infrastructure platform, an LLM-powered customer support bot, and an AI-powered code assistant. Each design addresses unique challenges and patterns in AI system architecture, emphasizing aspects such as content generation, data management, retrieval accuracy, and tool utilization. The designs will consider core requirements, high-level architecture, and trade-offs related to latency, throughput, and cost.
The previous lessons covered how AI changed system design, the engineering job market, and the roles involved in building AI systems. This chapter applies those ideas to concrete system design problems. Over the next four chapters, we’ll design four systems: ChatGPT, a data infrastructure platform, an LLM-powered customer support bot, and an AI-powered code assistant, applying the same building-block thinking and the same cost–latency–quality triangle to systems built around a model.
What each problem covers
Each of these highlights a different pattern in how AI systems are built and designed, so together they give you a well-rounded view of AI-enabled design:
Generation: Producing content from a model at scale → Design ChatGPT
The data foundation: The data layer that ML systems are built on → Design a Data Infrastructure system
Retrieval-grounded assistants: Pairing a model with your own data to stay accurate → LLM-Powered Customer Support Bot
Agentic (tool-using assistants): Models that act, not just answer → AI-Powered Code Assistant
Here's a closer look at each:
Design ChatGPT
A text-to-text generation system that interprets open-ended, ambiguous input and responds in context. We’ll define the core requirements and design the large-scale inference and serving path, where latency, throughput, and cost per request dominate the trade-offs.
Design a Data Infrastructure system
The foundational platform beneath every ML system: data ingestion, processing, feature storage, and low-latency serving across the ML life cycle. We’ll look at why general-purpose warehouses fall short (training–serving skew, feature reuse, reproducibility) and design a platform for high throughput and scale.
Design an LLM-Powered Customer Support Bot
A retrieval-augmented assistant that grounds every answer in company knowledge rather than the model’s static training data, cutting hallucinations while holding context across a conversation. We’ll define requirements and design the retrieval-plus-generation path, including grounding and graceful fallback.
Design an AI-Powered Code Assistant
A context-aware, tool-using assistant embedded in the developer workflow that must understand large codebases, retrieve the right context, and often act in multiple steps. We’ll design the context/retrieval, inference, and integration path, with a sharp eye on inline latency and cost.
Note: We keep each of these at the System Design level: the requirements, the high-level architecture, and the trade-offs you'd defend in an interview. For how the underlying pieces actually work, how models are served and scaled, and the deeper ML and inference mechanics, see our Grokking the Generative AI System Design course.