Models as Components in a System
Learn how a model fits into a system as a component: the outputs it produces and how the application handles them, how training, inference, and orchestration differ, when to run inference synchronously, asynchronously, or ahead of time, where to host it, and how to fall back when a model is wrong or unavailable.
Earlier chapters covered the building blocks of distributed systems, such as load balancers, caches, and queues. AI-enabled systems reuse all of them and add a new set of components, each with its own costs and failure modes. This chapter covers these AI building blocks, one per lesson. For each one, we look at what it does, where it sits in the architecture, what it costs, and how it fails:
Models as components in a system: Where a model fits and how the system uses its output.
Ranking, prediction, and classification pipelines: Scoring live requests within tight deadlines.
Tokens, context windows, and generation: Budgeting what fits in one LLM request.
Embeddings and retrieval: Finding relevant content by meaning, not just keywords.
Retrieval-augmented generation (RAG): Grounding answers in evidence, and extending the pattern to agents.
Serving models within latency and cost budgets: Running models fast and affordably at scale.
Evaluating and protecting AI features: Proving a feature is good enough and safe enough to ship.
Each block builds on the ones before it, as the illustration below shows.
We start with the most fundamental block: the model itself. In System Design, a model is a component that turns input into an ...