Architecture of a Retrieval-Augmented Generation Application
Understand the architecture of production-ready retrieval-augmented generation applications by exploring offline ingestion and online inference pipelines. Learn to manage data freshness, latency, operational failures, and scalability through decoupled processes, caching, and orchestration strategies. Discover how detailed observability and versioned indexes support reliability and fault tolerance in real-world LLM applications.
In theory, an LLM application can be a single API call wrapped in a prompt.
In practice, this can be classified as a demo, not a complete system. It has no data life cycle, no state, and no meaningful operational surface area. For this course, we treat retrieval-augmented generation (RAG) as the simplest real LLM application.
RAG introduces external data, evaluation requirements, and operational constraints while maintaining an architecturally minimal approach. It is the smallest design that fully exercises the LLMOps life cycle.
A production RAG system must ingest and update data, retrieve context under latency and cost constraints, dynamically assemble prompts, and generate answers reliably at scale.
Each step has distinct failure modes and performance trade-offs. In this lesson, we define a reference architecture for RAG-based LLM applications. We will decompose the workflow into explicit layers, assign clear responsibilities, and show how this structure enables scalability, reliability, and debuggability in production.
The operational problem
LLMs are often described as large pretrained statistical models whose parameters reflect the state of data available at training time.
They do not have access to private systems unless that data is explicitly provided at inference time. This limitation surfaces when LLMs are expected to reason over fresh or access-controlled data. A common first attempt is context stuffing, where large documents are pasted into each prompt.
This approach breaks down in production environments.
Longer prompts increase latency and cost, and large context windows can dilute signal and degrade answer quality. RAG addresses these constraints by retrieving only the relevant context at inference time. LLMOps starts once retrieval is treated as a production system concern.
The primary risk in production RAG systems is data divergence, defined as the gap between user expectations and what the retrieval index actually contains. Documents change, user permissions vary, and indexing pipelines are not instantaneous.
From an LLMOps perspective, this creates three immediate concerns:
Freshness ...