System Design: The RAG System
Explore the design and implementation of a retrieval-augmented generation system tailored for private, permission-aware question answering over financial documents. Understand ingestion pipelines, chunking strategies, embedding models, vector stores, hybrid retrieval techniques, and answer generation within a secure private cloud setup.
The decomposition lesson established what we are building and why. This lesson designs the actual system. Each component is configured for this specific use case, and every major decision connects back to a constraint we surfaced during decomposition. Before designing individual components, we start with an architectural overview of the full system.
Architecture overview
To meet the bank’s requirement of private, permission-aware question answering over internal documents, we implement a retrieval system. RAG is the appropriate pattern because the bank’s document corpus is too large to fit in any model’s context window, too frequently updated to justify retraining, and too sensitive to send to an external model API. The system retrieves relevant document sections on demand and uses them to generate a grounded answer.
The system has two distinct pipelines. The ingestion pipeline runs on a schedule and populates the vector index with document chunks, embeddings, and ACL metadata. The query pipeline runs in real time: it embeds the user’s question, filters the index to permitted documents, retrieves the most relevant chunks, and generates a cited answer. Every component in both pipelines runs within the bank’s private cloud.
These two pipelines share the embedding model and the vector store. The sections below design each component in pipeline order, starting with ingestion.
Document ingestion
Three connectors pull documents from SharePoint, the email archive, and the file server on a schedule. Beyond the document content, each connector also reads the document’s ACL (access control list) from the source system at ingestion time. The ACL records which users and groups are permitted to access that document. It is stored as metadata on every chunk derived from that document and travels through the rest of the pipeline alongside the content.
SharePoint connector
The connector authenticates via