Setting Up a Knowledge Base with Llama Stack
Learn how to set up a vector database as a knowledge base in Llama Stack and insert documents into it using the Vector IO API.
Large language models are inherently stateless. Once a prompt is completed, they forget the interaction. Even with system messages and history, their ability to recall facts or access up-to-date information is limited. When building real-world applications, we often need to give the model access to external knowledge, like internal docs, product FAQs, legal references, or even PDFs.
This is where retrieval-augmented generation (RAG) comes in.
RAG allows us to ingest documents into a semantic memory system, typically a vector database, and then retrieve the most relevant chunks at query time. These chunks are fed to the model as context, grounding its responses in real data.
In this lesson, we’ll set up that knowledge base: register a vector database, ingest documents into it, and run a test retrieval to ensure it’s working. But by the end, we’ll have a searchable, structured memory bank ready to power document-aware assistants.
Why use a vector database?
When working with documents, we can’t just pass the entire corpus into the model; most models have strict token and context window limits. Instead, we:
Break the document into chunks.
Convert each chunk into a vector embedding.
Store those vectors in a vector database.
At query time, embed the user’s question.
Use semantic search to find the most similar chunks.
Feed those chunks into the model as ...