Chunking and Embedding with Stable IDs
Explore how to split documents into stable, deterministic chunks and generate consistent IDs for reliable retrieval. Understand embedding principles and how to manage embedding model changes to maintain valid citations in retrieval-augmented generation systems.
The previous lesson established that citations must point to stable identifiers. This lesson builds those identifiers.
A chunk is a passage small enough to be useful as context. Splitting documents into chunks is necessary because a whole policy document is usually too large to fit in a prompt alongside everything else, and because retrieval works better when it can return the one relevant paragraph instead of forty pages.
The requirement is that chunking behaves like a pure function. Given the same document and the same configuration, it must emit identical chunk text and identical chunk IDs every time. If the same content produces different IDs across runs, then a citation recorded yesterday points at nothing today.