Embedding Service and Vector Store Design

A RAG prototype works well with thousands of chunks, but scaling to millions creates immediate bottlenecks in latency, memory cost, and index updates. In production, these challenges center on the vector tier, the subsystem connecting chunk ingestion to semantic retrieval.

This tier generates embeddings, stores vectors with metadata, executes nearest neighbor search, and applies tenant and permission filters under millisecond latency budgets. In this lesson, you will learn how to design an embedding service, choose the right approximate nearest neighbor index, scale vector storage across machines, and manage index freshness over time.

Embedding service design

The embedding service is not a helper function. In production, it is a distributed subsystem with separate traffic paths, separate SLOs, and separate failure modes.

One path handles bulk ingestion for large corpora. The other handles low-latency embedding for user queries and fast-changing content. Keeping both on one fleet usually causes trouble, because the batch side wants large accelerator batches while the online side wants quick turnarounds and predictable tails.

Batch and real-time paths

The system usually splits the request flow into the following paths.

• Batch ingestion: Workers pull raw chunks from durable storage, group them into larger batches, run them on GPUs, and write versioned vectors into staging storage before index builders consume them.

• Real-time embedding: Online workers accept single items or microbatches, respect stricter latency budgets, and isolate failures so a backlog in ingestion does not stall user-facing requests.

• Shared model registry: Both paths read model artifacts and model version metadata from the same registry so every emitted vector is tagged consistently.

That separation leads to cleaner scaling decisions in the serving tier.

Throughput tuning

Batching improves accelerator utilization ...