Retrieval Techniques: Cosine Similarity and kNN

When a user types a question into a RAG-powered application, the system converts that question into a vector and then searches through millions of stored vectors to find the most relevant document chunks. The previous lesson covered how indexing algorithms like PQ, LSH, and HNSW organize vectors for fast access. But speed alone does not guarantee good results. The retrieval system still needs a way to decide which vectors are genuinely “close” to the query, and that decision depends entirely on the distance metric used during comparison. A poorly chosen metric can return chunks that look close in vector space but carry no semantic relevance to the question. This lesson breaks down the two dominant distance metrics, cosine similarity and Euclidean distance, and then explains how k-nearest neighbor search uses them to power retrieval in vector databases.

Cosine similarity

Cosine similarity measures the cosine of the angle between two vectors. Instead of asking “how far apart are these two points?”, it asks “are these two vectors pointing in the same direction?” The result is a value between −1 and 1, where 1 means the vectors are perfectly aligned, 0 means they are orthogonal (unrelated), and −1 means they point in opposite directions. For normalized embeddings, the range narrows to 0 through 1.

The formula is straightforward:

The numerator computes the dot productThe sum of element-wise multiplications of two vectors, producing a single scalar that reflects how much the vectors align. of vectors A and B, while the denominator normalizes by each vector’s magnitude. This normalization is what makes cosine similarity magnitude-invariant. Two vectors pointing in the same direction score 1.0 regardless of whether one is twice as long as the other.

This property matters for text embeddings. A short sentence and a long paragraph about the same topic produce vectors with different magnitudes, but their semantic meaning is captured in direction, not length. Most modern embedding models, including those from OpenAI, Cohere, and the sentence-transformers library, produce ...