Visual Search: Data Strategy & Embedding Generation

With the problem formulation locked down and the latency budget decomposed from the previous lesson, every downstream decision in your visual search system now hinges on two questions: what data enters the pipeline, and how that data is transformed into embeddings. Picture a concrete interview scenario where you are asked to design the data and embedding layer for a Pinterest Lens–style system indexing billions of images. The quality of your embedding space determines retrieval recall, and the cleanliness of your data pipeline determines whether that recall is trustworthy.

This lesson covers four interconnected design decisions. You will select an embedding architecture by reasoning through trade-offs rather than picking a “best” model. You will design multi-modal signals that combine visual and text metadata for re-ranking. You will build a deduplication stage that protects result diversity at billion scale. And you will place NSFW and policy filtering upstream in the pipeline as a safety gate, not a downstream patch.

At L5+ interviews, candidates are expected to justify each pipeline stage with trade-off reasoning. Simply listing components is not enough.

Embedding architecture comparison

Choosing an embedding architecture is the single highest-leverage decision in the data layer. The architecture determines the structure of your vector space, which in turn controls what your ANN index can and cannot retrieve. Three dominant options exist for visual search, and each carries a distinct trade-off profile.

ResNet (CNN-based)

ResNet leverages convolutional layers that encode strong inductive biasesBuilt-in assumptions a model architecture makes about the structure of data, such as translation invariance in CNNs, which reduce the amount of training data needed to learn useful representations.. Translation invariance through convolutions means the network recognizes an object regardless of where it appears in the frame. Fine-tuning cost is low because pretrained ResNet checkpoints are widely available and converge quickly on domain-specific catalogs. Inference latency sits around 10 ms on a single GPU, making it attractive for latency-constrained mobile inference paths.

The limitation is generalization. ResNet embeddings trained on one visual domain (say, fashion) struggle when the query distribution shifts to home décor or food. There is also no native text understanding, so cross-modal retrieval requires a separate text encoder and a learned alignment ...