Neural Architectures for NLP and Vision

In the previous lesson, you saw how structured-feature architectures, such as two-tower models, Wide & Deep, and DCN, handle tabular signals like user demographics and item metadata. But production ML systems rarely stop at structured data. The moment your system must understand a search query, moderate an uploaded photo, or match a product image to a text description, you cross into unstructured territory where those tabular architectures no longer apply.

This shift matters in interviews. In MAANG system design rounds, you will be asked to design NLP pipelines for query understanding or content moderation, vision pipelines for image search or visual recommendation, and increasingly, multimodal systems that reason over text and images together. Interviewers expect you to justify the architecture you choose with concrete trade-offs across latency, throughput, memory bandwidth, serving complexity, and model quality, rather than only naming model families.

This lesson covers three architecture families: transformer-based models for NLP, CNN/ResNet/ViT for vision, and CLIP/Flamingo for multimodal tasks. The emphasis throughout is on system design reasoning rather than training theory.

Transformer architectures for NLP systems

The transformer is the industry-standard backbone for NLP tasks in production ML systems. To reason about it in system design, you need to understand one key property of its core mechanism.

Self-attention and its system-level cost

The self-attention mechanismA computation where every token in a sequence attends to every other token, producing context-aware representations that capture long-range dependencies. gives transformers their power, but it comes with quadratic complexity in sequence length. For a sequence of length $n$, the attention computation scales as $O(n^2)$ ...