Masking

Stepping into a GenAI interview, you can almost bet that masking in attention will pop up on the whiteboard. Masking looks deceptively simple, yet it sits at the crossroads of architecture, probability, and code hygiene. A candidate who can unpack it shows they understand Transformers beyond the buzzwords: information flow, training-time vs. inference-time constraints, and the single PyTorch line that can make or break an LLM. In short, the question separates engineers who merely use large models from those who can debug or extend them.

Masking also teases out your sense of causality. Generative models such as GPT succeed precisely because every prediction rests only on past context. If the model could peek at future tokens during training, accuracy would skyrocket—yet generation quality would crash. Explaining how a mask enforces that one-way flow of time is a litmus test for understanding language modeling.

The interviewer is quietly checking:

• Can you define the mask?

• Can you name the two classic problems it solves—padding and causality—and show how?

• Can you distinguish decoder-only (GPT) from encoder (BERT) use-cases?

Nail those points, and you’ll convince them you can reason about any GenAI stack they throw at you. In the rest of this lesson, we’ll tackle each building block in turn.

What exactly is masking?

In machine learning, a mask is a tool that selectively allows or blocks certain parts of data or computations. In attention mechanisms, a mask is usually a matrix, often filled with 0 / 1 or 0 and ∞—, that tells the model which positions to consider and which to ignore in the input. For example, when we compute the attention weights for a token, we take all pairwise dot products of its query with every key in the sequence. A mask can be applied to that dot-product matrix to zero out entries we don’t want the model to see. Essentially, masking enforces constraints on attention: it’s like placing a ...