Mixture of Experts

Interviewers at major tech companies often ask candidates to explain Mixture of Experts (MoE) models. This question has become common because MoE is a cutting-edge technique that trains massive AI models efficiently. For example, Waymo (an Alphabet/Google company) specifically asked a candidate how to improve the efficiency of a transformer’s feedforward network—the expected answer was to use a Mixture of Experts approach. Likewise, Meta’s latest Llama 4 models are the first in that family to employ MoE, and even OpenAI has hinted that future systems might incorporate MoE as it’s known to yield better results with fewer resources. In short, MoE has become a hot topic across Google, Meta, Amazon, OpenAI, and others, making it a likely interview subject.

What is a Mixture of Experts, and what problem does it solve?

Imagine a team of specialists, each expert in a different area, and a smart manager who directs tasks to the most qualified specialists. That’s the intuition behind Mixture of Experts (MoE). In an MoE model, instead of one monolithic neural network handling every input, you have multiple smaller networks—experts—each trained to handle certain inputs. A separate component, called the gating network (or router), examines each incoming sample and determines which expert (or experts) should handle it. The model “mixes” the expertise of many sub-models, but only a few are active for any given input.

This design allows the model to specialize and scale. Each expert can focus on a subset of the problem space (for example, one expert might become specialized in math problems while another specializes in language syntax). When a new input comes in, the gating network routes it to, say, the math expert if it’s a math question or the syntax expert if it’s a grammar question. For instance, in the DeepSeek MoE-based model, asking “What’s 2 + 2?” would activate the math expert (which responds “4”), while a request to “Write Python code for a loop” would engage the coding expert. Only the relevant parts of the network work on each query. By combining the outputs of the selected experts, the model produces a final result as if a panel of specialists solved the problem together.

Formally, MoE is a form of conditional computation: different inputs activate different parts of the network. This concept originated in the early 1990s with Jacobs and Hinton’s work on adaptive mixtures of local experts, but it has seen a resurgence in deep learning. The key idea is that “Mixture of Experts layers are simple and allow us to increase the size or capacity of a language model without a corresponding increase in compute”. We achieve this by having multiple copies of a layer (the experts, each with its own parameters) and a gating mechanism that sparsely selects a subset of those experts for each input. In other words, an MoE layer is an ensemble of many networks, but thanks to the gating, only a few experts’ outputs are combined for any given data point. This yields a huge model capacity (parameters across all experts) while keeping the computation per sample much lower than using all parts of the model at once. ...