Mixture of Experts
A neural network architecture where each input is routed to a small subset of specialized sub-networks (experts) by a learned gating function, enabling very large total parameter counts while keeping per-token computation constant.
In a Mixture of Experts (MoE) transformer, each feed-forward layer is replaced with N expert networks and a gating function that selects the top-k experts for each token. Only k/N of the experts' parameters are active for any given token, making inference cost similar to a dense model of size k/N × total while benefiting from the full expressivity of total parameters during training.
Mixtral 8×7B (Mistral, 2023) popularized open MoE models: it has ~46B total parameters but only ~13B active per token, matching Llama 2 70B quality at Llama 2 13B inference cost. GPT-4 is widely believed to be MoE-based. Gemini 2.5 and other frontier models use MoE to achieve their scale-quality-cost tradeoff.
Load balancing is the main engineering challenge: if the gating always selects the same top-k experts, most of the model is underutilized. Auxiliary loss terms (load balancing loss) are added during training to encourage uniform expert utilization. Expert routing is non-differentiable through hard selection; the straight-through estimator or soft routing (softmax over experts) is used during training.
## Related Terms
- [[glossary/scaling-laws|Scaling Laws]] — the empirical relationships between model size, compute, data, and performance - [[glossary/transformer|Transformer]] — the neural architecture underlying virtually all modern LLMs - [[glossary/emergent-abilities|Emergent Abilities]] — capabilities that appear in large models but are absent in smaller ones at the same task - [[courses/gemini-enterprise-agents|Course: Gemini Enterprise Agents]] — hands-on practice with the concepts covered in this entry