Mixture of Experts (MoE)

Mixture of Experts (MoE) is a neural network architecture design where, instead of every input passing through the model's entire set of parameters (a "dense" model), the model is composed of many smaller specialized sub-networks ("experts"), and a lightweight routing mechanism decides which handful of experts should process each specific input token — meaning only a fraction of the model's total parameters actually activate for any given piece of input, even though the full model may contain a much larger total parameter count. This matters enormously for SaaS and AI builders because it changes the cost/capability trade-off in a favorable direction: an MoE model can have a very large total parameter count (giving it broad capability and specialized knowledge spread across many experts) while keeping inference cost much closer to that of a much smaller dense model, because only the "active" parameters for each token actually need to be computed. Several leading frontier and open-weight models use MoE architectures, including Mistral's Mixtral models and reportedly some of the largest closed frontier models, because it's one of the more effective known techniques for scaling total model capability without proportionally scaling inference cost and latency. A concrete illustrative example: a hypothetical MoE model might have a total of 141 billion parameters spread across 8 experts, but route each token through only 2 of those experts at a time — meaning inference cost and latency for any given request resembles a roughly 39-billion-active-parameter dense model, even though the model's full capacity (and the memory needed to store all the experts, even unused ones) reflects the full 141 billion. For builders, this explains why some very capable models are also surprisingly fast and affordable to run — MoE is often the architectural reason — and why MoE models have a somewhat unusual resource profile: they need enough memory/VRAM to hold the entire model (all experts, since which ones activate depends on the input and can't be predicted in advance), but the actual compute per request is much lower than a dense model of equivalent total size, which matters when evaluating self-hosting infrastructure requirements. MoE architectures also introduce operational considerations distinct from dense models: because routing decisions happen per-token and can vary between tokens in the same request, batching multiple users' requests together for throughput efficiency is more complex than with a dense model (where every token takes the same computational path), and serving infrastructure for MoE models needs to account for this when optimizing for cost and latency at scale — a detail mostly invisible to builders consuming MoE models via a standard API, but directly relevant to teams self-hosting one.