A machine learning technique that divides a problem space into homogeneous regions using multiple expert networks. This approach, also known as ensemble learning, leverages multiple specialized sub-networks to handle different aspects of a problem. Originally referred to as committee machines, the Mixture of Experts (MoE) architecture includes a routing network that decides which experts should process each input, optimizing both performance and cost.
How Mixture of Experts Works
The MoE architecture consists of several key components that work together to process inputs efficiently:
- Experts (E): Individual sub-networks, each with its own parameters, that specialize in different patterns within the data.
- Gating Network (G): Decides which experts to activate for each input and how heavily to weight each one.
- Sparse Activation: Only the selected experts run for any given token, allowing the model to scale up in capacity without a proportional increase in compute per token.
Loading formula...
Scale Without the Full Cost
MoE allows frontier large language models (LLMs) to reach trillions of total parameters while only activating a small fraction for any single request. This keeps inference cost manageable, making it a practical solution for large-scale applications.
Specialization
Different experts within the MoE architecture can implicitly specialize in different types of content, languages, or reasoning patterns. This specialization enhances the model's ability to handle diverse and complex problem spaces effectively.
