loading
loading
An architecture where only a few specialized sub-networks fire per token, so a huge model runs at a small model's cost.
Instead of pushing every token through all parameters, an MoE model has many 'expert' sub-networks and a router that picks just a couple per token. So a model can have, say, 400B total parameters but only activate ~30B per token — you get the capacity of a giant model at the inference cost of a much smaller one. This is why frontier models are fast and cheap relative to their size; the tradeoff is they need all experts loaded in memory and routing can be uneven.
Plainly
Think of Mixture of Experts (MoE) as the brain part that guesses or decides. An architecture where only a few specialized sub-networks fire per token, so a huge model runs at a small model's cost.
In practice
Use it when model choice, prompts, latency, cost, or quality affect the product result. In practice, define the owner, input, output, and failure mode before you rely on it.