Mixture of Experts (MoE)

Mixture of Experts is a model architecture where each layer has many "expert" sub-networks, and a router picks just two or three of them to run for each token. The published parameter count looks huge (e.g., 671B for DeepSeek-V3) but only a fraction (37B) is actually used per inference step.

The win: capacity without proportional cost. The trade-off: more memory required (all experts must be loaded), training is harder, and inference parallelism is trickier. From a buyer's perspective you mostly don't need to care — you'll see it in the model card. From a self-hosting perspective it's a major engineering decision.