What Is Mixture of Experts (MoE)? How It Works

Published:

15 minute read

Acar Diveroli
Written by: Acar Diveroli
Wireframe sphere split into expert patches; a token passes a router box and lights two patches, one of them blue

An open model's technical report gives its size as 47 billion parameters and, in the same sentence, says it uses only 13 billion of them for each token. Both numbers are correct, and the gap between them is the whole idea of mixture of experts: the model owns far more parameters than it uses for any single token it reads or writes. If you plan to run such a model yourself, the first number decides how much memory you need and the second decides most of the compute.

This post explains what a mixture-of-experts (MoE) model is, how the router picks experts, what total and active parameters mean, how MoE compares with a dense model, what it costs and whether the experts really specialise.

What is mixture of experts?

Mixture of experts is a way of building a neural network layer out of several smaller networks, the experts, and a router (also called a gate) that decides which of them handle each input. A dense layer runs all of its weights on every input; an MoE layer runs only the experts the router picks, so most of its weights sit idle at any moment. That is why MoE models are also called sparse models: for each token, only a sparse subset of the network is active.

The design matters most in large language models, neural networks trained on huge amounts of text to predict the next piece of text; our LLM guide covers how they are built. Each transformer layer of such a model has an attention block, which lets every token take the other tokens into account, and a feed-forward block, which then processes each token on its own. An MoE model keeps the attention block and replaces the single feed-forward block with many expert versions of it, each with its own weights.

The word "expert" is a name, not a job title: nobody assigns an expert to a subject, and whatever division of work appears is learned in training.

Where did the idea come from?

The idea is more than thirty years old. In 1991, Robert Jacobs, Michael Jordan, Steven Nowlan and Geoffrey Hinton published "Adaptive Mixtures of Local Experts" in the journal Neural Computation: separate networks, each learning a subset of the training cases, with a gating network deciding which one to trust.

The form used in today's language models starts with a 2017 paper by Noam Shazeer and colleagues, Outrageously Large Neural Networks. It placed a sparsely-gated MoE layer with up to thousands of experts between the layers of a language model, reached up to 137 billion parameters, and reported more than 1000 times more model capacity with only minor losses in computational efficiency.

In 2021 the Switch Transformer paper sent each token to a single expert and trained models of up to 1.6 trillion parameters. Its authors reported up to 7 times faster pre-training than a dense T5-Base model with the same computational resources, and named what had held MoE back: complexity, communication costs and training instability. Open-weight MoE language models followed, among them Mistral AI's Mixtral 8x7B and DeepSeek-V3.

How does a mixture-of-experts layer work?

An MoE layer scores every expert for each token, runs only the best few and blends their results. Before that point the text has been cut into tokens, the small pieces a model reads, such as words or parts of words (our guide to tokens in AI shows how), and each token travels through the model as a vector, a long list of numbers.

  1. The token arrives as a vector from the attention block of the same layer, which has already mixed in information from the surrounding tokens.
  2. The router scores every expert. The router is a small learned layer that gives each expert one score for this token. It is cheap to run, however large the experts are.
  3. Only the top-k experts are kept. The number k is fixed when the model is designed: 1 in the Switch Transformer, 2 in Mixtral, 8 of 256 routed experts in DeepSeek-V3. All other experts get a weight of zero.
  4. The kept scores become weights. A softmax function turns them into shares that add up to 1.
  5. The chosen experts process the token. Each expert is an ordinary feed-forward network with its own weights.
  6. The outputs are mixed. The layer's output is the weighted sum of the chosen experts' outputs, and it moves on to the next layer.
  7. The next layer chooses again. Every MoE layer has its own router and experts, so one token can use experts 2 and 5 in one layer and experts 1 and 7 in the next.

A made-up example shows the arithmetic. In a layer with 8 experts, the router gives a token a score of 2.1 for expert 3, 1.3 for expert 6 and lower scores for the other six. With top-2 routing only experts 3 and 6 run; a softmax over the two kept scores gives weights of about 0.69 and 0.31, so the layer's output is 69 percent expert 3's result and 31 percent expert 6's. Six experts did no work for this token.

Designs vary: DeepSeek-V3, for example, adds one shared expert that every token passes through, next to the routed experts.

Total vs active parameters: what is the difference?

Total parameters are every weight the model has; active parameters are the weights actually used to process one token. A parameter is one learned number inside the network. A dense model uses all of its parameters for every token, so for it the two counts are the same; in an MoE model they split apart.

The Mixtral paper gives a clean example. Each layer has 8 experts and the router picks 2 for every token, so each token has access to 47 billion parameters but uses only 13 billion. The name "8x7B" suggests 56 billion, but only the feed-forward blocks exist eight times over; attention and the other parts of each layer are shared and counted once.

DeepSeek-V3 widens the gap: 671 billion total parameters, of which 37 billion are activated for each token, roughly one weight in eighteen.

The two numbers answer different questions:

  • Total parameters set the memory. Every expert has to be loaded, because the next token may need any of them. Mixtral's authors write that its serving memory cost is proportional to the total, sparse parameter count.
  • Active parameters set most of the compute per token, and with it much of the cost of each generated token.
  • Neither number alone tells you quality. A model with 13 billion active parameters is not the same as a dense 13-billion-parameter model, because it draws on a much larger pool of stored weights.

Dense vs MoE model: what is the difference?

A dense model runs every parameter for every token, while an MoE model of the same total size runs only a fraction of them, which saves compute but not memory. The table compares the two at the same total parameter count.

Dense modelMoE model
Parameters used per tokenAll of themOnly the chosen experts and shared parts
Compute per tokenGrows with total sizeGrows with active size
Memory to serveTotal sizeTotal size: every expert loaded
TrainingWell understoodLoad balancing, less stable
Across many chipsSplit by layers or tensorsExperts on different chips, more traffic
Fine-tuning on little dataPredictableCan overfit sooner

The picture changes if you hold compute fixed instead. Against a dense model with the same number of active parameters, an MoE model does about the same arithmetic per token but needs several times the memory, and in return it has far more weights in which to store what it learned.

What does mixture of experts cost?

Mixture of experts trades compute for memory, and it brings three problems a dense model does not have: uneven load on the experts, less stable training and more communication between chips.

Memory. Every expert has to be loaded on some accelerator, although most of them are idle for any given token. The DeepSeek-V3 technical report describes a minimum deployment unit of 40 nodes with 320 GPUs for the stage that generates text, with each GPU holding one expert. Running an MoE model on a single machine therefore means fitting all of its experts in memory, not just the active ones.

Load balancing. If the router keeps choosing the same few experts, those experts get more training, get better and are chosen even more often; Shazeer's paper calls this imbalance self-reinforcing. The usual fix is an auxiliary loss, an extra term in the training objective that penalises uneven loads. The Switch Transformer also gives each expert a fixed capacity per batch: a token that arrives when its expert is full skips the expert and goes straight to the next layer, which is called token dropping. DeepSeek-V3 uses a per-expert bias adjusted during training instead, because, its authors write, too large an auxiliary loss impairs model performance.

Training instability. Routers are sensitive to small numerical errors. The Switch Transformer authors found that instability stopped them from training in bfloat16, a low-precision number format, and fixed it by computing the router in 32-bit precision and starting from smaller initial weights. Meta's Llama 3 report makes the opposite choice for the same reason: its largest model, with 405 billion parameters, is a standard dense transformer, chosen over a mixture of experts to maximise training stability.

Communication. Large MoE models spread their experts over many chips, so every MoE layer has to send each token to the chips that hold its chosen experts and collect the results. DeepSeek-V3 limits this traffic by sending each token to experts on at most 4 nodes.

Are the experts specialists in topics?

Not in the way the name suggests: studies of trained MoE models find experts that handle kinds of tokens, such as punctuation or pieces of code syntax, rather than subjects such as medicine or law.

The Mixtral team checked which experts were picked for text from different sources, including arXiv papers, biology abstracts from PubMed, philosophy papers and GitHub code, and found no obvious pattern by topic. Routing looked more tied to syntax: the word self in Python and the indentation tokens in code often went to the same experts, and consecutive tokens often shared experts.

Earlier work points the same way. Shazeer's 2017 paper reported experts that became highly specialised by syntax and semantics. The ST-MoE paper from 2022 found encoder experts for punctuation, verbs, proper names and counting, much weaker specialisation in the decoder, and no evidence that experts specialised by language in a multilingual model.

So a prompt cannot be pointed at "the legal expert": the router decides per token and per layer, and the division of work it learns is statistical.

Where do you meet mixture of experts?

  • Model cards and technical reports that give two sizes, total and active parameters: the clearest sign of an MoE model.
  • Open-weight models you download and run, such as Mixtral and DeepSeek-V3, where the total size decides whether the model fits on your hardware.
  • Closed models behind an API, where the architecture is often unpublished. Some vendors publish it: Google's Gemini 1.5 report describes Gemini 1.5 Pro as a sparse mixture-of-experts transformer. Where a vendor has said nothing, claims about its design are guesses.

Does MoE change the data a model needs? No. Sparse or dense, a language model is trained on very large collections of text, much of it from public web pages. Teams that tune, ground or evaluate models keep collecting pages as well, often from several countries, because prices, rules and product details differ by market. That collection work is where proxies fit: a Residential Proxy lets a crawler load a public page as a visitor in a given country sees it, at a request rate the site can handle.

Common mistakes

  • Sizing hardware by active parameters. A model with 13 billion active parameters still needs memory for all 47 billion.
  • Multiplying the name. "8x7B" is not 56 billion parameters; the shared parts of each layer are counted once.
  • Comparing models by one number. Use active parameters for compute, total parameters for memory, and your own test prompts for quality.
  • Expecting topic experts. A prompt cannot select an expert, and experts rarely map to subjects.
  • Fine-tuning on a tiny dataset without a held-out check. The ST-MoE study found sparse models more prone to overfitting on small tasks, so measure on examples the model has not seen; the basics are in our fine-tuning guide.
  • Repeating rumours about closed models. If a vendor has not published the architecture, it is unknown.

Decision guide

Your situationWhat MoE means for you
You call a model through a paid APICompare price, latency and answers; the architecture is the provider's concern
You run an open model on your own GPUAll total parameters must fit in memory
You want more quality for the same computeAn MoE model is worth testing
Memory is your hard limitA smaller dense model is simpler
You serve many users on a GPU clusterPlan expert placement and chip-to-chip traffic
You fine-tune on a small datasetWatch held-out results for overfitting
You compare two model cardsRead total and active parameters separately

Frequently asked questions

Is an MoE model faster than a dense model?

Per token, yes, compared with a dense model of the same total size, because it computes only the chosen experts. Compared with a dense model of the same active size, it does about the same arithmetic but moves more data, because different tokens in a batch call different experts. Real speed depends on the hardware, the batch size and where the experts are placed.

Why do MoE models need so much memory?

Because every expert has to stay loaded, although each token uses only a few. The router decides at run time, token by token, so any expert may be needed next. Mixtral's authors state that its serving memory follows its 47 billion total parameters, not its 13 billion active ones.

What does top-k routing mean?

Top-k routing means the router keeps the k experts with the highest scores for a token and ignores the rest. The value of k is a design choice: the Switch Transformer uses 1, Mixtral 2, and DeepSeek-V3 8 of its 256 routed experts. A larger k spends more compute per token; a smaller one is cheaper.

Are the experts in an MoE model specialised by topic?

Mostly not. Studies of trained models find experts tied to token types and syntax, such as punctuation, verbs or indentation in code, and the Mixtral team found no obvious pattern by subject across science, biology, philosophy and code text.

What is the difference between mixture of experts and a model ensemble?

An ensemble runs several complete, separately trained models on the same input and combines their answers, so it pays for every model on every input. An MoE model is a single network trained as a whole, in which a router inside each MoE layer sends each token to only a few experts.

Do all large language models use mixture of experts?

No. Mixtral and DeepSeek-V3 are MoE models, and Google's Gemini 1.5 report describes Gemini 1.5 Pro as one, while the largest Llama 3 model from Meta is a dense transformer with 405 billion parameters. Many vendors of closed models do not publish their architecture at all.

Summary

Mixture of experts splits a layer into many expert networks and lets a router send each token to only a few of them, so a model can hold far more parameters than it computes with for any one token. Total parameters decide memory, active parameters decide most of the compute, and the costs are memory, load balancing, training stability and traffic between chips. The experts are learned, not assigned to subjects. For teams that need public web pages collected at scale to build or test models, our Web Scraping API, a managed page extraction service offered through the sales team, is described on the data scraping page.

Ask ChatGPTAsk Claude