What "671 billion parameters" really means: mixture-of-experts explained

Mixture-of-experts (MoE) is a way of building an AI model so that only a few of its parts, called experts, work on each token. DeepSeek-V3 has 671 billion parameters, but only 37 billion of them do any work on a given token. That gap is the whole idea behind MoE: a model that is very large on disk but behaves like a much smaller one each time it produces a word.

A parameter is one of the numbers a model learns during training. More parameters usually means more knowledge and skill, and, in a classic model, more computing work for every token it reads or writes. MoE breaks that link.

How does a dense model work?

In a dense model, each token passes through every layer, and every parameter in those layers takes part in the calculation. A 70-billion-parameter dense model does roughly 70 billion parameters' worth of maths for each token. Size and cost per token rise together.

How does a mixture-of-experts model work?

An MoE model replaces some layers with a set of smaller sub-networks called experts. A small extra network, the router (also called the gate), looks at each token and picks which experts should handle it. The rest sit idle for that token.

Mixtral 8x7B from Mistral AI is a clear example. Each of its MoE layers holds 8 experts, and the router chooses 2 of them for every token at every layer. Mistral says the model has 46.7 billion parameters in total but uses 12.9 billion per token, so it runs at the speed and cost of a 12.9-billion-parameter model.

Newer models use many more experts. Qwen3-235B-A22B has 128 experts per MoE layer and activates 8 of them for each token.

Where does the idea come from?

The modern version starts with a 2017 paper by Noam Shazeer and colleagues. It described a "sparsely-gated" MoE layer with up to thousands of experts and a trainable gate that sends each input to a small subset of them. The authors reported more than a 1,000-fold increase in model capacity with only minor losses in computational efficiency, and trained a model of 137 billion parameters.

In 2021 the Switch Transformer paper simplified the design: route each token to a single expert instead of several. The authors said this cut routing computation and communication costs while keeping quality, and reported up to a 7x speed-up in pre-training over comparable dense models using the same compute. They trained models of up to a trillion parameters.

What is the difference between total and active parameters?

Total is everything stored in the model, and active is the part used for each token. Model cards for MoE models now usually give both numbers:

• Total parameters: every number in the model, across all experts. This is what has to be stored.
• Active parameters: the share actually used to process one token. This is what sets the compute per token.

Figures are from each model's official card or announcement. The Qwen model's name spells out both numbers: 235 billion total, "A22B" for 22 billion active.

Does MoE make a model cheaper to run?

It cuts compute, not memory. MoE cuts the arithmetic per token, because only the chosen experts run. It does not cut memory. The router can pick any expert for the next token, so all of them must be loaded and ready.

That is why the total count still matters for anyone running a model. OpenAI says gpt-oss-120b, with 117 billion total parameters, is designed to fit on a single 80GB GPU. DeepSeek publishes V3's weights in FP8, a format that uses one byte per number, so the 671 billion main-model parameters alone come to roughly 671GB before any working memory. Our arithmetic, not DeepSeek's figure.

Training has its own savings. DeepSeek says the full training of V3 took 2.788 million H800 GPU hours on 14.8 trillion tokens. Only 37 billion parameters are active per token, so each training step costs far less compute than it would if all 671 billion ran every time.

What are the downsides of MoE?

The design brings problems of its own.

• Load balancing. If the router sends most tokens to a few favourite experts, those experts become a bottleneck and the rest go unused. The Switch Transformer added an extra training penalty to spread tokens out. DeepSeek-V3 says it balances load without that extra loss term.
• Dropped tokens. In the Switch design each expert has a fixed capacity. When too many tokens pick the same expert, the overflow skips that layer. The authors kept this under about 1% in their experiments.
• Communication. Experts are often spread across many chips, so tokens have to be shipped between them. The original papers name communication cost and training instability as reasons MoE was slow to catch on.
• Hardware fit. A model with few active parameters can be fast, but only once you have enough memory to hold all of it.

Our take

The active count is the better guide to how fast and how cheap each answer can be. The total count is the better guide to what it takes to host the model. Neither alone says how good it is: a 671-billion-parameter MoE and a 671-billion-parameter dense model are very different machines.

How to read a parameter claim

1. Look for two numbers. If a launch quotes one large figure, check whether it is total or active.
2. Use total for hardware. Multiply by bytes per parameter (two for 16-bit weights, one for 8-bit) for a rough memory floor.
3. Use active for speed and cost. It is the closer match to a dense model of the same compute.
4. Check the benchmarks separately. Parameter counts describe size, not results.