Insight · October 28, 2026
Many experts,
a few wake up.
Mixture of experts splits a model into many specialists and routes each token to only a few. It is how a model holds huge total parameters while running a small slice of itself on every word.
01 · The short answer
One model, many specialists, a router that picks a few.
Mixture of experts is a neural network design that splits a model into many specialized subnetworks called experts and adds a router that picks a few of them for each input. Only the chosen experts run, so a model can hold a large number of total parameters while using a small fraction on any single token.
The opposite is a dense model, where every parameter runs on every token. Mixture of experts is called sparse because most of the model stays idle on any given token. That one difference is what lets the open models you can name grow to hundreds of billions of parameters without the cost of running all of them at once.
02 · Why it exists
More capacity is useful. Running all of it is not.
A bigger model usually knows more and reasons better, but a dense model pays for its size on every token. Double the parameters and you roughly double the cost of every answer. For a model served millions of times a day, that cost is the ceiling on how large it can be.
Mixture of experts breaks the link between how large a model is and how much it costs to run. The idea was named in a 2017 paper from Google Brain, which set out a layer of many expert networks and a gating network whose output is a sparse vector, so most experts are skipped for any given input. The authors reported more than a thousandfold increase in model capacity with only minor losses in efficiency. You add capacity by adding experts. You hold the running cost flat by routing each token to only a few.
The one line to keep
“A model can be enormous on paper and cheap on every token, as long as it runs only a small part of itself at a time.”
03 · How it works
Five moves, token by token.
01
The router scores
A small gating network reads the incoming token and scores every expert for it. The router trains alongside the rest of the model, so the scores are learned, not fixed. Nothing has run yet. The router has only decided who should.
02
Pick the top few
The router keeps only the highest scoring experts, often the top two, and ignores the rest. This is the whole trick. The experts it skips cost nothing on this token, which is why a model can be enormous on paper and cheap to run.
03
The chosen experts run
Only the selected experts process the token. Each is a feedforward network with its own weights, trained to handle part of the space of inputs. In a language model these expert layers sit where the ordinary feedforward block used to be.
04
Combine the outputs
The outputs of the chosen experts are blended into one result. The router’s own scores double as the mixing weights, so an expert the router trusted more counts for more. The next token starts the whole process again, and the chosen pair can change.
05
Keep the load balanced
Left alone, a router tends to send most tokens to a few favorite experts and starve the others. Training adds a load balancing pressure that spreads tokens across experts so the capacity is actually used, not collapsed onto a handful.
04 · The two numbers
Total parameters and active parameters are not the same bill.
Every mixture of experts model has two sizes, and reading them correctly is most of understanding the architecture. The total parameter count is how large the model is, the sum of every expert. The active parameter count is how much of it actually runs on a single token. In a dense model the two are equal. In a sparse one they are far apart.
Mixtral 8x7B is about 47 billion parameters in total and about 13 billion active. DeepSeek-V3 is 671 billion total and 37 billion active. The quality tends to follow the total size, while the speed and the compute cost follow the active size. That split is the whole appeal.
In a language model the experts are not the whole network. They replace the ordinary feedforward block inside each layer, the part that does most of the per token computation, and leave the attention machinery shared. So a mixture of experts model is a mostly familiar model with its feedforward layers swapped for a routed set of experts.
05 · The method, in public
The architecture under the models you can download.
Mixture of experts is not a whiteboard idea. It is the shape of the open models that have defined the recent frontier, and the figures each one publishes tell the same story.
Mixtral 8x7B
Mistral AI’s open model, released in January 2024, is the clearest worked example. Each layer holds eight feedforward experts, and a router picks two of them per token, with the chosen pair free to change from one token to the next. The model is about 47 billion parameters in total but uses only about 13 billion for any given token. It ships under Apache 2.0 with a 32k token context, and the paper reports it matching or beating Llama 2 70B and GPT-3.5 on its benchmarks.
DeepSeek-V3
The December 2024 technical report describes a mixture of experts model with 671 billion total parameters and 37 billion activated for each token, pre-trained on 14.8 trillion tokens. The gap between those two numbers is the point of the architecture. Almost twenty times the parameters sit in the model than run on any single token.
Switch Transformer
The Google paper by Fedus, Zoph, and Shazeer, published in JMLR in 2022, pushed routing to its simplest form: one expert per token rather than several. That choice cut the communication and complexity cost of earlier designs and let the authors scale to trillion parameter models, with the largest version reaching roughly 1.6 trillion.
Llama 4 Maverick
Meta’s open weight model, released in April 2025, carries 400 billion total parameters but activates only 17 billion per token, routed across 128 experts plus a shared one, with dense and expert layers alternating. It is the same move at a different scale: hold a very large model, run a small slice of it.
06 · What it costs
Sparse is cheaper to run, not free to own.
The savings are real but narrow. Mixture of experts cuts the compute per token, not the memory. Every expert has to be loaded and ready in case the router picks it, so the hardware you need to serve the model tracks the total size, not the active slice. A model that runs like a small one can still demand the memory of a large one.
Routing is the second cost. A router left to itself tends to favor a few experts and let the rest go unused, which wastes the capacity you paid to add. Training has to push back on that so work spreads across the experts. Simplifying the router is a line of research in its own right. The 2022 Switch Transformer paper cut routing to a single expert per token to tame the complexity, communication cost, and training instability of earlier designs.
These models are also harder to train and fine-tune cleanly. The same Switch Transformer work that simplified routing was aimed in part at the training instability that sparse models are prone to, and the extra capacity can overfit where a comparable dense model would not. So mixture of experts is a trade, not a free lunch: you spend memory and training care to buy cheaper inference. For a model served at scale, that is usually the right trade.
07 · Where it belongs
A router inside the model, a router around it.
For most teams, mixture of experts is not a thing you build. It is a thing you buy, already baked into the models you choose. The useful move is reading the two numbers correctly. A model with a large total and a small active size gives you more capability per dollar of inference, and that ratio belongs in the decision as much as the benchmark scores do.
The idea also rhymes with the layer above the model. Inside a mixture of experts model, a router picks the right expert for each token. Across a stack of models, the same logic picks the right model for each request. That outer router is what we build at Bttr. as the token control plane, the middle layer between a company and every AI vendor. For every question your team asks, it picks the right AI to answer it: hard reasoning goes to one model, quick lookups go to a cheaper one.
It is the same principle at two scales. Route each token to the expert that should handle it, and route each request to the model that should handle it. A mixture of experts model is one lesson in not running your largest option on work that does not need it. Routing across models is the same lesson one level up.
Closing
A big model can run a small part of itself.
Next time a model lists two parameter counts, read both. The large one tells you how much it knows. The small one tells you what it costs to ask. Mixture of experts is the architecture that holds those two numbers apart on purpose.
Written by · October 28, 2026
Donny Smith · ECD, Founder, Bttr.
Over the past 15 years, he has led creative teams and contributed to products used by millions of people worldwide, working with companies including Apple, Opendoor, JP Morgan, GE Aerospace, Pepsi, and Alterra Mountain Company.
LinkedInFacts current as of October 2026 · Shazeer et al. · Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer · January 2017 · Mistral AI · Mixtral of Experts · January 2024 · DeepSeek-AI · DeepSeek-V3 Technical Report · December 2024 · Fedus, Zoph & Shazeer · Switch Transformers: Scaling to Trillion Parameter Models · JMLR 2022 · Meta AI · The Llama 4 Herd · April 2025
Share this perspective
More insights
Adjacent perspectives.
Bttr. Field Brief
The brief Bttr. writes for senior buyers.
Monthly. One signal worth your time on Brand Operating Systems, AI search visibility, and the infrastructure buildout. No filler.