Insight · October 26, 2026
A big model
teaches a small one.
Model distillation trains a small model to copy a large one. You keep most of the capability and shed most of the cost, which is why the smallest useful models of 2026 were taught by bigger ones.
01 · The short answer
A big model teaches a small one to do its job.
Model distillation is a training method that transfers the behavior of a large, capable model into a smaller one. The large model, the teacher, produces outputs. The small model, the student, trains to reproduce them. The result is a cheaper, faster model that keeps most of the teacher’s skill on a defined task.
The idea is older than the current wave. It was named in a 2015 paper, Distilling the Knowledge in a Neural Network, by Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. What changed is the stakes. When the strongest models are expensive to run, a smaller model that behaves like one stops being an academic trick. It becomes the difference between a feature you can afford to ship and one you cannot.
02 · Why it exists
The best model is often too expensive to serve.
A frontier model is trained once and then answers requests forever, and every one of those requests costs money, power, and time. For a narrow job done millions of times, paying frontier prices on every call is the most common reason an AI feature never reaches a margin.
Distillation is one answer. Instead of serving the large model, you use it once as a teacher to train a small model, then serve the small one. The teacher’s cost is paid during training, not on every request. What you ship is a model small enough to run cheaply, sometimes on a single GPU, a laptop, or a phone, that still does the one job you trained it for. It is the training method behind many of the small specialists that run at a fraction of frontier cost.
The one line to keep
“A frontier model you cannot afford to serve can still teach a smaller one that you can.”
03 · How distillation works
Five moves, teacher to student.
01
Pick the teacher
Start with a model that already does the job well, usually a large and expensive one. It does not have to be yours. Any model you can run and collect outputs from can teach.
02
Generate the lessons
Run the teacher across a large set of inputs and record what it produces. Those outputs, not hand labels, become the training data. The student learns from the teacher’s answers, including the shades of its judgment.
03
Train the student
A smaller model trains to match the teacher. In the classic form it learns from soft targets, the teacher’s full spread of probabilities at a raised temperature, which carries more signal than a single right answer. In the common form today it trains on the teacher’s generated text.
04
Serve the small one
You deploy the student and take the teacher out of the request path. The teacher’s cost stays in the past. Every call from here runs on the cheaper model.
05
Measure the gap
A student is never a perfect copy. You compare it to the teacher on the task that matters and decide whether the quality you kept is worth the cost you cut. That gap is the whole decision.
04 · Not the same lever
Distillation, fine-tuning, and quantization are different jobs.
Three methods get confused because all three can end in a cheaper or more specialized model. They change different things.
Fine-tuning changes a model’s behavior by training it further on your own examples. It starts from one model and adjusts it. Distillation starts from two, a teacher and a student, and moves capability from the large one into the small one. You can fine-tune a model during distillation. They are not rivals.
Quantization leaves the model’s knowledge alone and stores its numbers at lower precision, so the same model runs smaller and faster with a little quality lost. Pruning cuts the weights a model barely uses. Both compress a model in place. Distillation does not shrink a model at all. It trains a new, smaller one to behave like the big one.
The clean line: quantization and pruning make the same model lighter. Distillation makes a different, smaller model that acts like the heavy one.
05 · The method, in public
Two models that prove it works.
Distillation is not a whiteboard idea. Two well known models were built by it, one recent and one a template the field still copies.
DeepSeek-R1 distilled models
In early 2025 DeepSeek released six smaller dense models distilled from its DeepSeek-R1 reasoning model, built on the Qwen2.5 and Llama3 series at 1.5B, 7B, 8B, 14B, 32B, and 70B. They were fine-tuned on 800,000 samples generated by DeepSeek-R1, using supervised fine-tuning alone with no separate reinforcement learning stage. DeepSeek-R1-Distill-Qwen-7B scored 55.5 percent on AIME 2024, and the 32B version reached 72.6 percent on AIME 2024 and 94.3 percent on MATH-500, results DeepSeek reports as comparable to OpenAI’s o1-mini.
DistilBERT
The template example. Hugging Face created DistilBERT by knowledge distillation to be a smaller language model with faster inference that needs less compute to train, using a triple loss during pretraining that combines a language modeling loss, a distillation loss, and a cosine-distance loss. It showed that a small student could carry much of a larger model’s language ability, the result that made distillation a standard tool.
06 · What you keep, what you give up
A transfer with a known loss, not free capability.
Distillation buys a model that is cheaper to run, faster to answer, and small enough to put where a frontier model cannot go. For a narrow, repeated task, a distilled student can match its teacher closely enough that no user would notice the difference.
The costs are real and worth naming. A student inherits the teacher’s blind spots along with its skill, so a flaw in the teacher ships inside the student. The student is narrow, matching the teacher on the task it was trained for and often falling short outside it. And the method needs a teacher you are allowed to learn from. Training a model on another vendor’s outputs can cross that vendor’s terms, so the license matters as much as the technique.
So distillation is not free capability. It is a transfer with a known loss, and the job is to measure that loss on the work you actually do.
07 · Where it belongs
A distilled model is the cheap specialist your stack routes to.
On its own, a smaller and cheaper model is only half the win. The other half is a system that knows when to use it. A distilled student is exactly the kind of model a routing layer reaches for on easy, repeated work, with a stronger model held back for the rest.
That is the layer we build at Bttr. as the token control plane, the middle layer between a company and every AI vendor. One of its four jobs is to pick the right AI to answer each question. Hard reasoning goes to one model. Quick lookups go to a cheaper one. A distilled model, trained once for a job you run constantly, is the cheaper one worth owning.
Put the two together and the economics change. The teacher trains the specialist. The control plane decides when the specialist is enough. What you spend to serve a feature stops being set by your largest model.
Closing
You do not have to serve the model that taught you.
Pick one job you run constantly on an expensive model. Use that model to generate a few thousand good answers. Train a small model on them and measure the gap on your own task. The first time the small one holds, you will stop paying frontier prices for work a student can do.
Written by · October 26, 2026
Donny Smith · ECD, Founder, Bttr.
Over the past 15 years, he has led creative teams and contributed to products used by millions of people worldwide, working with companies including Apple, Opendoor, JP Morgan, GE Aerospace, Pepsi, and Alterra Mountain Company.
LinkedInShare this perspective
More insights
Adjacent perspectives.
Bttr. Field Brief
The brief Bttr. writes for senior buyers.
Monthly. One signal worth your time on Brand Operating Systems, AI search visibility, and the infrastructure buildout. No filler.