A team at a health-tech startup ships a symptom-checker app that runs a 70-billion-parameter model in the cloud. It works well. Then finance asks why the inference bill is bigger than payroll, and the mobile team asks why the app needs a data connection just to answer “is this rash normal.” The fix everyone reaches for is the same DeepSeek R1 was based on this: don’t shrink the model by cutting it apart. Teach a small model to think like the big one.
That’s distillation, and it’s become the default answer to “how do we make AI cheap enough to actually deploy.”
What Is Distillation
Distillation is how you take a really smart, really expensive AI model and teach a smaller, cheaper one to act almost as smart, without starting from zero.
Think of it like an apprenticeship. You’ve got a master craftsperson, someone who’s spent years learning a trade, and a new apprentice who just started. Instead of making the apprentice spend fifteen years figuring everything out through trial and error the way the master did, the master just shows the apprentice how they work: here’s how I approach this problem, here’s what I consider, here’s why I choose this option over that one. The apprentice learns much faster because they’re learning from someone who already did the hard work of becoming an expert.
The teacher doesn’t hand over its actual code or its internal wiring. Instead, the student watches how the teacher answers a huge number of questions, and the student is trained to produce answers that match the teacher’s. Over time, the student starts responding the way the teacher would, even though it’s a fraction of the size and never went through the teacher’s original, expensive training process.
That’s the whole idea in a sentence: a big model teaches a small model to imitate it, and the small model ends up surprisingly capable for how little it costs to run.
Why You Can’t Just Delete Model Parameters
The obvious move is compression: prune weights, quantize to lower precision, cut layers. All of this works to a point, but it treats the model like a file you’re zipping. It doesn’t ask what the model actually learned, only how to represent it with fewer bits.
Distillation asks a different question. Instead of shrinking the teacher, you train a smaller student from scratch and use the teacher’s outputs as the curriculum. The insight, formalized by Geoffrey Hinton and collaborators in 2015, is that a trained model’s wrong answers carry information a correct answer never gets to show.
Ask a friend who’s eaten at every restaurant in town where to get dinner, and they might just say “the Italian place on 5th.” Useful, but thin. Ask a food critic the same question and you get a ranked list: Italian first, then the ramen place, then Thai, with ratings for everything down to the gas station hot dogs they’d only eat if starving. That ranking, the relative confidence across every option, tells you something the single answer doesn’t. It tells you why the top pick is the top pick, because you can see what it beat and by how much.
Neural networks do this too. A well-trained image classifier looking at a BMW doesn’t just say “car.” Internally it assigns tiny probabilities to every wrong class, and those probabilities aren’t random. It might give a garbage truck a probability of one in a billion and a carrot one in a trillion, because a BMW shares more visual structure with a truck than with a vegetable. That’s the ranked list. Hinton called it dark knowledge, and standard training throws it away because the loss function only cares about the top answer.
To recover it, you soften the teacher’s output with a temperature parameter before training the student on it. Turn up the temperature and the near-zero probabilities stop looking like zero. The student trains on this softened distribution instead of a single correct label, and it ends up absorbing the teacher’s whole sense of how classes relate to each other, not just which one wins.
Types of Distillation
Let’s see the different types of distillation.
