Garden of Thoughts | SherryAnalytics | August 17, 2026
Sub-title: Why the insect analogy, and what's hiding in a model's wrong answers.
I went back to the source this week: Distilling the Knowledge in a Neural Network (Hinton, Vinyals & Dean, 2015). Everyone cites it, fewer people read the introduction closely — and the introduction is where the whole argument actually lives.
The paper opens with a comparison to insect metamorphosis: a caterpillar is built for extracting energy from the environment, a butterfly for traveling and reproducing. Same organism, two completely different bodies, because the two life stages have completely different jobs.
Hinton's point is that ML had been quietly ignoring this distinction. Training and deployment are also two different jobs. Training just needs to extract structure from a huge, redundant dataset — it doesn't need to run in real time, and it can burn as much compute as it wants. Deployment has to serve real users, which means low latency and tight resource budgets. Despite that, we were using the same model for both. The analogy gives permission to stop doing that: train something as large and cumbersome as it takes to learn well — an ensemble, or one huge heavily-regularized model — and only worry about shrinking it once training is done.
The insect image is doing more than justifying two different model sizes, though. It's dismantling a specific mental block: we tend to treat a model's knowledge as identical to its learned parameter values. Under that view, distillation looks incoherent — different weights should mean different knowledge, full stop.
A caterpillar becoming a butterfly is a counterexample everyone already accepts without thinking about it. The physical form changes entirely, but the genome doesn't — same DNA runs both bodies, just expressed differently at each stage. Nothing about the animal's essential nature was "left behind" in the old body. That's the move Hinton wants the reader to make: define knowledge not as the parameters themselves, but as the mapping a model has learned — the function from inputs to output probabilities. A mapping can survive a change of physical implementation. Weights can't be transplanted like a genome can't be swapped out; a function can be re-learned by something else.
Once knowledge is redefined as "the mapping," the next question is what that mapping actually contains that's worth transferring. A classifier trained the normal way only gets a training signal on the correct label. Nothing directly teaches it what probability to put on the wrong ones — and yet those probabilities aren't uniform. They end up structured, because the model's internal representations generalize.
The paper's example: a picture of a BMW might get a probability of 10⁻⁶ of being a garbage truck, and 10⁻⁹ of being a carrot. Both round to "never" under a hard label, and a hard label training target throws that distinction away completely. But the ratio is not noise — it's the model telling you it has learned that BMWs and garbage trucks are both vehicles, share shapes and contexts, in a way neither shares with a carrot. That's real, acquired knowledge about how classes relate to each other, and it lives almost entirely in probabilities small enough that we normally treat them as zero.
Hinton later gave this a name: "dark knowledge." Dark as in dark matter — most of what the model actually learned is invisible to the training objective and to anyone just reading off the top prediction, but it's there, it has mass, and it shapes how the model generalizes. Distillation is the process of making that invisible structure visible enough to transfer.
So the small model in distillation doesn't inherit the big model's weights at all — it can be a completely different size or architecture, initialized from scratch. What it inherits is behavior: it's trained to match the big model's full soft output distribution, dark knowledge included, rather than just the hard correct label. Two models, unrelated weights, and — if distillation works — a similar learned mapping. The butterfly, again: different implementation, same knowledge carried forward.
Signoff: The insect analogy isn't decoration — it's the paper's answer to "how can two different sets of weights hold the same knowledge?"
Idea, outline, and argument: mine. I used Claude to help tighten the prose.