PHASE 13Advanced ~22 min· topic 6 of 8

Topic 13.6

Distillation and Small Models

In one line

Distillation teaches a small model from the outputs of a large one. The large model labels many of your real inputs, you keep the good labels, and the small model is trained on them. It is cheap to run afterwards, but it inherits the teacher's mistakes and the terms of use apply.

Think of it like this

An apprentice learning from a master baker. The apprentice watches the master handle hundreds of orders and copies the results. Soon the apprentice makes the usual orders as well as the master, much faster and cheaper, but still cannot do the master's special wedding cake, and copies the master's bad habits too. That is distillation: a small student learns to imitate a large teacher on a narrow job.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Teacher: The large, accurate model that produces the training labels.
Student: The small model that is trained to imitate the teacher on a narrow task.
Distillation: Training a student on a teacher's outputs so it copies the teacher's behaviour cheaply.
Synthetic data: Training examples written by a model instead of collected from people.
Confidence: How sure the teacher was about a label. Low-confidence labels are better dropped than learned.
Held-out set: Real cases with trusted labels that were never used to build or train the student.
Terms of use: The provider's rules for the service, including what you may do with its outputs.

Step by step

01Step 1: choose a narrow, high-volume task

Distillation pays off when the same kind of call happens a very large number of times and the output is short and checkable: a label, a few fields, a rewrite in a fixed style. Open-ended chat across every topic is a poor fit, because a small model cannot cover it.

02Step 2: let the teacher label real inputs

In the example the teacher is a fake function that returns a label and a confidence, which stands in for a real model call with a validated JSON reply (Topic 4.8). Twelve real-looking tickets go in. Two are vague or outside its rules, and the teacher is unsure about them.

teacher.pywhole filepython

label, confidence = teacher("Nice product by the way")
print(label, confidence)  # other 40 -> too unsure to teach from

03Step 3: filter before you train

Keep only labels at or above a confidence threshold. The example keeps ten of twelve, lists the skipped ones, and prints the first two lines of the training file as compact JSON. Real pipelines also validate the format, remove duplicates and sample for human review.

Step 3: filter before you traindiagram

Rendering diagram…

04Step 4: train a student

The toy student counts which words appear with which label and picks the label whose words match best. It is nothing like a real language model, but it has the property that matters: it knows only what its data showed it. A real student would be a small model fine-tuned with LoRA or full training.

05Step 5: test on cases the student has not seen

Six new messages, each with the label we expect. The student gets five right and one wrong: on "Screen goes white when I tap save" the only words it knows are common ones such as "I", which it saw more often with billing, so it answers billing. Read the failures; they show what to add to the data.

06Step 6: check the economics and the terms

One-off cost: the number of teacher calls times their price. Savings: the difference per call between teacher and student times the monthly volume. Compare them with the maths of Topic 13.2. Then read the terms of use for the teacher and the student base before you ship anything.

Try it yourself

1
Change the threshold
Set MIN_CONFIDENCE to 90. Which prompts are skipped now? Predict how many lines the training file will have, then run. What does a higher threshold cost you?
2
Teach the student something new
Add the prompt Screen goes white after I tap save to the pool and give the teacher a new rule that labels prompts containing white as bug with confidence 90. Run again: does the last test line now pass?
3
Test an unseen label
Add a test ("Please delete my account", "other"). What does the student answer, and why does it behave like this on words it has never seen?

Code & diagrams

A distillation data set from a fake teacher New tab

The teacher is a rules function with a made-up confidence, standing in for a large model. The student is a toy word counter.

Expected output

teacher labelled 12 prompts, kept 10, skipped 2 as low confidence
  skipped: Cancel my subscription now
  skipped: Nice product by the way
first two training lines:
  {"prompt":"I was charged twice","completion":"billing"}
  {"prompt":"Refund for my last order","completion":"billing"}
  My card was charged again -> student says billing, expected billing
  Checkout crash again -> student says bug, expected bug
  Is the app broken today -> student says bug, expected bug
  Do you ship to Nepal -> student says other, expected other
  Screen goes white when I tap save -> student says billing, expected bug
  Please cancel my subscription -> student says billing, expected billing
student agrees with the expected label on 5 of 6

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Keep every label

Set MIN_CONFIDENCE = 0.

terminal

$ Run

── what you'll see ──

teacher labelled 12 prompts, kept 12, skipped 0 as low confidence

first two training lines:

{"prompt":"I was charged twice","completion":"billing"}

{"prompt":"Refund for my last order","completion":"billing"}

My card was charged again -> student says billing, expected billing

Checkout crash again -> student says bug, expected bug

Is the app broken today -> student says bug, expected bug

Do you ship to Nepal -> student says other, expected other

Screen goes white when I tap save -> student says billing, expected bug

Please cancel my subscription -> student says other, expected billing

student agrees with the expected label on 4 of 6

Myth vs fact

Myth

A distilled model is as good as the teacher.

Fact

It can approach the teacher on the narrow task it was trained for and is usually much worse outside it.

Myth

Any model's outputs may be used to train another model.

Fact

Providers' terms and open-weights licences may restrict it. Read the terms for the exact models involved.

Myth

More teacher labels always help.

Fact

Wrong or unsure labels hurt. Filtering and checking a sample matter more than volume.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Measure the teacher against your gold labels first. If the teacher gets 90% right, the student is capped near 90%, and its errors are the teacher's errors, only more consistent.
▸
Mine hard cases: the inputs where teacher and student disagree, or where the student is unsure, are the most valuable to label by hand and add to the next round (Topic 11.6).
▸
Keep a canary set of real cases with human labels that never goes into training, and re-run it on every student version, so a better-looking average cannot hide a regression on cases you care about.
▸
Account for the full cost: teacher calls, human review time, training runs, and serving the student. Distillation is a project with a payback period, so compute it (Topic 13.2).

Remember this

1
Why this exists. A large model is accurate but slow and expensive per call. For a narrow, high-volume task (label tickets, extract fields, route requests) paying for the large model on every call is wasteful when a much smaller model could do the same job well. But a small model without training is not good enough. Distillation closes the gap by using the large model as a cheap source of training examples.
2
The pipeline. (1) Collect real inputs from your use case. (2) Ask the teacher to produce the output you want for each one, in the exact format (Phase 4). (3) Filter: keep only labels that pass validation, drop answers the teacher was unsure about, remove duplicates, and have a person check a sample. (4) Fine-tune the student on the kept pairs (Topic 13.5). (5) Evaluate the student on held-out real cases against your gold labels, not against the teacher alone. The first example builds steps 1 to 3 and a toy student for step 4 and 5.
3
Where the data comes from. Data made by a model is called synthetic data. The classic form of distillation uses the teacher's probabilities over tokens; with hosted APIs you usually only get the text, so the student is trained on the text outputs. Both save work, but text-only labels cannot show the student how sure the teacher was, so keep any confidence you can compute (the fake teacher in the example returns one) and use it to filter.
4
Limits. The student cannot be better than its labels on the task, so measure the teacher on your gold set before you trust it as a labeller. Its mistakes are copied and, because they are consistent, can be harder to notice. The student is narrow: it handles inputs like the training data and can fail badly on others, as the fifth test line in the example shows, where the student has seen no word that points to a bug and falls back on common words. Keep real inputs flowing back into the data set (Topic 11.6).
5
Licence and terms. This is a legal question as well as a technical one. A provider's terms of use may limit or forbid using their model's outputs to train another model, and an open-weights licence may attach conditions to outputs or to models derived from it. Read the terms for the exact teacher and the exact student base, keep a record of what you used, and ask someone qualified if the use is commercial. Do not assume that because an output is easy to copy, it is free to train on.
6
Data safety. Real inputs can contain personal data. Redact before sending text to a teacher and before storing it in a data set (Topic 10.5), because trained models can repeat what they saw. Store the data set like any sensitive asset: access control, retention, and a way to delete a person's data.

Explain it without notes

What is distillation, and when is it worth doing?

Why filter the teacher's labels instead of using all of them?

What are the legal and privacy checks before distilling?

Practice

Write keep_confident(rows, minimum) where each row is (text, label, confidence); it returns the rows with confidence at least minimum. Print the texts kept for minimum 80 from three rows.

Write agreement(pairs) that takes a list of (student_label, expected_label) and returns the percentage that match as a whole number (100 * matches // total), and print it for 3 matches out of 5.

Trade-offs

↔
A student is cheap and fast to run but narrow and capped by its labels.
↔
A stricter filter gives cleaner data and less of it; a looser one gives more data and more noise.
↔
Using a strong teacher raises label quality but also raises the one-off cost and the terms-of-use questions.

Done when you can

Done when you can describe the teacher, filter, student, evaluation pipeline.
Done when you can filter labels by confidence and write the training file as compact JSON lines.
Done when you can explain why the student is capped by the teacher and its data.
Done when you can name the terms and privacy checks needed before distilling.

Command Palette

Words you'll meet

Step by step

01Step 1: choose a narrow, high-volume task

02Step 2: let the teacher label real inputs

03Step 3: filter before you train

04Step 4: train a student

05Step 5: test on cases the student has not seen

06Step 6: check the economics and the terms

Try it yourself

Change the threshold

Teach the student something new

Test an unseen label

Code & diagrams

Break it on purpose

Keep every label

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can