A distillation data set from a fake teacher
The teacher is a rules function with a made-up confidence, standing in for a large model. The student is a toy word counter.
Distillation teaches a small model from the outputs of a large one. The large model labels many of your real inputs, you keep the good labels, and the small model is trained on them. It is cheap to run afterwards, but it inherits the teacher's mistakes and the terms of use apply.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write keep_confident(rows, minimum) where each row is (text, label, confidence); it returns the rows with confidence at least minimum. Print the texts kept for minimum 80 from three rows.
Write agreement(pairs) that takes a list of (student_label, expected_label) and returns the percentage that match as a whole number (100 * matches // total), and print it for 3 matches out of 5.
Explain it without notes
What is distillation, and when is it worth doing?
Why filter the teacher's labels instead of using all of them?
What are the legal and privacy checks before distilling?
Expected output
teacher labelled 12 prompts, kept 10, skipped 2 as low confidence
skipped: Cancel my subscription now
skipped: Nice product by the way
first two training lines:
{"prompt":"I was charged twice","completion":"billing"}
{"prompt":"Refund for my last order","completion":"billing"}
My card was charged again -> student says billing, expected billing
Checkout crash again -> student says bug, expected bug
Is the app broken today -> student says bug, expected bug
Do you ship to Nepal -> student says other, expected other
Screen goes white when I tap save -> student says billing, expected bug
Please cancel my subscription -> student says billing, expected billing
student agrees with the expected label on 5 of 6