Three stages shift the next-word probabilities
A toy with counts instead of billions of parameters, but it shows the shape: later stages change habits, not the whole model.
A chat model is built in stages: pretraining on huge amounts of text to predict the next token, instruction tuning on examples of helpful answers, and preference tuning on human judgements. Each stage explains a behaviour you will meet, from knowledge cutoffs to refusals and flattery.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write probs(counts) that returns each word's share, rounded to 3 decimals, as a dictionary. Use {"What": 4, "How": 3, "Why": 1}.
Write tune(counts, word, n) that returns a new dictionary with n added to word. Show the probabilities before and after adding 16 to Answer: (starting from 0).
Explain it without notes
Describe the three main stages of making a chat model and what each adds.
Why does a model not know about last week's events, and what do you do about it?
What is the difference between a base model and a chat model, and why does it matter to you?
What does open weights mean, and what does it not mean?
Expected output
base model after a question mark: What=0.500 How=0.375 Why=0.125
after instruction tuning : Answer:=0.667 What=0.167 How=0.125 Why=0.042
after preference tuning : Answer:=0.774 What=0.129 How=0.097 Why=0.000
same program, same words, different habits