Topic 1.7
How Models Are Made
In one line
A chat model is built in stages: pretraining on huge amounts of text to predict the next token, instruction tuning on examples of helpful answers, and preference tuning on human judgements. Each stage explains a behaviour you will meet, from knowledge cutoffs to refusals and flattery.
Think of it like this
Raising a very well-read assistant. First they read a whole library and absorb how language works (pretraining). Then a tutor shows them thousands of examples of how to answer a customer politely and usefully (instruction tuning). Finally, managers rate pairs of their answers, "this one is better", and the assistant adjusts toward what people prefer (preference tuning).
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Parameters / weights
- The billions of numbers inside a model that training adjusts.
- Pretraining
- The first, huge stage: learn to predict the next token on a very large pile of text.
- Base model
- A model after pretraining only. It continues text and does not follow instructions well.
- Instruction tuning
- Further training on examples of instructions with good answers. Also called supervised fine-tuning.
- RLHF
- Reinforcement learning from human feedback: train toward the replies people preferred.
- Knowledge cutoff
- The date after which the model has seen no training text.
- Open weights
- The model's parameter files are published, so you can run and adapt it yourself.
- Sycophancy
- The habit of agreeing with the user or flattering them, even when they are wrong.
Step by step
01Step 1: start with a base model
The example's first line shows a toy base model. After a question mark, web text mostly starts another question, so What has the largest share. A base model is a very good text continuer, not an assistant.
02Step 2: show it good answers
Instruction tuning adds examples where a question is followed by an answer. In the example 16 such examples are added, and Answer: jumps to the top with about two thirds of the probability. The model did not learn new facts; it learned a habit.
03Step 3: nudge toward what people prefer
Preference tuning pushes the chosen style up and the rejected one down. In the toy, Answer: rises and Why drops to zero. Real preference tuning is more subtle, but the effect is the same: a shift in habits learned from judgements.
04Step 4: read behaviours as training effects
The model follows your format because of stage 2. It refuses some topics and apologises often because of stage 3. It does not know yesterday's news because of the cutoff. When it flatters you, that is a side effect of rewarding replies that raters liked.
05Step 5: decide what to do about it
Missing or new facts: put them in the prompt. Style, format or domain habits: prompt first, examples next, fine-tuning later. A behaviour you cannot prompt away (a refusal on a legitimate task, say): test another model, since providers tune differently.
Try it yourself
- 1
Predict the shift
Before running, calculate the probability of
Answer:after tuning with 16 examples: 16 / (4 + 3 + 1 + 16) = 0.667. Then run and check. - 2
Fewer examples
Change 16 to 4. How much does
Answer:gain? A small amount of tuning data changes habits less, which is the trade-off of cheap tuning. - 3
Teach the model a new habit
Add a stage where
Sorry(never seen before) is chosen 8 times by raters. Where does it rank? Preference data decides which habits get reinforced, good or bad.
Code & diagrams
A toy with counts instead of billions of parameters, but it shows the shape: later stages change habits, not the whole model.
Expected output
base model after a question mark: What=0.500 How=0.375 Why=0.125
after instruction tuning : Answer:=0.667 What=0.167 How=0.125 Why=0.042
after preference tuning : Answer:=0.774 What=0.129 How=0.097 Why=0.000
same program, same words, different habitsBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Tune far too hard
In the Python version change range(16) to range(1000) for the instruction-tuning loop.
Myth vs fact
Myth
The model looks answers up in a database of what it read.
Fact
Training compresses text into parameters. It does not store the documents and cannot cite where it learned something.
Myth
A model is always up to date.
Fact
It knows nothing after its cutoff date unless you provide the information in the prompt.
Myth
Open weights means anyone can see my data or the model is free of rules.
Fact
It means the parameter files are available. You still need hardware to run them, and the licence can restrict use.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Training data choices shape the model: the language mix changes tokens per word and quality per language, the code share changes coding strength, and filtering decisions change what it knows. Providers publish only part of this.
- ▸
Preference-trained models are optimised for rater approval, which can differ from truth. If you ask for a harsh review, test whether it really criticises, and design prompts that ask for evidence and counter-arguments (Phase 3).
- ▸
Model versions are snapshots. A provider may update an alias, so a name such as
-latestcan change behaviour without any change on your side. Pin a dated version in production and re-run your evaluation set before moving (Topic 3.8, Phase 9).
Remember this
- 1
Why you need to know this. Many surprises have a cause in training: why the model does not know last week's news, why it follows instructions, why it refuses some requests, why it sometimes agrees with you too much, and why two models from the same company behave differently. Knowing the stage that produced a behaviour tells you whether you can fix it with a prompt, with your own data, or only by choosing another model.
- 2
Stage 1: pretraining. The model is shown enormous amounts of text (web pages, books, code) and trained on one job: predict the next token. Each wrong guess changes its billions of parameters (the numbers inside, also called weights) a tiny bit. This takes thousands of specialised chips for weeks and costs millions of dollars, so only a few organisations do it. The result is a base model: it continues any text, but it is not a helpful assistant. Ask a base model a question and it may continue with more questions, like the first stage in the example.
- 3
Stage 2: instruction tuning (supervised fine-tuning). The base model is trained further on a much smaller set of examples written as (instruction, good answer), plus the chat format with roles. Now "What is the capital of France?" is followed by an answer, not another question. Behaviours such as formatting, tone and following a system prompt are mostly learned here. It is cheap compared to pretraining, and it is the same idea you can apply to your own data (Phase 13).
- 4
Stage 3: preference tuning. People (or other models guided by written rules) compare two replies to the same prompt and pick the better one. The model is trained to produce replies like the preferred ones, using methods usually called RLHF (reinforcement learning from human feedback) or simpler relatives such as DPO (direct preference optimisation). This shapes helpfulness, safety and style: it teaches refusing harmful requests, but also side effects. Replies that raters liked get rewarded, which can push models toward agreeing with the user (sycophancy), over-long answers and over-confident tones.
- 5
Knowledge cutoff. The model only learned from text up to a date. After that it knows nothing, and may still answer as if it did. Anything newer, or private to your company, must come from the prompt (Phase 8). The cutoff is also fuzzy: recent months are thin in the training data, so the model is least reliable just before its cutoff.
- 6
Open weights and closed models. With a hosted model you only send requests. With open weights the parameter files are published, so you can run the model yourself, fine-tune it and keep data in your own network, at the price of operating it. Open weights are not the same as open source: the training data and code often are not shared, and the licence may have conditions. Phase 13 compares the options.
Explain it without notes
Describe the three main stages of making a chat model and what each adds.
Why does a model not know about last week's events, and what do you do about it?
What is the difference between a base model and a chat model, and why does it matter to you?
What does open weights mean, and what does it not mean?
Practice
Write probs(counts) that returns each word's share, rounded to 3 decimals, as a dictionary. Use {"What": 4, "How": 3, "Why": 1}.
Write tune(counts, word, n) that returns a new dictionary with n added to word. Show the probabilities before and after adding 16 to Answer: (starting from 0).
Trade-offs
- ↔
Hosted models are strong and simple to use, and send your data to a provider. Open weights give control and need hardware and skills.
- ↔
Strong preference tuning makes models safer and more polite, and can make them refuse too much or flatter.
- ↔
Fine-tuning fixes a habit cheaply and risks making the model worse at everything else.
Done when you can
Done when you can name the three stages and what each one adds.
Done when you can explain the knowledge cutoff and how to work around it.
Done when you can explain base versus chat models.
Done when you can explain what open weights means and does not mean.