Topic 1.6
Sampling: Temperature, Top-k and Top-p
In one line
A model produces a score for every possible next token. Softmax turns scores into probabilities, and a sampler picks one. Temperature, top-k, top-p and the seed control how adventurous that pick is, which is why the same prompt can give different answers.
Think of it like this
A bag of coloured balls. The model fills the bag: many red balls where it is confident, a few blue ones where it is unsure. Sampling is drawing one ball without looking. Greedy always takes the colour with most balls. Temperature changes the mix in the bag, and top-k and top-p remove the rare colours before you draw.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Logit
- The raw score a model gives each possible next token before softmax.
- Softmax
- Turns logits into probabilities that add up to 1.
- Greedy decoding
- Always picking the most likely token.
- Temperature
- A number that divides the logits: low is cautious and repeatable, high is varied and risky.
- Top-k
- Keep only the k most likely tokens, then sample.
- Top-p
- Keep the smallest group of top tokens whose probabilities reach p, then sample. Also called nucleus sampling.
- Seed
- A starting number for the random generator, so draws can be repeated.
Step by step
01Step 1: from scores to a choice
After reading the prompt, the model gives one logit per token. Five tokens in our example: Paris 4.0, London 2.5, Rome 2.0, the 1.0, banana -1.0. A raw score is not a probability, so we need softmax first.
02Step 2: apply temperature
Run the first example. At T=0.5 Paris takes almost everything. At T=1.0 it keeps about 71 percent. At T=2.0 the distribution is flat: banana, with logit -1.0, has a real chance. The ordering never changes; only the spread does.
03Step 3: greedy ignores the spread
Greedy picks Paris at every temperature, because it looks only at the top score. Temperature 0 in an API means exactly this.
04Step 4: cut the tail
The second example sorts by probability. Top-k 3 keeps Paris, London, Rome. Top-p 0.9 adds tokens until the running total reaches 0.9: Paris 0.7067 plus London 0.1577 is 0.8644, still below 0.9, so Rome (total 0.9600) is added and three tokens are kept here too. After the cut, the kept probabilities are divided by their sum, so they add up to 1 again.
05Step 5: draw with a seeded generator
A tiny generator does state = state * 48271 % 2147483647 and then divides state by 2147483647, which gives a number between 0 and 1. Walk the kept tokens adding probabilities until the running total passes that number. Because the seed is fixed, the eight draws are the same on every run and in every language. A real API without a seed would differ each time.
state = 42
state = state * 48271 % 2147483647
print(state / 2147483647)Try it yourself
- 1
Predict T=0.25
Before running the first example, predict what Paris gets at T=0.25 (very close to 1, about 0.997). Add 0.25 to the temperatures and check.
- 2
Change top-p
Set top-p to 0.7. Paris alone already reaches 0.7067, so only one token is kept and its renormalised probability is 1.0000. What do the eight draws become?
- 3
Change the seed
Change the seed from 42 to 7 in all three languages and compare the draws. Same seed, same draws, any language.
- 4
Use the lab
In the lab, set temperature to 0 and send the same request three times. Are the replies identical? Why can they still differ slightly on a real service?
Code & diagrams
Same five logits at three temperatures. Greedy ignores the temperature.
Expected output
T=0.5: Paris=0.9340 London=0.0465 Rome=0.0171 the=0.0023 banana=0.0000
T=1.0: Paris=0.7067 London=0.1577 Rome=0.0956 the=0.0352 banana=0.0048
T=2.0: Paris=0.4661 London=0.2202 Rome=0.1715 the=0.1040 banana=0.0383
greedy pick (always the top score): ParisThe generator is the one used throughout this course, identical in all three languages.
Expected output
top-k 3: Paris=0.7361 London=0.1643 Rome=0.0996
top-p 0.9: Paris=0.7361 London=0.1643 Rome=0.0996
8 draws: Paris Paris Paris Paris Paris Rome Paris ParisTry: Send the request with temperature 0, then three times with temperature 1.5. Compare the four replies: how much do they differ? Then try temperature 0 twice more and notice how close, but not always identical, the replies are.
Goals
- The reply finished on its own
- The reply is short (40 words or fewer)
Sign in to send this request to a real model (free).
Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Temperature zero
In the Python softmax example change the temperatures to (0.0, 1.0) and run.
Myth vs fact
Myth
Temperature controls how correct the model is.
Fact
It controls randomness only. Low temperature repeats the most likely answer, and that answer can still be wrong.
Myth
Temperature 0 gives identical output every time.
Fact
Usually very close, but servers, batching and model updates can still change a token. Do not build logic that needs byte-identical replies.
Myth
Higher temperature makes the model more creative and smarter.
Fact
It makes unlikely tokens likelier. Past a point the text turns into nonsense.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Use one randomness dial at a time. Temperature and top-p both change the spread, and tuning both makes behaviour hard to reason about. Most teams fix top-p and tune temperature.
- ▸
For structured output (JSON, classification, extraction) use temperature 0 to 0.2 and validate the result (Phase 4). For chat, 0.5 to 0.8 is common. For brainstorming, 0.9 to 1.2. Treat these as starting points and test.
- ▸
Reasoning models and some providers fix or restrict sampling settings, so a parameter you set may be ignored or rejected. Read the provider's notes for the model you use, and look at the raw request in the lab.
- ▸
Ranges differ by provider: OpenAI-compatible APIs such as Groq accept temperature from 0 to 2, while Anthropic's Messages API accepts 0 to 1. A value that works on one can be rejected by another, so check the API reference of the provider you call.
- ▸
Other penalties exist (frequency and presence penalties) to discourage repetition. They are blunt tools: they can push a model away from a word it genuinely needs, such as a repeated name in a table.
Remember this
- 1
Why this exists. The model does not output a word. It outputs a logit (a raw score) for every token in its vocabulary. Something must turn those scores into one chosen token, and then the loop repeats (Topic 0.2). That something is the sampler, and its settings are the main dials you have over style, creativity and repeatability.
- 2
Softmax turns logits into probabilities.
p = exp(score) / sum of exp(all scores). All probabilities are positive and add up to 1. With the example logitsParis 4.0, London 2.5, Rome 2.0, the 1.0, banana -1.0, Paris gets about 0.71 at temperature 1. - 3
Greedy versus sampling. Greedy decoding always picks the highest probability token. It is repeatable, a little dull, and can loop in long text. Sampling draws a token at random according to the probabilities, so likely tokens are picked often and unlikely ones sometimes. That randomness is why two runs of the same prompt differ.
- 4
Temperature reshapes the distribution. Divide the logits by the temperature before softmax. Low temperature (0.5) makes the top token even more dominant, high temperature (2.0) flattens the choice so odd tokens become likely. Temperature near 0 behaves like greedy (APIs often accept 0 and treat it as greedy). Temperature does not add knowledge: it only changes how much risk the sampler takes.
- 5
Top-k and top-p cut the tail. Top-k keeps only the k most likely tokens. Top-p (nucleus sampling) keeps the smallest set of top tokens whose probabilities add up to at least p. In both cases the survivors are re-scaled to add up to 1 and then drawn from. They remove the long tail of silly tokens (like
banana) that high temperature would otherwise let through. Use temperature or top-p to control randomness, usually not both at once. - 6
Seeds and why answers still vary. A sampler fed with a random generator draws differently each time. With a fixed seed, the same generator gives the same draws (the example uses a tiny generator and starts at seed 42). Some APIs accept a
seed, but even then results are not guaranteed identical across servers or model versions. If you need exact repeatability, store the answer; if you need consistency, use temperature 0 and validate the output.
Explain it without notes
Explain the path from logits to a chosen token.
What does temperature do, mathematically and in practice?
How do top-k and top-p differ, and why use them?
Why does the same prompt give different answers, and how do you make behaviour more repeatable?
Practice
Write softmax_t(logits, t) and print the probabilities of [2.0, 1.0, 0.0] at T=1 with 4 decimals.
Write top_k_tokens(probs, k) where probs is a dictionary of token to probability, returning the k most likely tokens as a list (ties broken by name). Test it with {"a": 0.5, "b": 0.3, "c": 0.2} and k=2.
Trade-offs
- ↔
Low temperature gives repeatable, safe output and can feel repetitive.
- ↔
High temperature gives variety and raises the chance of nonsense or invented facts.
- ↔
Top-p adapts to the model's confidence, top-k is simpler and fixed, and both reduce variety a little.
Done when you can
Done when you can compute softmax with a temperature by hand.
Done when you can explain greedy versus sampling.
Done when you can explain top-k and top-p and when to use each.
Done when you can choose a starting temperature for a task and test it.