PHASE 0Beginner ~21 min· topic 2 of 7

Topic 0.2

Next-Token Prediction, Built by Hand

In one line

You build a tiny language model from word counts, turn the counts into probabilities, and generate text one token at a time, first by always picking the top word (it loops) and then by sampling (it varies). Real LLMs run the same loop with a far better predictor.

Think of it like this

A child who has heard only three bedtime stories. Ask them to tell a story and they stitch together bits they've heard: "the cat sat on the rug", "the dog ate the fish". Each next word is chosen because it often came after the previous one. That is a language model, in miniature.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Token: One small piece of text a model reads or writes. In this toy, one word; in real models, often part of a word.
Bigram model: A language model that predicts the next word from only the one word before it.
Corpus: The collection of text a model learns from.
Greedy decoding: Always picking the single most likely next token.
Sampling: Picking the next token at random, weighted by the probabilities.
Seed: A starting number for a random number generator. The same seed gives the same "random" numbers every run.
Stop condition: The rule that ends generation: an end token or a maximum number of tokens.

Step by step

01Step 1: split the text into tokens

Our corpus is three tiny sentences. Splitting on spaces gives 20 tokens, with . as its own token so the model can learn where sentences end.

tokens.pywhole filepython

corpus = "the cat sat on the mat . the cat ate the fish . the dog sat on the rug ."
words = corpus.split()
print(len(words), words[:6])
# 20 ['the', 'cat', 'sat', 'on', 'the', 'mat']

02Step 2: count pairs

Walk through neighbouring pairs (the→cat, cat→sat, sat→on…) and count each one in a table of tables: table[current][next] += 1. This table is the model. Training a real LLM is the same idea in spirit: adjust numbers so that the words that really came next get higher scores.

03Step 3: turn counts into probabilities

For one word, divide each count by the total. After sat we saw only on, twice, so on gets probability 1.00: the model is certain. After cat we saw sat and ate once each: 0.50 each, a coin flip. The first runnable example prints these tables.

04Step 4: greedy generation, and why it loops

Greedy picks the top word each time. From the: cat (2 of 6). From cat: ate and sat tie at 1 each; our rule breaks ties alphabetically, so ate. From ate: the. And we are back to the, which leads to cat again. The output repeats until the length limit stops it. Real models also fall into repetition under greedy decoding, which is one reason APIs sample by default.

Step 4: greedy generation, and why it loopsdiagram

Rendering diagram…

05Step 5: sampling with a seeded random number

To sample, draw a random number r between 0 and 1, walk through the candidate words adding up their probabilities, and pick the first word where the running total passes r. With r = 0.40 after the (candidates in alphabetical order: cat 0.33, dog 0.17, fish 0.17…): the total passes 0.40 at dog. We use a tiny seeded generator (state = state * 48271 % 2147483647) so that Python, Java and JavaScript draw exactly the same numbers and print the same sentences. Real APIs also accept a seed on some providers, but don't promise identical output (Topic 1.6).

sample.pywhole filepython

r = 0.40
running = 0.0
for word, p in [("cat", 2/6), ("dog", 1/6), ("fish", 1/6), ("mat", 1/6), ("rug", 1/6)]:
    running += p
    if r < running:
        print("picked", word)   # picked dog
        break

06Step 6: what real models add

Three things separate our toy from a real LLM. Context: it looks at thousands of previous tokens, not one, so it can keep a topic and follow instructions. Generalisation: it can give sensible probabilities for text it never saw exactly, because its weights capture patterns, not just counts. Scale: trillions of training tokens. The loop (predict, pick, append, repeat) is unchanged, and so are the behaviours you just saw: repetition, variation, fluent nonsense and the need for a stop rule.

Try it yourself

1
Predict the greedy loop
Before running the second example, write down the greedy sentence by hand from the counts in the first example. Then run it and compare.
2
Change the seed
Set state = 7 and run again. The sampled sentences change, but every word pair still appears somewhere in the corpus. Can you find one generated sentence that is fluent but never appeared in the corpus?
3
Grow the corpus
Add the sentence the dog ate the bone . to the corpus in both examples. How do the probabilities after the and dog change? Does greedy still loop the same way?

Code & diagrams

Count the next words and turn counts into probabilities New tab

This table is the whole model. A real LLM computes the same kind of table with billions of weights.

Expected output

After "the" (6 times):
  cat   2/6 = 0.33
  dog   1/6 = 0.17
  fish  1/6 = 0.17
  mat   1/6 = 0.17
  rug   1/6 = 0.17
After "cat" (2 times):
  ate   1/2 = 0.50
  sat   1/2 = 0.50
After "sat" (2 times):
  on    2/2 = 1.00

Generate text: greedy loops, sampling varies New tab

Seeded random numbers make the three languages pick the same words. Change the seed to see other sentences.

Expected output

greedy:  the cat ate the cat ate the cat ate
sampled: the cat sat on the fish .
sampled: the fish .
sampled: the cat ate the fish .

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Remove the stop condition

In generate, remove the length limit (keep only the . check) and run the greedy line.

terminal

$ Run

── what you'll see ──

The run never finishes. After the time limit the Output panel says: Stopped after … seconds. Look for a loop that never ends.

Myth vs fact

Myth

Models write the whole answer at once.

Fact

They write one token at a time, each chosen from probabilities computed on everything so far. That is why replies stream in word by word.

Myth

Random output means the model is broken.

Fact

Variation is designed in: APIs sample by default because pure greedy output is dull and repetitive. You control how much with temperature (Topic 1.6).

Myth

If every word pair is real, the sentence is true.

Fact

"The dog sat on the mat" is built from real pairs and was never said. Local fluency says nothing about truth; that is hallucination in miniature.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Our bigram model is an n-gram model with n = 2. Before neural networks, n-gram models with smoothing powered speech recognition and phone keyboards. They fail on long-range context ("The keys to the cabinet are…" needs to remember keys), which is the problem attention solves (Topic 1.4).
▸
Each generated token costs one full pass through the network, and later tokens see all earlier ones. That is why output tokens are slower and usually priced higher than input tokens, and why latency grows with reply length (Topic 6.3).
▸
Inverse-CDF sampling (walk the running total until it passes r) is exactly what real samplers do after temperature and top-p have reshaped the probabilities. Sorting candidates first matters only for reproducibility; real samplers use the model's vocabulary order.

Remember this

1
Why build one by hand? Because every behaviour you will fight later in this course is already visible in a 30-line model: answers that repeat themselves, answers that change between runs, sentences that sound right but are nonsense, and a loop that needs a rule to stop. Seeing them in code you understand makes the real thing far less mysterious.
2
Tokens. Models don't read whole sentences at once; they read small pieces called tokens. In our toy a token is a word (and . is its own token). Real models use pieces of words (un, believ, able), which Topic 1.1 explains. The model's whole job is: given the tokens so far, give a probability for each possible next token.
3
From counts to probabilities. Our toy is a bigram model: it looks only at the previous one word. We count, in a small text, how often each word follows each other word. After the, we saw cat 2 times and mat, fish, dog, rug once each: 6 in total. Dividing each count by the total gives probabilities: cat 2/6 = 0.33, the others 1/6 = 0.17. They add up to 1. A real LLM computes these probabilities with billions of weights and looks at thousands of previous tokens, but the output has the same shape.
4
Generating = predict, pick, append, repeat. Start with the. Ask for the next-word probabilities, pick one, append it, and ask again with the new last word. Stop at a stop condition: here the . token or a maximum length. Real APIs have both: the model emits a special end token, or you set max_tokens and the reply is cut off. The reply then reports *why* it stopped (finish_reason, Topic 2.3).
5
Greedy vs sampling. Greedy decoding always picks the most likely token. It is predictable, but it can loop: our toy writes "the cat ate the cat ate the cat…" forever, because after the the top word is always cat. Sampling picks randomly in proportion to the probabilities, so a 0.17 word is picked about 17% of the time. The text varies and loops are rarer. This is the root of why the same prompt gives different answers, and of the temperature setting (Topic 1.6).
6
Fluent is not true. Sampling can produce "the dog sat on the mat": every pair of words appeared in the training text, so it reads naturally, but no story said that. The toy has no idea what is true; it only knows what tends to follow what. Scale this up and you get hallucination: plausible text, built from patterns, that nobody ever checked.

Explain it without notes

Walk through how a bigram model generates a sentence, from training text to output.

Why does greedy decoding tend to repeat itself?

What does a seed do, and why don't real APIs fully guarantee identical outputs even with one?

Practice

Change the toy into a model that also counts how often each word starts a sentence (after . or at the beginning), and print the probabilities for the first word.

Write top_k_next(table, word, k) that returns the k most likely next words as a list, ties broken alphabetically.

Trade-offs

↔
Greedy: repeatable and focused, but repetitive. Sampling: varied and natural, but less predictable.
↔
Looking at more previous words gives better predictions, but needs far more data and computation. That cost is what real model architectures are built to manage.

Done when you can

Done when you can build a bigram table from text and compute next-word probabilities by hand.
Done when you can explain greedy decoding, sampling and why greedy loops.
Done when you can explain why a stop condition (end token or max length) is required.
Done when you can explain why a fluent sentence can be false.

Command Palette

Words you'll meet

Step by step

01Step 1: split the text into tokens

02Step 2: count pairs

03Step 3: turn counts into probabilities

04Step 4: greedy generation, and why it loops

05Step 5: sampling with a seeded random number

06Step 6: what real models add

Try it yourself

Predict the greedy loop

Change the seed

Grow the corpus

Code & diagrams

Break it on purpose

Remove the stop condition

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can