Count the next words and turn counts into probabilities
This table is the whole model. A real LLM computes the same kind of table with billions of weights.
You build a tiny language model from word counts, turn the counts into probabilities, and generate text one token at a time, first by always picking the top word (it loops) and then by sampling (it varies). Real LLMs run the same loop with a far better predictor.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Change the toy into a model that also counts how often each word starts a sentence (after . or at the beginning), and print the probabilities for the first word.
Write top_k_next(table, word, k) that returns the k most likely next words as a list, ties broken alphabetically.
Explain it without notes
Walk through how a bigram model generates a sentence, from training text to output.
Why does greedy decoding tend to repeat itself?
What does a seed do, and why don't real APIs fully guarantee identical outputs even with one?
Expected output
After "the" (6 times):
cat 2/6 = 0.33
dog 1/6 = 0.17
fish 1/6 = 0.17
mat 1/6 = 0.17
rug 1/6 = 0.17
After "cat" (2 times):
ate 1/2 = 0.50
sat 1/2 = 0.50
After "sat" (2 times):
on 2/2 = 1.00