Generate text: greedy loops, sampling varies
Seeded random numbers make the three languages pick the same words. Change the seed to see other sentences.
You build a tiny language model from word counts, turn the counts into probabilities, and generate text one token at a time, first by always picking the top word (it loops) and then by sampling (it varies). Real LLMs run the same loop with a far better predictor.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Change the toy into a model that also counts how often each word starts a sentence (after . or at the beginning), and print the probabilities for the first word.
Write top_k_next(table, word, k) that returns the k most likely next words as a list, ties broken alphabetically.
Explain it without notes
Walk through how a bigram model generates a sentence, from training text to output.
Why does greedy decoding tend to repeat itself?
What does a seed do, and why don't real APIs fully guarantee identical outputs even with one?
Expected output
greedy: the cat ate the cat ate the cat ate
sampled: the cat sat on the fish .
sampled: the fish .
sampled: the cat ate the fish .