Attention weights for bank in two contexts
One query, three keys and values with two numbers each. Real models do this with hundreds of numbers and many heads.
Attention lets every token look at the other tokens and decide which ones matter for its meaning. Scores become weights with softmax, and the token's new vector is a weighted average of what the others offer. Stacking this many times is the transformer.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 5 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write softmax(xs) and print the weights for [1.0, 2.0, 3.0] with 4 decimals, and their sum.
Write weighted_average(weights, values) for one-number values and print it for weights [0.5, 0.25, 0.25] and values [10, 20, 40].
Explain it without notes
What problem does attention solve?
Walk through computing attention for one token with three context tokens.
Why is a long prompt slower and more expensive than a short one?
What do layers and heads add?
Expected output
context: river the bank
weights: river=0.4306 the=0.1389 bank=0.4306
new vector for bank: [0.7153, 0.2847]
context: money the bank
weights: money=0.4306 the=0.1389 bank=0.4306
new vector for bank: [0.2847, 0.7153]