Topic 1.4
Attention in Plain Words
In one line
Attention lets every token look at the other tokens and decide which ones matter for its meaning. Scores become weights with softmax, and the token's new vector is a weighted average of what the others offer. Stacking this many times is the transformer.
Think of it like this
A group chat where you read the newest message, "Did you see it?". Alone it means nothing. You glance back at the earlier messages, give most attention to the one about the football match, and so "it" becomes "the match". Attention does that for every word, at the same time, using numbers.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Attention
- The step where each token mixes in information from other tokens, in proportion to how relevant they are.
- Query
- A vector saying what a token is looking for.
- Key
- A vector saying what a token offers to others.
- Value
- The content a token passes on when other tokens pay attention to it.
- Softmax
- A function that turns any list of scores into positive weights that add up to 1.
- Transformer
- The kind of network built from stacked attention and ordinary layers. Almost all modern LLMs use it.
- Layer / head
- A layer is one round of refinement. A head is one parallel attention pattern inside a layer.
Step by step
01Step 1: the problem of context
The word bank has one id in the vocabulary, so its starting vector is the same in every sentence. But its meaning depends on its neighbours. The model needs a way to let neighbours reshape the vector.
02Step 2: score every neighbour
Take the query of bank and the key of each token in the context (including itself). Dot product, then divide by the square root of the vector size (here sqrt(2)). In the first context the scores are river=1.4142, the=0.2828, bank=1.4142 (the dot products are 2.0, 0.4 and 2.0).
03Step 3: softmax to weights
exp(1.4142) is about 4.11 and exp(0.2828) about 1.33. Dividing each by the total (about 9.55) gives weights near 0.43, 0.14 and 0.43. They add up to 1. Run the example to see exact values.
import math
scores = [1.4142, 0.2828, 1.4142]
es = [math.exp(s) for s in scores]
print([round(e / sum(es), 4) for e in es])04Step 4: mix the values
Multiply each token's value by its weight and add them. With river the first number of the result is bigger, with money the second one is. The new vector of bank now carries the sense that fits its context.
05Step 5: stack it
A real model does this for every token at once, with learned matrices that produce the queries, keys and values, across dozens of layers and many heads. After the last layer the final token's vector is converted into scores for every possible next token. Nothing else is magic: it is this mixing, repeated.
Try it yourself
- 1
Predict the direction
Before running, decide: in the
moneycontext, which of the two output numbers should be larger? Run and check. - 2
Make the word
thematterChange the key of
theto[1.5, 1.5]. What happens to its weight and to the output vector? - 3
Sharper scores
Remove the division by
sqrt(2)in all three languages. The weights become more extreme. Why do real models scale the scores?
Code & diagrams
One query, three keys and values with two numbers each. Real models do this with hundreds of numbers and many heads.
Expected output
context: river the bank
weights: river=0.4306 the=0.1389 bank=0.4306
new vector for bank: [0.7153, 0.2847]
context: money the bank
weights: money=0.4306 the=0.1389 bank=0.4306
new vector for bank: [0.2847, 0.7153]Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Forget softmax
In the Python attend, use the raw scores as weights: replace weights = softmax(scores) with weights = scores.
Myth vs fact
Myth
Attention means the model understands what is important like a person.
Fact
It is arithmetic: dot products, softmax and averages with learned numbers. It often looks sensible because training rewarded predictions that need it.
Myth
The model reads left to right and forgets.
Fact
Within the context window each token can look directly at every earlier token. Forgetting comes from the window's limit and from how well attention is spread over long input (Topic 1.5).
Myth
Attention weights explain why the model answered what it did.
Fact
They show one internal step among many layers and heads. They are a clue, not a proof.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
In generation models each token can only attend to earlier tokens (a causal mask). That is what lets the model be trained to predict the next token and generate one token at a time.
- ▸
Generation stores each token's keys and values in a KV cache, so each new token does not recompute the past. It makes decoding fast and uses lots of memory, which is one reason long contexts and many parallel users are expensive to serve.
- ▸
The quadratic growth of attention is why long-context models use tricks (sparse or sliding windows, grouped heads, better kernels). Even so, cost and latency still rise with prompt length, so trimming prompts pays off.
Remember this
- 1
Why this exists. The meaning of a word depends on the words around it. "Bank" in "river bank" is not "bank" in "money bank", and "it" means whatever noun it refers to. Older approaches read text left to right through a narrow channel and forgot the beginning of long sentences. Attention gives each token direct access to the other tokens in the context (in a text-generating model, to all the tokens before it), so context can change its meaning directly.
- 2
Query, key and value. Each token produces three small vectors. The query says what this token is looking for. The key says what this token has to offer. The value is the content it passes on if chosen. A score for a pair is the dot product of one token's query with the other token's key (Topic 1.3): high when what I want matches what you offer.
- 3
Softmax turns scores into weights. Raw scores can be any numbers. Softmax turns them into positive weights that add up to 1:
weight = exp(score) / sum of exp(all scores). Larger scores get larger weights, and the biggest score wins most of the share but not all of it. The example divides scores by the square root of the vector size, a standard step that keeps scores from growing as vectors get longer. - 4
The new vector is a weighted average. Multiply each token's value by its weight and add them up. A token that got a high weight contributes a lot to the result. In the example, the word
bankis computed twice: next toriverthe result leans toward the river value (first number larger), next tomoneytoward the money value (second number larger). Same word, different context, different vector. That is how context changes meaning. - 5
Layers and heads. A real transformer repeats this many times. Each layer refines every token's vector using the others, and each layer runs several attention heads in parallel, each free to look for a different kind of relation (grammar, reference, position). The last token's final vector is turned into the probabilities for the next token (Topic 1.6). Training adjusts the numbers that make queries, keys and values until predictions are good.
- 6
The cost you will meet. Every token looks at every other token, so the work grows with the square of the length: ten times more tokens means about a hundred times more attention work. That is why long contexts are slower and pricier, and why providers set a maximum context window (Topic 1.5).
Explain it without notes
What problem does attention solve?
Walk through computing attention for one token with three context tokens.
Why is a long prompt slower and more expensive than a short one?
What do layers and heads add?
Practice
Write softmax(xs) and print the weights for [1.0, 2.0, 3.0] with 4 decimals, and their sum.
Write weighted_average(weights, values) for one-number values and print it for weights [0.5, 0.25, 0.25] and values [10, 20, 40].
Trade-offs
- ↔
Attention over all tokens gives rich context but costs work that grows with the square of the length.
- ↔
More layers and heads give more ability and cost more compute and memory per token.
- ↔
A longer context lets you include more information, and more irrelevant text can dilute what matters.
Done when you can
Done when you can explain query, key and value in one sentence each.
Done when you can compute softmax weights and a weighted average by hand.
Done when you can explain why the same word gets different vectors in different contexts.
Done when you can explain why long prompts cost more.