Topic 2.3
Normalising Vectors
In one line
Normalising a vector means scaling it to length 1 without changing its direction. Then the dot product equals cosine similarity, which is why most stores assume unit vectors. Mixing normalised and raw vectors silently corrupts a ranking.
Think of it like this
Comparing recipes by the *proportions* of their ingredients, not the batch size. A cake for 4 people and a cake for 40 have the same recipe. Normalising a vector is the same: you keep the proportions (the direction) and throw away the size (the length), so two vectors can be compared fairly.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Normalise
- Scale a vector so its length is 1, keeping its direction.
- L2 norm (length)
- The square root of the sum of the squares of the numbers. The 'L2' names this particular kind of length.
- Unit vector
- A vector with length 1.
- Tolerance
- How far from 1 a length may be and still count as 'about 1', because decimals are rounded. Something like 0.001.
- NaN
- 'Not a number', the result of an impossible calculation such as 0 divided by 0. It spreads through every later calculation.
Step by step
01Step 1: measure the lengths
A is [0.6, 0.8, 0.0]: sqrt(0.36 + 0.64) = 1.000. B is [3.0, 3.0, 0.0]: sqrt(9 + 9) = 4.243. A is a unit vector. B is not.
def length(v):
return math.sqrt(sum(x * x for x in v))
def normalise(v):
n = length(v)
return [x / n for x in v]02Step 2: rank with the dot product on mixed vectors
The query is [0, 1, 0], a pure 'second direction'. A scores 0.800 and B scores 3.000, so B is first. B only wins because it is long. By direction, A is more aligned with the query.
03Step 3: normalise, rank again
Scale B to [0.707, 0.707, 0.0]. Now A scores 0.800 and B 0.707: A is first, as it should be.
04Step 4: add a check that fails loudly
require_unit(v) raises an error when the length is not within 0.001 of 1. The example runs it on A (ok), B (rejected), and B normalised (ok).
05Step 5: put the check where vectors enter
Run the check when vectors are created and when they are loaded from storage, not inside the search loop. Then a bad vector is caught once, at the door, and every vector inside is known to be good.
06Step 6: normalise the query too
The query vector must follow the same rule as the documents. A normalised index with a raw query still ranks correctly (the query's length scales every score equally), but scores are no longer cosine, which breaks any threshold you set.
Try it yourself
- 1
Change the query
Set the query to
[1.0, 0.0, 0.0]. Before running, predict which document wins on the mixed vectors and which on the normalised ones. Does the bug still show? - 2
Add a zero vector
Add
"C": [0.0, 0.0, 0.0]to the raw vectors and run. What happens innormalise? Add a guard that returns the vector unchanged when its length is 0. - 3
Loosen the tolerance
Call
require_unit(v, 5)on B. What does that say about choosing a tolerance? Which value would you use for vectors stored with 3 decimals?
Code & diagrams
The check raises an error and the example catches it and prints the message.
Expected output
lengths: A=1.000, B=4.243
dot product on mixed lengths: B=3.000, A=0.800
dot product after normalising: A=0.800, B=0.707
A: ok
B: rejected, vector has length 4.243, expected 1.000 (normalise it first)
B normalised: okBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Forget to normalise the documents
In the Python tab, replace the line that builds fixed with fixed = dict(RAW) (skip the normalising). Run.
Myth vs fact
Myth
Normalising throws away information.
Fact
It throws away length only. The direction, which carries the meaning for cosine-trained models, is untouched.
Myth
If the API returns vectors, they are already normalised.
Fact
Some are, some are not, and it can depend on a flag. Compute the length of a few and check.
Myth
A wrong ranking would show up as an error.
Fact
No. Mixed lengths give a perfectly valid-looking ranking that is quietly wrong, which is why you add the check.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Floating point makes 'exactly 1' impossible, and stores that save vectors as 16-bit or 8-bit numbers (Topic 2.5) make lengths drift further. Use a tolerance, and re-normalise after any lossy step such as shortening.
- ▸
Normalise once at write time, not at every query. For an index of a million vectors, one pass of dividing by the length is cheap, while normalising on every query is repeated work in the hot path.
- ▸
Keep the original length if you might need it later, for example to undo the scaling or to debug a model that does not return unit vectors. It costs one number per vector.
Remember this
- 1
Why this exists. The fastest similarity a store can compute is the dot product, but the dot product grows with length (Topic 2.2). If every vector has length 1, the dot product *is* the cosine, so the fast operation gives the right answer. Normalising once, when a vector is created, buys cheap and correct comparison forever after.
- 2
How to do it. Compute the length,
sqrt(sum of squares), and divide every number by it.[3, 3, 0]has length 4.243, so it becomes[0.707, 0.707, 0.0]. The direction is unchanged and the length is now 1. A vector of all zeros has length 0 and cannot be normalised: check for it, because dividing by 0 givesNaNor an error. - 3
What models give you. Many embedding APIs and libraries return unit vectors, some only if you ask (a flag such as
normalize_embeddings=Truein sentence-transformers ornormalize: truein Transformers.js), and some do not. Do not assume. Compute the length of a few vectors from your model and look. - 4
The bug: mixing. If some vectors in an index are normalised and others are not, the dot product mixes direction with length. In the example, document B
[3.0, 3.0, 0.0]has length 4.243 and A is[0.6, 0.8, 0.0]. For the query[0, 1, 0]the dot product ranks B first (3.000 against 0.800), but by direction A is closer: after normalising, A scores 0.800 and B 0.707. Nothing crashes. The ranking is simply wrong. - 5
Check, do not trust. Add a small check function,
require_unit, that fails loudly when a vector's length is not about 1. Run it on what your embedding step returns and in tests. The example shows its message: 'vector has length 4.243, expected 1.000 (normalise it first)'. A bug found at write time costs minutes, one found by a user costs weeks. - 6
When not to normalise. If a model is trained to use vector length (some are trained for the dot product, not cosine), normalising changes its behaviour. Follow the model's documentation. For the common cosine-trained text models, normalising is safe and expected.
Explain it without notes
Why does normalising make the dot product equal to the cosine similarity?
What is the bug when an index mixes normalised and raw vectors, and why is it hard to notice?
How do you normalise a vector, and what must you handle as a special case?
Where in a pipeline should you check that vectors have length 1, and why there?
Practice
Write normalise(v) that returns the unit vector (and returns a zero vector unchanged), and print normalise([3, 4]) and normalise([0, 0]) to three decimals.
Write ranks_differ(query, docs) that returns True when ranking docs (a dict of name to vector) by raw dot product gives a different order from ranking by dot product after normalising everything. Print the result for the query [0, 1] and docs {"a": [0.6, 0.8], "b": [3, 3]}.
Trade-offs
- ↔
Normalise at write time: cheap queries and correct dot products, at the price of losing the original length.
- ↔
Normalise at query time: no change to stored data, but repeated work and an easy step to forget.
- ↔
A strict check (fail on any non-unit vector): safe, but it must have a sensible tolerance or it fails on harmless rounding.
Done when you can
Done when you can normalise a vector by hand, including the zero case.
Done when you can explain why the dot product equals cosine on unit vectors.
Done when you can show how mixed lengths change a ranking.
Done when you have a length check in your pipeline and a test for it.