Topic 12.3
More Than One Vector per Passage
In one line
One vector per passage is a compromise. A passage can have several: one per sentence, one per question it answers, or one per token with late interaction. Each helps a different failure and costs more storage.
Think of it like this
A book in a library has one card in the catalogue under its title, but it can also have cards under the author, the subject and the series. A reader looking from any of those angles finds the same book. Several vectors per passage are several cards for the same thing.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Multi-vector
- Several vectors that all point at the same passage.
- Hypothetical question
- A question a model writes at ingestion time that the passage would answer. It is embedded and linked to the passage.
- HyDE
- Hypothetical Document Embeddings (Gao et al., 2022): at query time, a model writes a guessed answer and that text is embedded instead of the question.
- Late interaction
- Keep a vector per token and combine query and passage vectors only at scoring time.
- ColBERT
- A retrieval model (Khattab and Zaharia, 2020) that uses late interaction.
- MaxSim
- For each query token, the best similarity to any passage token. The scores are added up.
- Pooling
- Averaging many token vectors into one vector.
Step by step
01Step 1: a question that misses the passage
"I forgot my login, how do I get back in?" is far, in words, from the password passage. One vector for the passage may or may not catch it. We want a second way in.
02Step 2: write questions for each passage
For every passage keep a few questions it answers. Here they are written by hand; a model would write them at ingestion. Each question is a key that points to the passage.
PASSAGES = [
("D3", "To reset your password, open the app ...",
["I forgot my password, how do I get back in?", "How long does the reset link work?"]),
]03Step 3: score by the best key
Compare the user's question with the passage text and with every written question, and keep the highest score per passage. The example uses Jaccard (shared words divided by all words) so long keys do not win by length.
04Step 4: read the numbers
For D3 the whole passage scores 0.037 and the best written question 0.800. For D1 (refunds) the best written question scores only 0.308. The extra key turned a miss into a clear win.
05Step 5: now by tokens, with MaxSim
The second example keeps a vector per token. For each query token it finds the best passage token, and adds those up. Compare with one pooled (mean) vector, which blurs the two query tokens into one.
06Step 6: count the cost
Multiply the vectors: passages times vectors per passage times dimensions times bytes. Decide per vector type whether the failure it fixes is worth that. Practice question 2 does this arithmetic.
Try it yourself
- 1
Change the question
In the first example, ask
"Can I get a refund?"and"What does the pass cost?". Which passage's best key wins and which key is it? - 2
Make MaxSim fail
In the MaxSim example, add a passage D with tokens
[(1.0, 0.0), (1.0, 0.0), (0.0, 1.0)]. Predict its MaxSim and its pooled score first, then run. - 3
Compare in the lab
In the lab, note the Cosine of the written question (about 0.55) and of the password passage (about 0.40) for the same question. Try a harder question such as
"my account is locked".
Code & diagrams
Jaccard on word sets stands in for a real similarity. The written questions are hand-made stand-ins for a model's output. The point is the rule: score a passage by its best key.
Expected output
Q: I forgot my login, how do I get back in?
D1 whole passage 0.074 | best question key 0.308
key: Can I get my money back if I cancel?
D3 whole passage 0.037 | best question key 0.800
key: I forgot my password, how do I get back in?A toy with literal vectors to show the arithmetic. Real late-interaction models produce one vector per token with many more numbers, and compress them.
Expected output
A: one pooled vector 0.500 | MaxSim 2.000
B: one pooled vector 0.700 | MaxSim 1.400
C: one pooled vector 0.500 | MaxSim 1.000Try: Press Search. The question "How do I get back into my account?" (written as if a model had generated it for the password passage) scores higher than the password passage itself for "I can't sign in". Write your own question key for the monthly pass passage and test a question that did not match before.
Goals
- The written question ranks first
- The password passage is in the top two
Press Search. Then change one thing (a word in the question, a passage, the ranker) and compare the scores.
Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Count shared words instead of using Jaccard
In the Python tab of the first example, replace return len(a & b) / len(a | b) with return float(len(a & b)). Run.
Break #2
Add the token matches instead of taking the best
In the Python tab of the MaxSim example, replace max(dot(q, d) for d in doc) with sum(dot(q, d) for d in doc). Run.
Myth vs fact
Myth
More vectors always give better retrieval.
Fact
They give a better chance of matching, and a bigger bill in storage, search time and updates. Add them for a measured failure.
Myth
Late interaction is just a bigger embedding.
Fact
It keeps a vector per token and defers the comparison until scoring. A pooled vector throws the token detail away first.
Myth
Model-written questions are as good as real user questions.
Fact
They are a guess at how users will ask. Real logged questions (Topic 9.8) are better keys when you have them.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Keep a vector's type (text, sentence, question, token) as metadata. In evaluation you can then see which kind of vector produced each hit and drop the ones that never help.
- ▸
When several vectors belong to one passage, de-duplicate hits by passage id before the top-k cut. Otherwise a passage with eight vectors can fill the whole top-k alone.
- ▸
Late interaction is usually a re-ranking step on a short list from a cheaper first stage, because scoring every token of every passage is expensive. It sits in the same place as a cross-encoder (Topic 6.4) with a different cost profile.
Remember this
- 1
Why this exists. One vector per passage must stand for everything the passage says and for every way a user might ask about it. A passage about password resets sits near "reset password" but may sit far from "I can't sign in". Giving a passage more than one vector, each pointing at it from a different angle, raises the chance that a question lands near one of them. Every vector keeps the id of its passage, and a hit on any vector is a hit on the passage.
- 2
Idea one: one vector per sentence. A long passage blurs its sentences together (the same dilution as in Topic 12.1). Embed each sentence separately, search the sentences, and score the passage by its best sentence. This is the same "index small" idea as parent-child; the difference is that here the passage is both what is returned and the thing the sentences point back to.
- 3
Idea two: vectors for questions the passage answers. At ingestion, ask a model for a few questions this passage answers, and embed those questions too. A user question is then compared with other questions, which look like it, not only with a document written in different words. This is the index-time cousin of HyDE (Topic 7.3), where a model guesses an answer at query time and the guess is embedded instead of the question (Gao et al., 2022). Cost: a model call per passage, and the generated questions can be wrong or off-topic.
- 4
Idea three: late interaction, one vector per token. Instead of squeezing a text into one vector, keep a small vector for every token. ColBERT (Khattab and Zaharia, 2020) does this: the query and the passage are encoded separately, and the score is computed afterwards from their token vectors. For each query token you take its best match among the passage tokens (its MaxSim), and add these up. A passage scores high when each query word has a good partner somewhere in it, even if the passage as a whole is a poor average.
- 5
By hand. The example uses 2-number vectors. Passage A has one token for each query token, passage B has two blurry tokens that are a bit of both, and C covers only one query token. One pooled vector (the mean) cannot tell A from C and prefers B (0.500, 0.700, 0.500). MaxSim ranks A, B, C in the order of how well each query token is covered (2.000, 1.400, 1.000).
- 6
What it costs. Vectors multiply. A passage of 200 tokens with a vector of 128 numbers per token stores 200 times more numbers than a single vector of the same width, which is why late-interaction systems compress their vectors heavily and need special indexes (Topic 3.5 covers the idea of compression). Sentence and question vectors cost less, but a corpus of 1 million passages with 8 vectors each is 8 million vectors to store, search and update (Topic 5.7). Use extra vectors where a measured failure justifies it.
Explain it without notes
Why might a passage need more than one vector?
What are hypothetical questions as keys, and how do they differ from HyDE?
Explain MaxSim and why a pooled vector can rank passages differently.
What are the cost trade-offs of the three multi-vector ideas?
Practice
Write best_key_score(question, keys) that returns the highest Jaccard similarity between the question's word set and the word set of any key (a key is a passage or a question written for it). Print it with 3 decimals for the question "How do I get my money back?" against the keys of passage D1 in Topic 12.3.
A passage of 200 tokens is stored in two ways: one vector per token with 128 numbers of 4 bytes each, or one vector of 384 numbers of 4 bytes. Write the byte counts and print how many times larger the per-token storage is, to one decimal. These sizes are examples for the arithmetic, not those of any named model.
Trade-offs
- ↔
Sentence vectors: sharp matches, more vectors, and a risk of losing the context of the sentence.
- ↔
Question vectors: bridge vocabulary gaps for known passages, but add a model call at ingestion and may drift from real user wording.
- ↔
Token vectors: the finest matching, at a much higher storage and compute cost.
Done when you can
Done when you can explain why a hit on any vector is a hit on the passage and how to avoid one passage filling the top-k.
Done when you can compute MaxSim and a pooled score by hand for two small vectors.
Done when you can name the cost of each of the three ideas.
Done when you can say how hypothetical question keys differ from HyDE (Topic 7.3).