Topic 1.5
Phrases, Fields and Boosts
In one line
Word counts ignore order, structure and exactness. Positions let a search match a phrase, fields let a title count for more than the body, and identifiers such as error codes must be matched exactly or not at all.
Think of it like this
Looking for a person in a phone book. You do not search for "Kumar" anywhere on the page: you look under the surname field, in the right order, and an exact spelling matters. The same letters in a street name are not a match. A good search treats *where* and *in what order* words appear as part of the match.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Position
- The index of a word inside a passage (the first word is 0). Stored so phrases can be matched.
- Phrase query
- A query that must match words next to each other, in order.
- Field
- One named part of a document, such as title or body.
- Boost
- A multiplier that makes matches in one field (or one term) count for more.
- Exact match
- A match on the whole value, character for character, with no stemming or folding.
- Identifier
- A code that names one thing: an error code, order id, part number.
Step by step
01Step 1: remember where words are
Instead of term -> [doc ids], store term -> {doc id -> [positions]}. Number the words of each passage from 0 as you tokenise it.
positions = {} # word -> {doc_id: [positions]}
for doc_id, text in HANDBOOK:
for pos, word in enumerate(tokenize(text)):
positions.setdefault(word, {}).setdefault(doc_id, []).append(pos)02Step 2: check consecutive positions
For a phrase, take each position of its first word and test that word i of the phrase sits at start + i in the same passage. The example prints the start position of each match (D1@18).
03Step 3: compare with AND
The last column of the first example shows the AND result. credit ride gets D1 from AND and nothing from the phrase check, because both words are in D1 but not in that order. Phrases are strictly stronger than AND.
04Step 4: index title and body separately
Tokenise title and text into two lists per passage and compute BM25 for each list against the question. The example prints the best passage from the body alone, from the title alone, and from the combined score.
05Step 5: add with a boost
combined = 3 * title_score + body_score. For late, D6 is first in the body (1.58) and in the title (1.36), and the combined score is 5.65. For damage, the body gives nothing and the title alone carries D5 to 5.35.
06Step 6: an identifier query
The third example adds four support passages and asks for ERR-4012. BM25 ranks E1 (1.18) and E3 (1.01) above E2 (0.36), which is right. The exact-match line agrees: E1 E3.
07Step 7: a typo in the identifier
ERR-4O12 has the letter O. BM25 still returns three passages (E1 0.40, E2 0.36, E3 0.34) as if it understood, but the only shared term is err, so the ranking is noise. The exact-match line says (none), which is the honest answer. Prefer saying "no such code" to guessing.
Try it yourself
- 1
Another phrase
In the first example add
"minutes pass"and"dollars and"to the list. Predict which match before running. Doesdollars andmatch D8? Check the positions by printingat("dollars", "D8")andat("and", "D8"). - 2
Change the boost
In the second example set
TITLE_BOOST = 0and then10. For which queries does the winner change, and for which does it not? What does that tell you about when a title boost matters? - 3
Break the identifier
In the third example try the query
ERR-4107(a code that has no passage in that example) and4012. What does BM25 return and what does the exact-match line say? - 4
Lab
In the lab, type
ERR-4O12with the letter O under Keyword and under Vector. Which ranker is more dangerous for a user who mistyped a code?
Code & diagrams
Four handbook passages (D1, D4, D6, D8). Positions start at 0.
Expected output
positions of 'ride' in D1: 1 18
positions of 'credit' in D1: 19
ride credit phrase: D1@18 all words: D1
credit ride phrase: (none) all words: D1
monthly pass phrase: D8@1 all words: D8
pass monthly phrase: (none) all words: D8
returned late phrase: D6@1 all words: D6
late returned phrase: (none) all words: D6BM25 is computed separately over the titles and over the bodies of six handbook passages (so the numbers differ from the twelve-passage ones in Topic 1.4).
Expected output
helmet body: D4=1.660 title: none combined: D4=1.660
damage body: none title: D5=1.784 combined: D5=5.351
refunds body: none title: D1=1.784 combined: D1=5.351
late body: D6=1.578 title: D6=1.356 combined: D6=5.645
monthly pass body: D8=3.237 title: D8=2.711 combined: D8=11.370Support passages E1 to E3 and H1 are made up for this example. They are not part of the handbook.
Expected output
ERR-4012
BM25: E1=1.182, E3=1.012, E2=0.357
exact match: E1 E3
ERR-4O12
BM25: E1=0.401, E2=0.357, E3=0.344
exact match: (none)Try: Press Search with Rank by set to Vector (it starts there). The passage that really contains ERR-4107 is not first, so the goal fails. Look at the Cosine column, then switch Rank by to Keyword (BM25) and search again. Then type ERR-4021 and see how the two rankers behave.
Goals
- The passage with ERR-4107 ranks first
Press Search. Then change one thing (a word in the question, a passage, the ranker) and compare the scores.
Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
A tokenizer that drops digits
In the Python tab of the third example, change the tokenizer's pattern from r"[a-z0-9]+" to r"[a-z]+" (letters only) and run.
Myth vs fact
Myth
Putting quotes around words is just a style choice.
Fact
Quotes switch the query from "these words anywhere" to "these words next to each other, in order". They need positions in the index and they change the results.
Myth
A boost of 2 or 3 on titles is always right.
Fact
It is a starting point. Titles in some collections are generic ("Introduction") and deserve no boost. Tune it against your questions.
Myth
Keyword search handles codes perfectly.
Fact
It handles correct codes well. A single wrong character in a code yields confident, arbitrary results unless you check for an exact match.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Positions can multiply index size, so engines make them optional per field. A common choice is positions on the body (phrases are wanted) and none on tags and ids.
- ▸
Many engines also support proximity (
ridewithin 3 words ofcredit) and slop, which score near-misses of a phrase. They are useful for names written in different orders ("Kumar Amit" and "Amit Kumar"). - ▸
Store identifiers in a separate field that is not analysed (Elasticsearch calls this type
keyword, a SQL database uses an indexed column). An exact lookup on it is cheap and unambiguous, and you can route to it when a question matches the id's shape (Topic 7.6). - ▸
In hybrid search (Phase 6), identifier questions are where the keyword side must be allowed to win. A fusion that lets vectors outvote an exact code match produces exactly the wrong top result shown in the lab.
Remember this
- 1
Why this exists. The index of Topic 1.2 and the scores of Topics 1.3 and 1.4 treat a passage as a bag of words.
late returnedandreturned latelook identical, and so do "credit ride" and "ride credit". A word in a title looks the same as the same word in the middle of a paragraph. AndERR-4012is just two words,errand4012, that could appear anywhere. Three small additions repair this. - 2
Positions and phrase queries. Store, for each term in each passage, the *positions* where it occurs (the first word is position 0). A phrase such as
ride creditmatches whenrideis at positionpandcreditatp + 1in the same passage. In D1,rideis at positions 1 and 18 andcreditat 19, so the phrase matches at 18.credit ridehas no match, although both words are in D1 and an AND query finds it. Positions make the index bigger, which is why engines let you turn them off for fields that never need phrases. - 3
Fields and boosts. Real documents have parts: a title, a heading, a body. A match in the title is stronger evidence than a match in the middle of a paragraph. Index each field separately, score each, and add them with a boost (a multiplier):
score = 3 * title_score + body_score. In the example, the questiondamagefinds D5 only through its title (the body saysdamaged, a different term), andrefundsfinds D1 only through its title (the body saysrefunded). A boost of 3 is a common starting point, not a law; tune it on your own questions. - 4
Titles rescue what stemming would. Both of those title matches exist because titles use the exact word the user typed. If the body had been stemmed (Topic 1.1) the body match would have worked too. When you cannot stem (names, codes, many languages), a descriptive title is the cheap fix. It also helps vectors (Topic 4.5 puts the title into each chunk).
- 5
Exact identifiers. Error codes, part numbers, order ids, function names and people's names have a special property: *nearly right is wrong*.
ERR-4012andERR-4021mean different things, and the user wants one of them. Keyword search does well here because after tokenising,4012is a rare term with a high weight. It still fails in a subtle way: a typo (ERR-4O12, letter O) yields only the shared termerr, so every error passage ties and the top result *looks* confident but is arbitrary. The example prints this. - 6
Defences for identifiers. First, when a question contains something shaped like an id, look it up with an exact match (a substring or a field stored without analysis) before ranking, and say clearly when nothing matches. Second, never stem or fold an identifier field. Third, test the failure: add an id question with a wrong character to your test set and check the system does not return a different, real code with confidence. Vector search is worse at this than keyword search, as the lab below shows.
- 7
One caution on adding field scores. The example simply adds BM25 scores of the title and the body. A cleaner method, BM25F, combines the *term frequencies* from the fields first and then applies saturation once, so a word that appears in both title and body cannot be counted twice at full strength. Real engines implement field handling for you (Topic 1.7); the point here is the idea.
Explain it without notes
How does a phrase query use positions, and how does it differ from an AND query?
Why are fields and boosts useful, and what is a risk of adding the scores of two fields?
Why must identifiers be matched exactly, and what goes wrong after a one-character typo?
Practice
Write has_phrase(words, phrase) that is true when the list phrase appears as consecutive items in the list words. Test it on the words of "Later cancellations get ride credit instead." with the phrases ride credit, credit ride and get ride.
Write combined(title_score, body_score, boost=3.0) returning the boosted sum and print it for the pairs (1.356, 1.578) and (0.0, 1.66) to 3 decimals.
Trade-offs
- ↔
Positions: phrase matching and proximity, at the cost of a larger index and slower indexing.
- ↔
Boosts: put the obvious field first, at the cost of one more number to tune and of hiding good body-only matches.
- ↔
Exact lookup on identifiers: precise and honest about "not found", but it will not forgive typos. A fuzzy fallback can offer "did you mean" instead of silently guessing.
Done when you can
Done when you can explain a phrase query with positions and why it is stricter than AND.
Done when you can add a title field with a boost and say how you would choose the boost.
Done when you can explain why
ERR-4O12must not return a confident result.Done when you can name what vectors get wrong about exact codes, using the lab as evidence.