Topic 4.5
Giving Chunks Their Context
In one line
A chunk cut from its page can say It costs 25 dollars without saying what costs it. Prepending the document title and heading path before embedding fixes that, while the model can be shown a tidier version of the same chunk.
Think of it like this
Someone hands you one page torn out of a book. It says It costs 25 dollars and includes unlimited rides. You cannot tell what it is about, because the chapter title was on the page before. If the torn-off page had the title written across its top, you would know at once. Chunks are torn-out pages. Chunk context writes the title back on.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Context (of a chunk)
- What the chunk is about according to where it sits: its document title and headings.
- Heading path
- The chain of headings above a chunk, joined as
Plans > Monthly pass. - Embedded text
- The exact text sent to the embedding model. May include context the user never sees.
- Display text
- The text shown to the model and the user, with its citation label.
- Neighbour expansion
- After matching a chunk, also adding the chunks before and after it to the prompt.
- Small-to-big
- Searching with small chunks, answering with their larger parent section.
Step by step
01Step 1: the problem chunks
Three chunks that all start It costs. Print them without their headings and ask which is the monthly pass. You cannot know.
02Step 2: keep the heading path while chunking
The structure-aware chunker from Topic 4.4 already returns (path, text) pairs, for example ("Plans > Monthly pass", "It costs 25 dollars ..."). Keep both, never just the text.
03Step 3: build the embedded text
path + ": " + text. This one line is the whole technique. Scoring the question against it instead of the bare text turns three zeros into a clear winner.
04Step 4: build the display text separately
The prompt and the citation need the source label and a short heading, not necessarily the whole path. Build it from the same record, and store both.
05Step 5: add neighbours when a chunk is not enough
Given the position of a hit, return the chunks from i - window to i + window, in document order, never crossing into another document.
def with_neighbours(chunks, i, window=1):
return chunks[max(0, i - window) : i + window + 1]06Step 6: try it with real embeddings
Use the lab: search without titles and see the monthly pass come last, then put its title in front and search again.
Try it yourself
- 1
Ask a third question
Add
"Is there an unlimited pass for one day?"to the question list of the first example. Predict the scores with and without the path before you run. - 2
Make the prefix too long
Edit the headings in
PLANSso each one is a long phrase, such asPricing information for Lumen Bikes rentals: monthly pass. Which words now match the question, and does the wrong chunk start to score too? - 3
Add neighbours
Write the
with_neighboursfunction from the walkthrough and call it for the monthly pass chunk with window 1. Which chunks come back, and would the model need them for this question?
Code & diagrams
Shared words stand in for a real ranker. A real embedding model is smoother, but it improves in the same direction, as the lab below shows.
Expected output
How much is the monthly pass?
Plans > Single ride text only 0 with its path 0
Plans > Day pass text only 0 with its path 1
Plans > Monthly pass text only 0 with its path 2
What does a single ride cost?
Plans > Single ride text only 0 with its path 2
Plans > Day pass text only 0 with its path 0
Plans > Monthly pass text only 0 with its path 0
embed: Plans > Monthly pass: It costs 25 dollars and includes unlimited rides of up to 45 minutes.
show: [plans.md] Monthly pass - It costs 25 dollars and includes unlimited rides of up to 45 minutes.Try: Press Search. Three chunks begin `It costs`, and the right one for the monthly pass question is not first. Edit the third passage so it begins with `Monthly pass: ` and search again. Then give the other two their titles as well. Does the order stay right?
Goals
- The monthly pass chunk comes first
Press Search. Then change one thing (a word in the question, a passage, the ranker) and compare the scores.
Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Score without the path
In the Python tab of the first example, change score(question, path + ": " + text) to score(question, text). Run.
Myth vs fact
Myth
Chunk context is only needed for keyword search.
Fact
Vectors benefit too: the lab's real embedding model ranks the right price chunk below the wrong ones until its title is added.
Myth
Longer context is always better.
Fact
A long prefix drowns the chunk's own words, and every chunk of a document starts to look alike. A short title and heading path is usually the best trade.
Myth
The model must see exactly what was embedded.
Fact
Embedding and prompting have different jobs. Embed what helps matching, show what helps the answer and the citation.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Changing what you embed (adding the path, changing its format) changes every vector. Include the exact embedded text in the chunk id hash (Topic 4.7), so a changed heading creates a new chunk id and a re-embed, instead of leaving an old vector behind.
- ▸
Headings are not always helpful (
Overview,Notes,FAQ). Prefer the document title plus heading path, and skip generic headings in the prefix. Look at your own heading names before choosing. - ▸
Neighbour expansion multiplies prompt size by up to the window (a window of 1 triples it). Budget tokens (Topic 8.2) and deduplicate when two hits are neighbours of each other.
Remember this
- 1
Why this exists. Chunking (Topic 4.4) cuts documents apart, and the words that said what a passage was about often stay behind in a heading. The price list has three sections, each with a sentence that starts
It costs .... Without its heading, each of those chunks is almost the same sentence with a different number. Neither a keyword search nor an embedding model can tell the monthly pass from the day pass byIt costs 12 dollarsalone. - 2
The fix: embed the heading path with the text. Build the text that gets embedded as
heading path + text:Plans > Monthly pass: It costs 25 dollars and includes unlimited rides of up to 45 minutes.The first example scores two questions by shared words as a stand-in for a real ranker. ForHow much is the monthly pass?every chunk scores 0 with text only; with the path the monthly pass scores 2 (monthly,pass), the day pass 1 (pass) and the single ride 0. ForWhat does a single ride cost?only the single ride chunk scores, with 2. - 3
Real embeddings show the same effect, only smoother. The lab uses a real model. Without titles, the question about the monthly pass scores the three price chunks about 0.45 (day pass), 0.43 (single ride) and 0.41 (monthly pass): the right chunk is only third overall (even the helmet passage, which has nothing to do with prices, scores about 0.34). Put
Monthly pass:in front of the right chunk and its score rises to about 0.79; with all three titled, the day pass scores about 0.62. A few words of context moved it from third place to a clear first. - 4
What to embed is not what to show. The embedded text carries the full path because it helps matching. The text shown to the model and the user can be tidier and more useful: a citation label (
[plans.md] Monthly pass) followed by the original text. The first example prints both forms of the same chunk. Keep both in the record: the vector comes from the first, the prompt and the citation from the second. - 5
Neighbours are context too. A chunk that says
Later cancellations get ride credit insteadonly makes sense after the sentence before it. A common pattern is to search over small chunks and then hand the model the matched chunk plus the chunks on either side (or the whole parent section). Search stays precise because the vectors are small, while the prompt is complete. It needs each chunk to know its position (chunk_index) and its parent. - 6
Keep the prefix short. Context is a few words to a line: the document title and the heading path. A prefix as long as the chunk itself dominates the vector, and every chunk of a document starts to look like every other. Some teams also ask a language model to write one sentence that situates each chunk in its document and prepend that; it can help on hard corpora, and costs one model call per chunk, so measure the gain (Phase 9) before paying for it.
Explain it without notes
Why can a perfectly good chunk be impossible to retrieve, and how does a heading path help?
Explain the difference between embedded text and display text, and why you store both.
What is neighbour expansion and what does it cost?
Why should the context prefix be short?
Practice
Write embedded_text(path, text) returning path: text, and display_text(source, path, text) returning [source] last heading - text. Print both for plans.md, Plans > Day pass, It costs 12 dollars..
Write with_neighbours(chunks, i, window) and print the result for the list ["a", "b", "c", "d", "e"] with i = 0 and window = 1, then i = 2, window = 1.
Trade-offs
- ↔
Context in the embedded text: better matching, slightly longer embedded input and a re-embed when headings change.
- ↔
A model-written situating sentence: richer context on hard documents, a model call per chunk and a new thing that can be wrong.
- ↔
Neighbour expansion: more complete prompts, more tokens and duplicate overlaps.
Done when you can
Done when you can explain why a chunk loses its subject and how a heading path restores it.
Done when you can build the embedded text and the display text from one chunk record.
Done when you can expand a hit with its neighbours without crossing document boundaries.
Done when you can show with the lab how a title changes the rank of the right chunk.