Topic 8.4
Chunking Documents
In one line
Real documents are long, but a search result and a prompt need short, focused pieces. Chunking cuts documents into pieces of a chosen size, with some overlap or along their structure, and attaches metadata so each piece can be found, cited and updated.
Think of it like this
Index cards for an exam. You do not photocopy the whole textbook. You write one idea per card, with the chapter name in the corner. Cards that are too big hide the idea in noise; cards that are too small (half a sentence) mean nothing alone. Chunking is making good index cards from long documents.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Chunk
- A piece of a document that is stored, searched and retrieved on its own.
- Chunk size
- How big a chunk is, counted in words, tokens or characters.
- Overlap
- How much of the end of one chunk is repeated at the start of the next.
- Metadata
- Information about a chunk (source, title, number, date, access rules) stored beside its text.
- Structure-aware chunking
- Cutting along headings, paragraphs and sentences instead of at fixed counts.
- Ingestion
- Loading documents, chunking them, embedding or indexing them and storing the result.
Step by step
01Step 1: why not index whole documents
A whole manual as one search unit is huge in the prompt and blurry in the index. Short units are what you want to retrieve and to quote.
02Step 2: cut into windows
The first example splits a 46-word warranty text into windows of 12 words. With no overlap, the claim instruction is cut in the middle (a chunk ends with "and a" and the next starts "photo of the serial number").
03Step 3: add overlap
With overlap 4 each window starts 8 words after the previous one. Look closely at the output: the 13-word claim instruction ("send the receipt and a photo of the serial number to the workshop") is still not whole in any chunk, because it is longer than the 4-word overlap. Overlap only rescues passages shorter than itself. The program also prints the count the formula predicts, so you can check your arithmetic.
04Step 4: cut along the structure instead
The second example reads a small Markdown-like manual. It splits at # headings, then packs whole sentences into chunks of at most 20 words. Every chunk keeps its source and heading as metadata.
chunk = {
"source": "manual.md",
"title": "Battery warranty",
"text": "To claim, send the receipt and a photo of the serial number to the workshop.",
}
label = f"[{chunk['source']} > {chunk['title']}]" # used when citing
text_to_embed = chunk["title"] + ": " + chunk["text"] # the title gives a short chunk its context05Step 5: tune with real questions
No size is right for every collection. Pick a starting point, run real questions, look at which chunks come back, and adjust one setting at a time. Topic 8.8 turns that into a small repeatable test.
Try it yourself
- 1
Predict the count
Before running the first example, use the formula to predict the chunk count for size 10, overlap 5 on 46 words (step 5, so (46 - 5 + 4) // 5). Then change the settings and compare.
- 2
Find the cut sentence
With overlap 0, find the chunk that ends in the middle of the claim instruction. With overlap 4, check whether any chunk holds the instruction from "send" to "workshop" in one piece. If none does, raise the overlap (try 14) until one does, and count the chunks you pay for.
- 3
Change the maximum
In the second example set
MAX_WORDSto 40, then to 8. How many chunks do you get? What happens to the 15-word sentence when the maximum is 8?
Code & diagrams
Sizes are counted in words here to keep the example readable. Real systems usually count tokens.
Expected output
46 words
size 12, overlap 0: 4 chunks (formula says 4)
words 1-12: The battery warranty lasts 2 years. It covers defects in the cells
words 13-24: but not damage from water. To claim, send the receipt and a
words 25-36: photo of the serial number to the workshop. Claims are answered within
words 37-46: 5 days. Frames have a longer warranty of 5 years.
size 12, overlap 4: 6 chunks (formula says 6)
words 1-12: The battery warranty lasts 2 years. It covers defects in the cells
words 9-20: defects in the cells but not damage from water. To claim, send
words 17-28: water. To claim, send the receipt and a photo of the serial
words 25-36: photo of the serial number to the workshop. Claims are answered within
words 33-44: Claims are answered within 5 days. Frames have a longer warranty of
words 41-46: a longer warranty of 5 years.Splits at headings, then packs whole sentences up to a maximum size.
Expected output
5 chunks, at most 20 words each (a longer sentence stays whole)
[1] manual.md > Returns (15 words)
Unused bikes can be returned within 30 days with the receipt for a full refund.
[2] manual.md > Battery warranty (17 words)
The battery warranty lasts 2 years. It covers defects in the cells but not damage from water.
[3] manual.md > Battery warranty (15 words)
To claim, send the receipt and a photo of the serial number to the workshop.
[4] manual.md > Battery warranty (14 words)
Claims are answered within 5 days. Frames have a longer warranty of 5 years.
[5] manual.md > Hours (20 words)
The shop is open 9 to 6 on weekdays and 10 to 4 on Saturday. It is closed on Sunday.Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Chunks too small
In the first example, change the first setting from (12, 0) to (5, 0) in the list of settings.
Myth vs fact
Myth
Smaller chunks are always better because they are more precise.
Fact
Very small chunks lose context and force you to retrieve many of them. Too large and they blur. The right size is found by testing on your own questions.
Myth
Overlap fixes every bad cut.
Fact
It helps only when the overlap is at least as long as the sentence that was cut. Cutting at sentence or heading boundaries is more reliable than any overlap.
Myth
Chunking is a one-time setup.
Fact
Documents change. Chunk metadata (source, version) lets you remove and re-ingest the changed document without rebuilding everything.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Count chunk size in tokens of your embedding model, not words or characters: embedding models have a maximum input length, and text beyond it is cut off or rejected.
- ▸
Keep chunk identity stable (source plus position or a content hash) so that re-ingesting an unchanged document does not create duplicates and a changed one replaces the old chunks. Topic 8.8 shows a hash-based re-index.
- ▸
Store the parent context too: retrieve small chunks for accuracy, then optionally send the surrounding section to the model. This "small to big" idea is covered in the coming RAG course.
- ▸
Tables, code, PDFs and slides need their own cutting rules (keep a table row with its header, a function with its docstring). Extraction quality from PDFs is a frequent hidden cause of poor retrieval.
Remember this
- 1
Why this exists. Retrieval returns whole units, and the prompt has a limited budget. If a unit is a 50-page manual, one match drags in 50 pages (slow, costly, mostly irrelevant). Embeddings also blur: one vector for a long text averages many topics, so it matches none of them well. A chunk is a piece of a document, small enough to be about one thing and large enough to make sense alone.
- 2
Fixed-size chunking with overlap. The simplest cut: windows of
sizewords (or tokens, or characters), each startingsize - overlapafter the previous one. With 46 words, size 12 and overlap 0 you get ceil(46 / 12) = 4 chunks. With overlap 4 the step is 8 and you get ceil(42 / 8) = 6 chunks. The overlap repeats the end of each chunk at the start of the next, so a short passage cut at a boundary still appears whole in one of them (a passage longer than the overlap does not). The cost: more chunks, more storage, and repeated text that may appear twice in the prompt. - 3
Why bad cuts hurt. In the example, the instruction "send the receipt and a photo of the serial number to the workshop" is split between two chunks when overlap is 0. A question about how to claim retrieves one half, and the model either misses a step or invents the rest. Too-small chunks (5 words) are worse: every chunk is a fragment with no subject.
- 4
Structure-aware chunking. Documents have natural boundaries: headings, paragraphs, sentences, list items, function definitions. Cut along those first: split at headings, then pack whole sentences into chunks up to a maximum size, never splitting a sentence (a sentence longer than the maximum stays whole). Chunks follow the author's meaning rather than an arbitrary count, and a chunk never starts in the middle of a thought.
- 5
Metadata makes chunks usable. Store with every chunk: the
source(file or URL), thetitleor heading path, a chunk number, the page or date, and who may see it. Use them to cite ("manual.md > Returns"), to filter before searching (only this product, only documents this user may read), to show the reader where an answer came from, and to update: when a document changes, delete its chunks by source and re-ingest. Putting the heading in front of the chunk text when you embed it often helps, because a chunk like "It covers defects in the cells" only makes sense with the title "Battery warranty". - 6
Choosing sizes. There is no universal best. Typical starting points are a few hundred tokens with 10 to 20 percent overlap, but the right size depends on your documents and questions: factual lookup likes small chunks, explanations that span paragraphs like larger ones. Treat size and overlap as settings to test with a set of real questions (Topic 8.8 shows a retrieval test), and change them one at a time.
Explain it without notes
Why do we chunk documents instead of embedding each document whole?
What does overlap do, and what does it cost?
Name four pieces of metadata to store with a chunk and what each is for.
Practice
Write chunk_count(n_words, size, overlap) using the formula and print it for 46 words with (12, 0) and (12, 4).
Write chunk_words(words, size, overlap) that returns the list of chunks as strings, and print the number of chunks for a list of 10 words with size 4 and overlap 1.
Trade-offs
- ↔
Bigger chunks keep more context but cost more tokens per result and blur embeddings; smaller chunks are precise but fragmentary.
- ↔
Overlap protects against bad cuts but multiplies chunks and repeats text in prompts.
- ↔
Structure-aware cutting follows meaning but needs a parser for each document type; fixed windows work on anything.
Done when you can
Done when you can chunk a text by size and overlap and predict the chunk count.
Done when you can explain why a cut sentence hurts retrieval and two ways to prevent it.
Done when you can cut by headings and sentences and keep metadata on every chunk.
Done when you can name the settings you would tune and how you would test them.