Topic 4.4
Chunking Strategies
In one line
A chunk is the unit that gets embedded, retrieved and shown to the model. Fixed-size, sentence, recursive and structure-aware chunking trade simplicity against keeping ideas whole, and size and overlap should be measured, not guessed.
Think of it like this
Cutting a long cake into servings. You can cut every 5 centimetres with a ruler, whatever is underneath (a strawberry gets sliced in half), or cut along the decoration lines the baker made. Chunking is cutting documents into servings the model can take in one bite. Ruler cuts are fast; cuts along the lines keep each piece meaningful.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Chunk
- One piece of a document that is embedded, retrieved and shown to the model.
- Overlap
- Text repeated at the end of one chunk and the start of the next, so a cut sentence appears whole somewhere.
- Packing
- Adding whole sentences to a chunk until the next one would exceed the limit.
- Recursive splitting
- Try a big separator first, and split the pieces that are still too big with smaller ones.
- Structure-aware chunking
- Splitting at the document's own structure, such as headings.
- Heading path
- The chain of headings above a chunk, such as
Rental guide > Booking. - Token
- The unit a model reads; roughly three quarters of an English word.
Step by step
01Step 1: cut with a ruler
Fixed-size chunks with and without overlap, printed between bars so you can see exactly where the cuts fall. Notice the cut in the middle of start time and the doubled words at the boundary when overlap is on.
02Step 2: split into sentences
A regular expression that splits after ., ! or ? followed by whitespace. It is good enough for the handbook and wrong for abbreviations, so test it on your own text.
import re
def sentences(text):
return re.split(r"(?<=[.!?])\s+", text.strip())03Step 3: pack sentences up to a limit
Add sentences while the chunk stays under the limit. When the next one does not fit, start a new chunk. A single sentence longer than the limit stays whole rather than being cut.
04Step 4: split a Markdown document by headings
Walk the lines. A line starting with one to three # signs changes the heading path; other lines belong to the current section. Sections with no text are dropped.
05Step 5: compare the strategies on the same text
The second example runs fixed, sentence and structure-aware chunking on the rental guide (a new handbook document, D14) with a 130-character limit and prints: how many chunks, the longest, how many start mid-sentence, and how many know their section.
06Step 6: measure with questions, not intuition
The right size depends on your questions. Build the retrieval harness from Phase 9 and compare sizes on it. Until then, keep chunks to one idea and keep the numbers in configuration.
Try it yourself
- 1
Choose the cut
In the first example, try size 40 with overlap 0, 10 and 30. Count the chunks each time. At what overlap does the chunk count start to grow fast, and why?
- 2
Shrink the limit
In the second example set
MAX = 70. How do the numbers change for each strategy? Which one is hurt most by a small limit? - 3
Cut a sentence on purpose
Add the sentence
Visit www.lumen.example.com today.to the guide and run the sentence strategy. Does the sentence splitter break the address? How would you protect it? - 4
Use the lab
In the lab, put the lock sentence in its own passage by deleting the other sentences from the long one, then search again. Now add the sentence about helmets back. How fast does the score fall as unrelated sentences are added?
Code & diagrams
Sizes here are characters to keep the output short. Real systems usually count tokens (LLM Engineering Topic 1.2).
Expected output
size 50, overlap 0: 3 chunks
|A ride cancelled at least 2 hours before its start|
| time is refunded in full. Later cancellations get|
| ride credit instead.|
size 50, overlap 15: 4 chunks
|A ride cancelled at least 2 hours before its start|
|efore its start time is refunded in full. Later ca|
| full. Later cancellations get ride credit instead|
| credit instead.|The rental guide is a new handbook document, D14: a Markdown file with three sections. D1 to D12 are unchanged.
Expected output
fixed 4 chunks, longest 130, 3 start mid-sentence, 0 know their section
sentences 6 chunks, longest 121, 0 start mid-sentence, 0 know their section
structure 6 chunks, longest 121, 0 start mid-sentence, 6 know their sectionTry: The question asks where to lock the bike. The lock rule sits at the end of one long chunk that mixes five topics, so its score is low. Edit that passage down to just the lock sentence, press Search and compare the score. Which size would you pick for chunks in a handbook like this one?
Goals
- The chunk with the lock rule scores at least 0.60
Press Search. Then change one thing (a word in the question, a passage, the ranker) and compare the scores.
Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Overlap almost as large as the chunk
In the Python tab of the first example, change the settings list to [(50, 0), (50, 45)]. Run (only the first lines of the second list are shown here).
Myth vs fact
Myth
Smaller chunks are always more precise.
Fact
They are more specific, but an answer split over two tiny chunks is held by neither. Test sizes on your own questions.
Myth
Overlap fixes bad cuts.
Fact
It makes one more chunk contain the cut sentence whole, at the price of duplicate storage. It does not make the cut meaningful.
Myth
One chunking strategy fits every document type.
Fact
Prose, code, tables, transcripts and FAQs all split differently. Use structure where it exists.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Chunk by tokens, not characters, when you can: the embedding model's input limit (512 tokens for the lab's model) and the context window are both measured in tokens, and the same characters tokenise differently in other languages. Over-length input is silently truncated by many embedding models, so the end of a long chunk is not embedded at all.
- ▸
Overlap has a cost you can compute. Chunks advance by
size - overlap, so a corpus needs aboutN / (size - overlap)chunks: with size 1,000 and overlap 200 that is 1 / 800 per token against 1 / 1,000 without, 25 percent more vectors, embedding calls and storage. In the first example, size 50 with overlap 15 gave 4 chunks instead of 3. - ▸
Keep the chunker version in the chunk metadata (Topic 4.6) and in the chunk id (Topic 4.7). Changing chunk size changes every id, which forces a rebuild; stamping the version lets you find stale chunks and backfill (Topic 4.8).
- ▸
Small-to-big retrieval embeds small chunks for precise matching but hands the model the chunk's parent section or its neighbours (Topic 4.5). It buys both precision and context at the cost of storing the parent relation.
Remember this
- 1
Why this exists. A whole document is too long to embed well (a model reads only so many tokens: the lab's model reads at most 512) and too large to show in full on every question. And one vector for a long passage averages many topics, so it matches none of them well. A chunk should be about one idea: small enough to be specific, big enough to make sense alone. LLM Engineering Topic 8.4 gave the short version; this topic measures the options.
- 2
Fixed-size chunking. Cut every
sizecharacters (or tokens), optionally with an overlap so that a sentence cut at a boundary appears whole in the next chunk. It is trivial to write and fast, and it ignores meaning: the first example cuts... before its start|and starts the next chunk withtime is refunded. With size 50 and no overlap the 121-character refund passage becomes 3 chunks; with an overlap of 15 it becomes 4, and neighbouring chunks repeat the end of the previous one (chunks 3 and 4 both containcredit instead). Overlap costs extra storage and does not repair a bad cut, it only hides some. - 3
Sentence chunking. Split at sentence ends, then pack whole sentences into a chunk until adding one more would pass a limit. No sentence is ever cut, so no chunk starts mid-sentence. The weakness: a sentence only makes sense with its neighbours (
Later cancellations get ride credit insteadmeans nothing alone), and sentence splitting by punctuation fails one.g.or numbers like3.5. - 4
Recursive splitting. Try the largest separator first (blank line between paragraphs), and only for pieces still too big try the next (a line break, then a sentence end, then a space). It adapts to text with and without structure, and is the usual default in libraries. The playbook's chunker is a version of it that splits by headings first.
- 5
Structure-aware chunking. Documents already have structure: headings, lists, tables, code blocks. Split at headings first, so a chunk never crosses from
BookingintoReturning a bike, and keep the heading path (Rental guide > Booking) with each chunk (Topic 4.5 uses it). Only a section that is still too long is split further, by sentences. On the example guide the fixed cuts give 4 chunks and 3 of them start mid-sentence, sentence packing gives 6 chunks and none starts mid-sentence, and structure-aware chunking gives the same 6 chunks but is the only one where all 6 know their section. - 6
Choose size and overlap by measuring. There is no best number. Pick candidates, build the index for each, and measure retrieval on your own questions (Phase 9 builds the harness): did the right chunk appear in the top results? Rules of thumb to start from: a few hundred tokens, enough for one idea; smaller for precise factual questions, larger for explanations. Watch the failure on each side. Too small: the answer is split across chunks and none holds it alone. Too large: the vector blurs several topics (the lab shows a lock rule scoring 0.73 alone and about 0.41 when buried in a long mixed chunk).
Explain it without notes
Compare fixed-size, sentence, recursive and structure-aware chunking.
What does overlap do, what does it cost and what does it not fix?
Why can very large chunks hurt retrieval even when the answer is inside them?
How would you decide on a chunk size for a new corpus?
Practice
Write pack(sentences, limit) that joins sentences with single spaces into chunks no longer than limit characters (a sentence longer than the limit stays alone). Print the chunks for the sentences ["A.", "BB.", "CCC."] with limit 7.
Write starts_mid_sentence(chunks) that counts chunks whose first character is not an upper-case letter. Print the count for ["Hello there.", "ello there.", "world"].
Trade-offs
- ↔
Small chunks: precise matches and cheap prompts, but answers can be split. Large chunks: complete ideas, but blurry vectors and costly prompts.
- ↔
Overlap: safer boundaries, more storage and duplicates.
- ↔
Structure-aware chunking: best quality when documents have structure, extra code per format, and falls back to sentences when they do not.
Done when you can
Done when you can write fixed-size chunking with overlap and show where it cuts badly.
Done when you can pack sentences up to a limit and split Markdown by headings.
Done when you can explain what too small and too large chunks each break.
Done when you can describe how to choose a chunk size by measurement.