Fixed-size chunks, with and without overlap
Sizes are counted in words here to keep the example readable. Real systems usually count tokens.
Real documents are long, but a search result and a prompt need short, focused pieces. Chunking cuts documents into pieces of a chosen size, with some overlap or along their structure, and attaches metadata so each piece can be found, cited and updated.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write chunk_count(n_words, size, overlap) using the formula and print it for 46 words with (12, 0) and (12, 4).
Write chunk_words(words, size, overlap) that returns the list of chunks as strings, and print the number of chunks for a list of 10 words with size 4 and overlap 1.
Explain it without notes
Why do we chunk documents instead of embedding each document whole?
What does overlap do, and what does it cost?
Name four pieces of metadata to store with a chunk and what each is for.
Expected output
46 words
size 12, overlap 0: 4 chunks (formula says 4)
words 1-12: The battery warranty lasts 2 years. It covers defects in the cells
words 13-24: but not damage from water. To claim, send the receipt and a
words 25-36: photo of the serial number to the workshop. Claims are answered within
words 37-46: 5 days. Frames have a longer warranty of 5 years.
size 12, overlap 4: 6 chunks (formula says 6)
words 1-12: The battery warranty lasts 2 years. It covers defects in the cells
words 9-20: defects in the cells but not damage from water. To claim, send
words 17-28: water. To claim, send the receipt and a photo of the serial
words 25-36: photo of the serial number to the workshop. Claims are answered within
words 33-44: Claims are answered within 5 days. Frames have a longer warranty of
words 41-46: a longer warranty of 5 years.