Chunking by structure, with metadata
Splits at headings, then packs whole sentences up to a maximum size.
Real documents are long, but a search result and a prompt need short, focused pieces. Chunking cuts documents into pieces of a chosen size, with some overlap or along their structure, and attaches metadata so each piece can be found, cited and updated.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write chunk_count(n_words, size, overlap) using the formula and print it for 46 words with (12, 0) and (12, 4).
Write chunk_words(words, size, overlap) that returns the list of chunks as strings, and print the number of chunks for a list of 10 words with size 4 and overlap 1.
Explain it without notes
Why do we chunk documents instead of embedding each document whole?
What does overlap do, and what does it cost?
Name four pieces of metadata to store with a chunk and what each is for.
Expected output
5 chunks, at most 20 words each (a longer sentence stays whole)
[1] manual.md > Returns (15 words)
Unused bikes can be returned within 30 days with the receipt for a full refund.
[2] manual.md > Battery warranty (17 words)
The battery warranty lasts 2 years. It covers defects in the cells but not damage from water.
[3] manual.md > Battery warranty (15 words)
To claim, send the receipt and a photo of the serial number to the workshop.
[4] manual.md > Battery warranty (14 words)
Claims are answered within 5 days. Frames have a longer warranty of 5 years.
[5] manual.md > Hours (20 words)
The shop is open 9 to 6 on weekdays and 10 to 4 on Saturday. It is closed on Sunday.