Recipe: a structure-aware chunker with limits
LIMIT is in characters to keep the output short. Use tokens in production, and the real tokenizer of your embedding model.
Three recipes to copy into any RAG project: a structure-aware chunker with limits, a near-duplicate filter that feeds an idempotent sync plan, and the ingestion checklist to run before you trust an index.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write dry_run(stored, wanted) that prints the plan lines add: ... and delete: ... (sorted, comma separated, or none) without changing anything. Call it for stored {"a", "b"} and wanted {"b", "c"}, then for equal sets.
Write safe_to_delete(stored_count, delete_count, share) returning True when delete_count is at most share of stored_count. Print it for 10 stored and 3 to delete at 0.5, and for 10 stored and 8 to delete.
Explain it without notes
What does the structure-aware chunker do with an empty section and with a sentence longer than the limit, and why?
Why must the second run of a correct sync plan be empty?
Why drop near-duplicates before planning the sync rather than after?
Why add a guard against mass deletions?
Expected output
Rental guide > Booking | 61 | Book a bike in the app. Pick a shop, a bike and a start time.
Rental guide > Booking | 76 | A ride cancelled at least 2 hours before its start time is refunded in full.
Rental guide > Booking | 44 | Later cancellations get ride credit instead.
Rental guide > Safety | 44 | A helmet is included free with every rental.
Rental guide > Safety | 145 | Always lock the bike to a rack with the built-in chain, even for a minute, because a bike stolen while unlocked costs the rider a 150 dollar fee.
chunks: 5, over the limit: 1 (a single long sentence is kept whole)