Recipe: drop near-duplicates, then plan the sync
The threshold 0.6 is chosen for this pair (0.647). Choose yours by looking at real pairs. Ids use the hand-written FNV-1a so every language agrees; use SHA-256 in production.
Three recipes to copy into any RAG project: a structure-aware chunker with limits, a near-duplicate filter that feeds an idempotent sync plan, and the ingestion checklist to run before you trust an index.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write dry_run(stored, wanted) that prints the plan lines add: ... and delete: ... (sorted, comma separated, or none) without changing anything. Call it for stored {"a", "b"} and wanted {"b", "c"}, then for equal sets.
Write safe_to_delete(stored_count, delete_count, share) returning True when delete_count is at most share of stored_count. Print it for 10 stored and 3 to delete at 0.5, and for 10 stored and 8 to delete.
Explain it without notes
What does the structure-aware chunker do with an empty section and with a sentence longer than the limit, and why?
Why must the second run of a correct sync plan be empty?
Why drop near-duplicates before planning the sync rather than after?
Why add a guard against mass deletions?
Expected output
run 1: add 1, delete 1, keep 1, near-duplicates dropped 1 (0.647)
run 2: add 0, delete 0, keep 2, near-duplicates dropped 1 (0.647)