PHASE 4Intermediate ~21 min· topic 10 of 10

Topic 4.10

Playbook: Ingestion Recipes

In one line

Three recipes to copy into any RAG project: a structure-aware chunker with limits, a near-duplicate filter that feeds an idempotent sync plan, and the ingestion checklist to run before you trust an index.

Think of it like this

A professional kitchen keeps a card for each dish: ingredients, quantities, steps, and the one mistake to avoid. Cooks do not reinvent the dish each night. This playbook is a set of recipe cards for ingestion that you can adapt.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Recipe: A small, tested piece of code with a clear purpose that you can copy into another project.
Plan: The list of changes a sync will make, computed before anything is written.
Dry run: Computing and printing the plan without applying it.
Golden file: A saved correct output that a test compares the current output against.

Step by step

01Step 1: copy the chunker, then test it on three awkward documents

A document with no headings, one with an empty section and one with a sentence longer than the limit. The recipe's output for the last two is printed below.

Step 1: copy the chunker, then test it on three awkward docudiagram

Rendering diagram…

02Step 2: copy the dedupe-and-plan recipe

Compare each new text against the ones already kept, using 3-word shingles and Jaccard. Keep the first copy. Then compute the plan against the stored ids.

03Step 3: run the plan twice

The second plan must be empty. If it is not, an id is unstable or a stage is not deterministic.

04Step 4: add the loaders for your own files

The polyShow block reads every Markdown and HTML file under a folder as UTF-8. Replace the folder and add formats one loader at a time (Topics 4.1 and 4.2).

05Step 5: wire in retries and stamps

Put the embedding call behind the batch-and-retry loop from Topic 4.8 and stamp each record with the chunker and model versions.

06Step 6: read the checklist

Before you trust an index, walk the checklist below and sample real chunks: read twenty of them, from different documents, as the model would.

Try it yourself

1
Tighten the limit
Set LIMIT to 50 in the chunker recipe. How many chunks do you get, and which ones are over the limit? What does that tell you about the single-sentence rule?
2
Raise the threshold
In the second recipe change the threshold from 0.6 to 0.7. Does the near-duplicate survive? What would the model see then, and which rule would you apply to choose which copy to keep?
3
Make it a dry run
Change the second recipe so that it prints the plan but does not update existing. Run it twice. What does the second plan look like, and why is this the same as a failed run that crashed before applying?

Code & diagrams

Recipe: a structure-aware chunker with limits New tab

LIMIT is in characters to keep the output short. Use tokens in production, and the real tokenizer of your embedding model.

Expected output

Rental guide > Booking | 61 | Book a bike in the app. Pick a shop, a bike and a start time.
Rental guide > Booking | 76 | A ride cancelled at least 2 hours before its start time is refunded in full.
Rental guide > Booking | 44 | Later cancellations get ride credit instead.
Rental guide > Safety | 44 | A helmet is included free with every rental.
Rental guide > Safety | 145 | Always lock the bike to a rack with the built-in chain, even for a minute, because a bike stolen while unlocked costs the rider a 150 dollar fee.
chunks: 5, over the limit: 1 (a single long sentence is kept whole)

Recipe: drop near-duplicates, then plan the sync New tab

The threshold 0.6 is chosen for this pair (0.647). Choose yours by looking at real pairs. Ids use the hand-written FNV-1a so every language agrees; use SHA-256 in production.

Expected output

run 1: add 1, delete 1, keep 1, near-duplicates dropped 1 (0.647)
run 2: add 0, delete 0, keep 2, near-duplicates dropped 1 (0.647)

Where your own files plug inpython

Not run here: it reads a folder named handbook from disk. Reading as UTF-8 on purpose means a bad byte is an error you see (Topic 4.1). The JavaScript uses Node's recursive readdir.

from pathlib import Path

# Every Markdown and HTML file under ./handbook, read as UTF-8 (fail loudly on a bad file, do not guess).
sources = sorted(path for path in Path("handbook").rglob("*") if path.suffix in {".md", ".html"})
raw_documents = {str(path): path.read_text(encoding="utf-8") for path in sources}
print(len(raw_documents), "files to load")

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

A threshold that lets the copy through

In the Python tab of the second recipe, change plan_ingest("d14", existing, TEXTS, 0.6) to use 0.7. Run.

terminal

$ Run

── what you'll see ──

run 1: add 2, delete 1, keep 1, near-duplicates dropped 0 ()

run 2: add 0, delete 0, keep 3, near-duplicates dropped 0 ()

Myth vs fact

Myth

A recipe that works on the handbook will work on any corpus.

Fact

Limits, thresholds and heading levels depend on the data. Treat a recipe as a starting point and test it on your own awkward documents.

Myth

Dry runs are for beginners.

Fact

Printing the plan before applying it is how experienced teams catch a wrong id or a mass deletion before it happens.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Keep a small corpus of awkward documents in your repository (no headings, empty sections, huge sentences, a table, a scan) and run the whole pipeline on it in CI. Compare the produced chunks to saved golden files; a diff tells you what a change really did.
▸
Add a guard against mass deletion: if a sync would delete more than a set share of a source's chunks (say half), stop and ask. An empty file or a failed download looks exactly like a deleted document.
▸
Log per run: files read, chunks produced, added, updated, deleted, embeddings made, failures and elapsed time. A trend line of these numbers finds regressions long before users do.

Remember this

1
Recipe: a structure-aware chunker with limits. When to use it: your documents are Markdown or have headings. What it guards against: chunks that cut sentences, cross sections, or lose their subject. How to adapt it: change the limit (in tokens for production), the heading levels (the example reads # to ###), and add a fallback for documents with no headings (sentence packing). It skips empty sections and keeps a single sentence that is longer than the limit whole instead of cutting it.
2
Recipe: drop near-duplicates, then plan the sync. When to use it: you ingest from several sources that overlap (exports, mirrored pages, drafts). What it guards against: duplicate hits crowding the prompt, and re-embedding unchanged chunks. How to adapt it: choose shingle size and threshold by reading real pairs; log the dropped twin; let metadata decide which copy survives; in production use SHA-256 for ids and MinHash when pairs get too many to compare.
3
Recipe: the idempotent sync plan. Compute add, delete and keep from the stored ids and the wanted ids, apply them, and run the plan again: the second plan must be empty. The example prints add 1, delete 1, keep 1 on run 1 and add 0, delete 0, keep 2 on run 2. How to adapt it: add update for metadata-only changes (Topic 4.7), and write new chunks before deleting old ones.
4
Recipe: the ingestion checklist. Run it after every change to a loader, a cleaner, the chunker or the embedding step. Read the checklist at the end of this topic: it is the list of things that were wrong in real systems and cost days to find.
5
Defensive notes. Everything you ingest comes from outside the model and should be treated as data. Load only from sources you trust or label by trust level (Topic 10.7), strip hidden elements from HTML, keep access labels on every chunk (Topic 10.3) and make sure deleted documents really leave the index (Topic 10.6).

Explain it without notes

What does the structure-aware chunker do with an empty section and with a sentence longer than the limit, and why?

Why must the second run of a correct sync plan be empty?

Why drop near-duplicates before planning the sync rather than after?

Why add a guard against mass deletions?

Practice

Write dry_run(stored, wanted) that prints the plan lines add: ... and delete: ... (sorted, comma separated, or none) without changing anything. Call it for stored {"a", "b"} and wanted {"b", "c"}, then for equal sets.

Write safe_to_delete(stored_count, delete_count, share) returning True when delete_count is at most share of stored_count. Print it for 10 stored and 3 to delete at 0.5, and for 10 stored and 8 to delete.

Trade-offs

↔
A strict guard on deletions: protects against accidents, and needs a person to confirm legitimate large removals.
↔
Running recipes in CI on awkward documents: catches regressions, costs a little maintenance of golden files.
↔
Copying recipes: fast start, but you own them. Read every line you copy.

Done when you can

Before you ship: sample twenty chunks from different documents and read them as the model would see them.
Before you ship: every file is loaded or listed as not loaded, with a reason.
Before you ship: no navigation, headers, footers or page numbers remain in the text.
Before you ship: each chunk has its heading path, source, section, date, version, language and access label, and bad metadata is rejected.
Before you ship: chunk ids come from the embedded content, and a second run changes nothing.
Before you ship: deleted and renamed documents leave the index, a mass-deletion guard exists, and every record is stamped with the chunker and model versions.

Command Palette