Topic 4.7
Re-ingesting When Documents Change
In one line
Documents change, and rebuilding everything on each change is slow and costly. Stable chunk ids built from content let a sync add new chunks, skip unchanged ones and delete the ones that no longer exist, so running it twice does nothing the second time.
Think of it like this
Updating a shared shopping list. You do not throw the list away and write it again. You look at what is on it, add the missing items, cross out what is no longer needed and leave the rest. A sync does that for chunks: compare what you have with what you should have, then make the smallest change.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Stable id
- An id that stays the same as long as the thing it names does not change.
- Positional id
- An id built from the chunk's position, such as
d14:2. Changes when text is inserted before it. - Content id
- An id built from a hash of the chunk's content.
- Sync
- Making the store match the current documents with the smallest set of changes.
- Upsert
- Insert if the id is new, update if it exists.
- Idempotent
- Safe to repeat: running it again changes nothing.
- Orphan chunk
- A chunk whose source text no longer exists but which is still in the index.
Step by step
01Step 1: try positional ids and watch them fail
Insert one paragraph at the top of a three-paragraph document. Every position now holds different text. All four chunks look new.
02Step 2: hash the content instead
The id is doc: plus the hash of the chunk's embedded text. The three old paragraphs hash to the same values as before. Only the new paragraph has a new id.
def chunk_id(doc, embedded_text):
return doc + ":" + format(fnv1a(embedded_text), "08x")03Step 3: compute added, deleted, updated and unchanged
Compare the wanted ids with the ids the store has for this document. Handle the four cases separately; each has a different cost.
04Step 4: run it five times
First run, the same again, a text edit, a section rename and a deleted document. Read the counts: 3 embeddings, 0, 1, 0 and 0, so 4 in total.
05Step 5: order the writes safely
Write new chunks before deleting old ones, so a search that runs in the middle of a sync still finds an answer. If the sync crashes between the two, the next run finishes the job, because it is idempotent.
06Step 6: skip unchanged files without even chunking
Store a hash of each raw file. If today's hash equals yesterday's, skip the whole file. Only changed files go through load, clean and chunk.
Try it yourself
- 1
Repeat the last run
Add a sixth run equal to the fifth one (
document deleted). What does it print? Why is that the correct answer? - 2
Move a paragraph
Swap the order of the first two chunks in the
same againrun. Predict the counts. Then run it. Did the order matter to the store? - 3
Two documents
Add a second document
d6with one chunk and sync both documents. Deleted14and check thatd6is untouched. Which part of the code guarantees that?
Code & diagrams
The hash is FNV-1a written out by hand so that the three languages agree. Use SHA-256 in a real system: 32 bits are not enough to avoid collisions in a large store.
Expected output
positional ids: d14:0, d14:1, d14:2, d14:3
chunks to embed again: 4 of 4
content ids: d14:73523ab5, d14:b00d60cf, d14:a53b8284, d14:2a4a4820
chunks to embed again: 1 of 4The store is a dictionary. A real store also keeps the text and the vector, and a real sync would call the embedding model where the example counts embeddings.
Expected output
first run added 3, updated 0, deleted 0, unchanged 0 | store 3, embeddings so far 3
same again added 0, updated 0, deleted 0, unchanged 3 | store 3, embeddings so far 3
helmet text edited added 1, updated 0, deleted 1, unchanged 2 | store 3, embeddings so far 4
section renamed added 0, updated 1, deleted 0, unchanged 2 | store 3, embeddings so far 4
document deleted added 0, updated 0, deleted 3, unchanged 0 | store 0, embeddings so far 4Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Hash the section instead of the text
In the Python tab of the second example, change format(fnv1a(text), "08x"): section for text, section in chunks to format(fnv1a(section), "08x"): section for text, section in chunks. Run.
Myth vs fact
Myth
Re-embedding everything each night is simpler, so it is better.
Fact
It works until the corpus is large or embedding costs matter. An idempotent sync is not much harder, and it gives you cheap, frequent updates.
Myth
Updating a document means updating its chunks in place.
Fact
Chunk boundaries move when text changes. Content ids make a changed chunk a new chunk and an unchanged chunk a no-op, with no in-place bookkeeping.
Myth
Deleted documents disappear from search by themselves.
Fact
Only if your sync deletes their chunks. Otherwise they stay in the index as orphans and keep being retrieved.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
A 32-bit hash has even odds of at least one collision at about 77,000 chunks (the birthday effect: about 1.18 times the square root of 2^32). Use a 128-bit or longer hash, such as SHA-256 cut to a length you choose, for real ids, and keep the document id in the id so collisions are at least confined to one document.
- ▸
Two identical paragraphs in one document hash to the same id and collapse into one chunk. That is usually right, but if you cite by position, add an occurrence number to the hashed text. Also never let two syncs of the same document run at the same time: each computes its plan from the same stored state and they can delete each other's new chunks. Take a lock per document.
- ▸
Changing the embedding model, the chunker or the context format invalidates every vector. Put their versions in the metadata (Topic 4.8) and treat a version bump as a planned backfill, not an incremental sync.
- ▸
Order matters under failure: upsert new, then delete old. If you delete first and crash, retrieval has a hole until the next run. For stricter guarantees, write a new version of the document's chunks and switch a pointer when it is complete.
Remember this
- 1
Why this exists. Handbooks are edited every week. If every edit meant re-reading, re-chunking and re-embedding the whole corpus, it would cost money (embedding calls are billed), take hours and make freshness a monthly event. Most edits touch one paragraph, so most chunks are unchanged and their vectors are still correct. The job is to recognise unchanged chunks and spend work only on the rest.
- 2
The id decides everything. The first example compares two ways to name a chunk after an editor adds one paragraph at the top. Positional ids (
d14:0,d14:1, ...) say where the chunk is: every position now holds different text, so all 4 chunks must be embedded again. Content ids (d14:plus the hash of the chunk's text) say what it says: the three old paragraphs keep their ids and only 1 chunk is new. Same edit, 4 embeddings against 1. - 3
What goes into the hash. Everything that is embedded: the text and, if you embed it (Topic 4.5), the heading path. If the embedded text changes, the vector must change, so the id must change. Fields that are not embedded (section label, updated date, access) stay out of the hash and are updated in place. The second example shows both: editing the helmet sentence adds one chunk and deletes one (a new vector), while renaming the section only updates a label (no new vector).
- 4
The sync, in four moves. For one document, work out the wanted ids from the new chunks.
added: wanted ids the store does not have, which are embedded and written.deleted: ids the store has for this document that are no longer wanted, which are removed.updated: ids present in both whose metadata changed, which are rewritten without re-embedding.unchanged: the rest, which are not touched. The example runs five times and prints the count of each. - 5
Idempotent means safe to repeat. Run the same sync twice and the second run must change nothing:
same againprintsadded 0, updated 0, deleted 0, unchanged 3and the embedding counter stays at 3. This is what lets you retry after a crash, run a nightly job without fear, and put the sync in a loop. - 6
Deletions matter as much as additions. When a paragraph is deleted, its chunk must go too, or the old rule keeps being retrieved forever (a stale chunk is the usual cause of an answer that is right last month and wrong now). Because each id starts with the document,
sync("d14", [])removes all three chunks when the document is deleted. Never rely on someone remembering to clean up by hand.
Explain it without notes
Why are content-based chunk ids better than positional ones for re-ingestion?
What does it mean for a sync to be idempotent, and why is it useful?
What belongs in the content hash and what does not?
How do you handle documents that are deleted at the source?
Practice
Write plan(stored_ids, wanted_ids) returning two sorted lists, the ids to add and the ids to delete. Print them for stored {"a", "b"} and wanted {"b", "c"}.
Write unchanged_share(old_chunks, new_chunks) that returns the fraction (as a string with 2 decimals) of new chunks whose exact text already existed in the old list. Print it for old ["x", "y", "z"] and new ["w", "x", "y", "z"].
Trade-offs
- ↔
Content ids: cheap updates and idempotence, but any change to the embedded format changes every id.
- ↔
Document-level hashes: skip whole files quickly, but any edit inside a file re-chunks it (still cheap, since unchanged chunks keep their ids).
- ↔
Delete-then-insert versus insert-then-delete: the second avoids a gap in results at the cost of briefly holding both versions.
Done when you can
Done when you can explain why content ids beat positional ids with numbers.
Done when you can write a sync that adds, updates, deletes and skips.
Done when you can show that running the sync twice changes nothing the second time.
Done when you can say what goes into the hash and why.