Topic 10.6
Deleting for Real
In one line
When a document is removed from the source, or a person asks to be forgotten, its text must leave the chunks, the vectors, the caches, the logs and the backups. Deleting from one table is not deleting, and only an audit proves it.
Think of it like this
You ask a friend to delete a photo of you. They delete it from their phone, but it is still in their cloud backup, in the group chat they sent it to, and in the album they printed. "Deleted" has to mean every copy, and the only way to know is to look in each place.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Derived copy
- Anything made from a document: chunks, vectors, cached answers, log lines, summaries.
- Tombstone
- A record that an id was deleted, kept so that a restore or a late write does not bring it back.
- Audit
- A search through every store to confirm nothing from a document remains.
- Right to erasure
- A person's legal right in some places to ask for their data to be deleted.
- Retention schedule
- A written rule for how long each kind of data is kept.
- Orphan
- A derived item (such as a vector) whose source chunk or document no longer exists.
- Deletion log
- A record that a deletion happened and was verified, without the deleted content.
Step by step
01Step 1: make every derived item point back to its document
Chunk ids carry the document id (D5#1). Vectors use the same id. A cached answer stores the ids of the chunks behind it. A log line lists the sources used. Without these links you can only delete by guessing.
02Step 2: list every store
Write the list: chunks, vectors, cache, logs. For each, note how to find items by document id and how to remove them.
03Step 3: write the audit first
audit(stores, doc_id) returns every leftover as store:item. Writing it before the delete shows you which stores you forgot.
04Step 4: delete only the chunks and run the audit
This is the common half-delete. The audit still lists the vectors, the cached answer and the log lines.
05Step 5: delete everywhere and run the audit again
Remove the vectors, the cache entries that cite the document, and the log lines. The audit returns nothing.
06Step 6: handle backups with tombstones
Keep the tombstone list outside the backups. After a restore, replay it. Set a backup expiry so old copies age out on a schedule.
Try it yourself
- 1
Delete another document
Change both delete calls to
"D6". Predict the leftover list before running, then run. - 2
Skip a store
Comment out the line that clears the cache in
delete_everywhereand run. Which store does the audit name? - 3
Audit by phrase
Add a log line that contains the text
report it in the appbut no chunk id. Does the id-based audit find it? Add a second check that searches every store for a phrase.
Code & diagrams
In production the stores are a database, a vector store, a cache server and a log system. The shape is the same: find by document id, remove, audit.
Expected output
before: chunks:D5#1, chunks:D5#2, vectors:D5#1, vectors:D5#2, cache:what if my bike is damaged, logs:1, logs:3
chunks only: vectors:D5#1, vectors:D5#2, cache:what if my bike is damaged, logs:1, logs:3
everywhere: nothing left
backup restored: 3 chunks, 1 after replaying tombstonesBreak it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Forget the cache
In the Python tab, delete the line stores["cache"] = ... from delete_everywhere, and run.
Myth vs fact
Myth
Deleting the source file deletes the data.
Fact
The index, cache, logs and backups still hold copies. A source deletion only starts the work.
Myth
Vectors are just numbers, so leaving them does no harm.
Fact
Leftover vectors can still be retrieved by their ids and can reveal topics, and some can be partly inverted back to words. Treat them as the document.
Myth
We cannot edit backups, so we cannot comply.
Fact
Keep a tombstone list, replay it after any restore, and let backups expire on a stated schedule. That is the accepted engineering answer.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Prefer soft delete then hard delete: mark the document deleted so it stops being retrievable at once (filter on the flag), then remove the data in a background job and run the audit. Retrieval never serves a flagged document, even while the cleanup is still running.
- ▸
A vector store may keep deleted rows until a cleanup (compaction or vacuum) runs. Know your store's behaviour (Topic 5.6) and schedule the cleanup so deletions become physical within your stated time.
- ▸
Model providers: if you send text to a hosted embedding or chat service, find out how long it retains inputs and whether you can opt out of retention for your account. Redacting before sending (Topic 10.5) means there is less to delete there.
- ▸
Do not log deleted content in the deletion log, in error messages, or in the audit output. The proof that a document is gone must not be a copy of it.
Remember this
- 1
Why it is needed. Documents are withdrawn all the time: a policy is replaced, a file was uploaded by mistake, a customer asks for their data to be erased, a contract ends. A RAG system makes many derived copies of each document: chunks, embeddings, cached answers that quote it, log lines that name its chunks, backups. If any of them survives, the system can still reveal the content, and you cannot honestly say it was deleted.
- 2
First, notice deletions at all. A sync job that only adds and updates will never see that a file was removed from the source. Compare the list of ids in the source with the list in the index on every run (Topic 4.7), and treat "present in the index, absent from the source" as a delete. Many stale-data bugs are simply deletions nobody noticed.
- 3
Then find every copy. Draw the data path and list each store, as in the example: chunk text, vectors, cached answers (keyed by question but pointing at chunks), logs and traces, and backups. Add anything specific to you: search-engine replicas, a model provider's retention, files in object storage, evaluation sets. Give every derived item the id of its source document (chunk ids like
D5#1carry it), so a delete can find them by prefix. An item you cannot trace back to its document cannot be deleted reliably. - 4
The two bugs the example shows. Deleting only the chunks leaves the vectors, so the passage can still be retrieved as an id (Topic 5.6 explains stores that mark rows deleted and clean up later), leaves a cached answer that quotes the text word for word, and leaves log lines naming the chunks. The first audit lists five leftovers after the chunk-only delete. The full delete removes all five and the audit prints "nothing left".
- 5
Backups: do not edit, replay. You cannot rewrite old backups cheaply. The standard answer is a tombstone list (a record of ids that must stay deleted). After any restore, replay the tombstones before the system serves traffic, as the last lines of the example do: the backup holds three chunks and one remains once the tombstoned document is dropped. Backups also expire on a schedule you can state, so a deleted item finally disappears from them within a known time.
- 6
Prove it, then record that you proved it. An audit searches every store for the document's id and for a distinctive phrase from its text, and must find nothing. Run it after every deletion in tests, and periodically in production. Keep a deletion log that records who, when, which document id and the audit result, but never the deleted text itself. For a person's data request this log is your evidence.
Explain it without notes
Name the derived copies a RAG system makes of a document, and say why deleting the source file is not enough.
How do you make sure a deletion in the source reaches the index?
What is a tombstone, and how does it solve the backup problem?
How do you prove a deletion happened, and what do you record?
Practice
Write delete_prefix(store, prefix) that removes every key starting with the prefix from a dict and returns how many were removed. Test it on a dict with ids D5#1, D5#2, D6#1.
Write audit_phrase(stores, phrase) where stores maps store name to a list of strings, returning the names of stores containing the phrase (case-insensitive). Test it with one clean and one dirty store.
Trade-offs
- ↔
Soft delete first: instant effect, the data still exists for a while. Hard delete in the background cleans it up. Use both.
- ↔
Short backup retention: faster real deletion and less history to restore from. Long retention: more recovery options and more data to account for.
- ↔
Logging full questions: easier debugging and more personal data to delete. Log ids and redact the rest.
Done when you can
Done when you can list every derived copy in your system and how to find each by document id.
Done when you can write a delete that reaches all stores and an audit that proves it.
Done when you can explain tombstones and how a restore replays them.
Done when you can say what a deletion log records and what it must not.