PHASE 4Intermediate ~22 min· topic 9 of 10

Topic 4.9

Build: The Ingestion Pipeline

In one line

AskDocs version 1: documents become chunks with heading paths and metadata, are embedded and upserted by content id, and a second run changes only what changed. Retrieval and the refusal still work on the stored chunks.

Think of it like this

A restaurant kitchen's prep station. Deliveries arrive (documents), someone washes and cuts them (clean and chunk), labels the containers (metadata) and puts them on the right shelf (upsert). Tomorrow's delivery only changes the containers whose contents changed. The cooks (the answerer) never touch raw deliveries. AskDocs version 1 is that prep station.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Ingest: Take documents in, and leave searchable chunks in the store.
Upsert by content id: Write a chunk only if no chunk with the same content id exists.
Embedding call counter: A number that proves how much work a run did: unchanged runs add nothing.
Version 1: AskDocs after this phase: real files in, chunks with context out.
Store: Where chunks and their vectors live. A dictionary for now, a vector database in Phase 5.

Step by step

01Step 1: documents are files

Three Markdown files: two single-section documents and a rental guide with two headed sections. The pipeline never sees a hard-coded passage again.

02Step 2: clean and chunk

Remove control characters, collapse spaces, then walk the lines: each heading starts a section, each section with text becomes a chunk that remembers its heading path.

Step 2: clean and chunkdiagram

Rendering diagram…

03Step 3: embed what was built

The toy embedder counts ten word families in the embedded text. Because the embedded text includes the heading path, Late returns contributes to the vector of the late-returns chunk.

04Step 4: upsert and delete by id

For each source compute the wanted ids, delete stored ids of that source that are not wanted, and write the ids that are missing. Count the embeddings.

05Step 5: run it again

Run 1: 4 chunks added. Run 2: nothing. Run 3 after the edit: 1 added, 1 deleted. The embedding counter goes 4, 4, 5.

06Step 6: ask questions of the store

ask embeds the question, scores every stored vector with a dot product, and answers from the best chunk or refuses when nothing scores above zero.

Try it yourself

1
Rename a heading
Change ## Safety to ## Safety rules in guide.md and ingest again. How many chunks are added and deleted? Why is the heading path part of the id?
2
Delete a file
Remove refunds.md from DOCS, then call ingest. The sources you pass no longer include it. Does the refunds chunk leave the store? What would you add so that it does?
3
Ask your own question
Ask Do riders need a helmet?. Which chunk is found, and why does the toy embedder find it? Then ask Where do I lock my bike? and read the result. Which word family is missing?

Code & diagrams

AskDocs v1: load, clean, chunk with context, embed, upsert, run again New tab

The embedder counts ten word families, a toy that stands in for a real model (Phase 2). The three Markdown files are made-up copies of handbook entries D1 and D6 and the new rental guide D14.

Expected output

run 1: added 4, deleted 0, store 4, embedding calls 4
run 2: added 0, deleted 0, store 4, embedding calls 4
run 3: added 1, deleted 1, store 4, embedding calls 5  (late.md edited)
Q: What is charged for a late return?
   late.md > Late returns | Bikes returned late are charged 3 dollars for every 15 minutes after the booked end time.
Q: Do you rent scooters?
   I can't find that in the handbook.

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Skip the deletion of stale chunks

In the Python tab, replace the line for chunk_id in [c for c in store if store[c]["source"] == source and c not in wanted]: with for chunk_id in []:. Run.

terminal

$ Run

── what you'll see ──

run 1: added 4, deleted 0, store 4, embedding calls 4

run 2: added 0, deleted 0, store 4, embedding calls 4

run 3: added 1, deleted 0, store 5, embedding calls 5 (late.md edited)

Q: What is charged for a late return?

late.md > Late returns | Bikes returned late are charged 2 dollars for every 15 minutes after the booked end time.

Q: Do you rent scooters?

I can't find that in the handbook.

Myth vs fact

Myth

Ingestion is a one-time job.

Fact

Documents change forever. Design for the second run on day one: stable ids, deletions and an idempotent sync.

Myth

The chunk text is all the index needs.

Fact

The path, the source and the metadata decide whether a hit is matched, cited, filtered or deleted correctly.

Myth

A toy embedder proves nothing.

Fact

It proves the plumbing: ids, deletions, counts and the interface. Quality questions need the real model and a golden set.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Make the embedding call count part of the run summary and alert when a no-change run embeds anything. A sudden jump usually means an id changed for the wrong reason, for example an unstable normalisation.
▸
Keep the pipeline a pure function of its inputs: files in, records out. Then you can run it in CI against the handbook on every pull request and compare the chunks before and after the change.
▸
When you add a real vector store (Phase 5), the store dictionary becomes an upsert and a delete(ids) on a VectorStore. The sync logic does not change.

Remember this

1
Why the product needs this now. Version 0 (Topic 0.7) held twelve passages typed into the program. A real handbook is files: Markdown with headings, pages that get edited. Phase 4's ideas turn those files into the Chunk records that retrieve searches. Everything after this phase (vector databases, hybrid search, answers) reads what this pipeline writes.
2
Before and after. Before this step, AskDocs searched twelve passages typed into the program, and an edit meant changing code. After it, AskDocs reads Markdown files, keeps each section's heading path, and updates its store when a file changes, doing work only for what changed. It still cannot read PDFs or HTML in this demo (the loaders of Topics 4.1 and 4.2 plug in at the first stage), and its search is still a toy.
3
What the step adds. An ingest(docs) function with five stages: load (title and body), clean (control characters and spaces), chunk (one chunk per heading with text, keeping its heading path), embed (the path plus the text, so the chunk carries its subject) and upsert (a chunk id from a hash of the embedded text, so unchanged chunks are skipped). It returns a summary: chunks added, chunks deleted, store size and total embedding calls.
4
How it plugs into what exists. It sits behind the same boundary as before. retrieve(question, k) still returns hits; it now reads the store the pipeline writes instead of a hard-coded list. The stored record gains a source, a path and a vector. build_prompt and answer are untouched. To keep the example short, its ask function stands in for all three: it finds the best chunk like retrieve and refuses like answer. The embedder is a toy that counts ten word families (a word start such as cancel, which matches cancelled and cancellations), so the example runs in all three languages; Phase 2 puts a real model behind the same Embedder.embed(text) call.
5
Run it three times. The first run adds 4 chunks from 3 files: Refunds, Late returns and two sections of the rental guide, with 4 embedding calls. The second run, with nothing changed, adds 0 and deletes 0, and the embedding counter stays at 4. In the third run late.md is edited (2 dollars becomes 3 dollars): one new chunk and one deleted, one embedding call, 5 in total.
6
Ask two questions. What is charged for a late return? finds late.md > Late returns and the text with the new 3 dollar rate: the store holds only the new version, so there is no conflict to explain. Do you rent scooters? shares no word family with any chunk, so no chunk scores above 0 and the pipeline answers I can't find that in the handbook. as version 0 did.
7
What is still missing. The embedder is a toy (Phase 2). Search compares the question with every chunk (Phase 3 and 5 index and store them). Ranking is a plain dot product (Phase 6 adds hybrid search and re-ranking). Metadata fields beyond the source and path are not stored yet (the schema of Topic 4.6 gets a real home in the vector store of Phase 5, Topic 5.4). There is no near-duplicate step (Topic 4.3) and no retry (Topic 4.8), because there is nothing to fail in an in-memory demo.

Explain it without notes

Walk through what ingest does for one file, from raw text to a stored vector.

Why is the second run supposed to do nothing, and how do the numbers show it?

How does the pipeline plug into the earlier retrieve and answer functions without changing them?

Name three things this version does not do yet and which phase adds each.

Practice

Write summary(added, deleted, store_size, embeds) returning the one-line run summary in the exact format added 4, deleted 0, store 4, embedding calls 4. Print it for those numbers.

Write stale_ids(store_ids, wanted_ids) returning the sorted ids in the store that are no longer wanted. Print them for {"x", "y", "z"} and {"y"}.

Trade-offs

↔
One chunk per heading: chunks follow the document's ideas, but sections of very different size make uneven chunks (Topic 4.4 splits the large ones).
↔
Embedding the heading path: better matching, but a renamed heading re-embeds its chunks.
↔
An in-memory store: simple and fast to test, gone when the program ends. Phase 5 persists it.

Done when you can

Done when you can run the pipeline in all three languages and explain each stage's output.
Done when you can show with the counters that a second run does no work.
Done when you can edit a file and show exactly one chunk added and one deleted.
Done when you can name the interfaces the pipeline keeps and the phases that replace its toy parts.

Command Palette

Words you'll meet

Step by step

01Step 1: documents are files

02Step 2: clean and chunk

03Step 3: embed what was built

04Step 4: upsert and delete by id

05Step 5: run it again

06Step 6: ask questions of the store

Try it yourself

Rename a heading

Delete a file

Ask your own question

Code & diagrams

Break it on purpose

Skip the deletion of stale chunks

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can