Command Palette

Search for a command to run...

Hectal
Phase 4Intermediate5 of 14 in RAG and Vector Search

Ingestion: From Files to Searchable Chunks

Load Markdown, HTML, PDFs and tables, clean and de-duplicate them, chunk with context and metadata, and re-ingest only what changed.

Retrieval can only find what ingestion put into the index. A page loaded with its navigation menu, a PDF read in the wrong column order, a table flattened into a row of numbers, a chunk cut in the middle of a sentence: each one is a wrong answer waiting to happen, and no embedding model or ranker can repair it afterwards. This phase is about the work that happens before search, and it is where most real RAG systems are won or lost.

You will build every step by hand, in Python, Java and JavaScript: a loader interface for Markdown and HTML, reading order for PDF columns, tables turned into sentences, cleaning, exact and near-duplicate detection with a hash and shingles plus Jaccard similarity, four chunking strategies measured side by side, heading paths and titles that give chunks their context, a metadata schema that is checked, and an incremental sync with stable chunk ids. Where real embeddings show something a toy cannot, you run a search lab in your browser.

The Build topic adds a complete ingestion pipeline to AskDocs: load, clean, chunk with context, attach metadata, embed and upsert, with a second run that changes only what changed. This phase adds new handbook documents (a price table, D13, and a Markdown rental guide, D14) and never changes D1 to D12.

0/10 · 0%
10 topics ~4 h 22 code blocks & diagrams
Start with the first topic
1
4.1

Loading Documents

Loading turns a file or a web page into a clean document with a source, a title and plain text. Doing it behind one Loader interface keeps navigation menus, stray markup and broken characters out of the index.

22 min 2 code practice

2
4.2

PDFs, Scans and Tables

A PDF stores where each piece of text sits on the page, not what order to read it in, and a scanned page has no text at all. Tables lose their meaning when flattened. Each problem has a known fix.

22 min 3 code practice

3
4.3

Cleaning and De-duplicating

Cleaning removes control characters, stray whitespace and headers repeated on every page. De-duplication removes exact copies with a hash and near-copies by comparing sets of overlapping word groups (shingles) with Jaccard similarity.

26 min 3 code practice

4
4.4

Chunking Strategies

A chunk is the unit that gets embedded, retrieved and shown to the model. Fixed-size, sentence, recursive and structure-aware chunking trade simplicity against keeping ideas whole, and size and overlap should be measured, not guessed.

25 min 3 code practice

5
4.5

Giving Chunks Their Context

A chunk cut from its page can say It costs 25 dollars without saying what costs it. Prepending the document title and heading path before embedding fixes that, while the model can be shown a tidier version of the same chunk.

21 min 2 code practice

6
4.6

Metadata That Pays Off

Metadata is the labelled information stored next to every chunk: source, section, date, version, language and who may read it. Each field exists because a later step needs it, and a schema that is checked at ingestion keeps bad records out.

22 min 2 code practice

7
4.7

Re-ingesting When Documents Change

Documents change, and rebuilding everything on each change is slow and costly. Stable chunk ids built from content let a sync add new chunks, skip unchanged ones and delete the ones that no longer exist, so running it twice does nothing the second time.

23 min 2 code practice

8
4.8

Ingestion as a Pipeline

Ingestion is a pipeline of stages. Embedding calls go out in batches, failures are retried with waits, one failed batch must not stop the rest, and changing the chunker or model means a planned backfill.

21 min 1 code practice

9
4.9

Build: The Ingestion Pipeline

AskDocs version 1: documents become chunks with heading paths and metadata, are embedded and upserted by content id, and a second run changes only what changed. Retrieval and the refusal still work on the stored chunks.

22 min 1 code practice

10
4.10

Playbook: Ingestion Recipes

Three recipes to copy into any RAG project: a structure-aware chunker with limits, a near-duplicate filter that feeds an idempotent sync plan, and the ingestion checklist to run before you trust an index.

21 min 3 code practice

Practise this phase

Hands-on exercises: write the code, run it, and press Check to have it tested automatically.

1

Turn an HTML page into clean text

Write html_to_text(raw) that returns the visible text of an HTML string.

Intermediate
2

Near-duplicates with shingles and Jaccard

Write shingles(text, k) that lower-cases the text, takes its words (runs of letters and digits) and returns the set of every window of k consecutive words joined by single spaces. A text with fewer than k words gives an empty set.

Intermediate
3

Chunk Markdown with heading paths

Write chunk_markdown(markdown) returning a list of (path, text) tuples, one for every heading that has text under it.

Intermediate
4

Plan a sync and prove it is idempotent

stored and wanted are dictionaries from chunk id to section name. Write plan_sync(stored, wanted) returning a dictionary with three sorted lists: add (ids only in wanted), update (ids in both whose section differs) and delete (ids only in stored).

Intermediate
5

Clean pages and drop repeated headers

Write clean_lines(page) that takes one page of text and returns its lines as a list: remove control characters (code 0 to 8 and 11 to 31, which keeps tab and newline), collapse runs of spaces and tabs to one space in each line, strip each line and drop lines that are empty.

Intermediate
6

Hash by hand and find exact duplicates

Write fnv1a(text) returning the 32-bit FNV-1a hash of the text as an integer: start with h = 2166136261; for each character, h = ((h ^ ord(ch)) * 16777619) & 0xFFFFFFFF.

Intermediate