Command Palette

Search for a command to run...

Hectal
PHASE 4Intermediate ~22 min· topic 1 of 10

Topic 4.1

Loading Documents

In one line

Loading turns a file or a web page into a clean document with a source, a title and plain text. Doing it behind one Loader interface keeps navigation menus, stray markup and broken characters out of the index.

0/10 · 0%

Think of it like this

A librarian adds a new book to the catalogue. She does not copy the dust jacket, the advertisement on the back or the page numbers. She writes down the title, where the book came from and its content, and she checks that the title is spelled correctly. Loading is that step for your documents: take what is on disk or on a web page and keep only what a reader should search.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Document
One loaded source: where it came from (source), its title and its clean text.
Loader
Code that turns one kind of file (Markdown, HTML, PDF) into a document.
Boilerplate
Parts of a page that repeat everywhere and are not content: menus, footers, banners, scripts.
Entity (HTML)
A written code for a character, such as & for & or   for a non-breaking space.
Encoding
The rule that turns bytes into characters. UTF-8 is the common one.
Mojibake
Garbled text caused by decoding bytes with the wrong encoding, such as é instead of é.
Replacement character
The symbol (code point 65533) that decoders insert for bytes they cannot read.
Registry
A lookup table, here from file extension to loader.
Tag (HTML)
A marker in angle brackets, such as <p> or </p>, that tells a browser how to show the text between them. It is not part of what a reader sees.
Regular expression
A short pattern that finds or replaces text, such as <[^>]+> for "anything between angle brackets".

Step by step

01Step 1: decide what a document is

Fix the shape first: source, title, text. Every loader returns exactly this, so the next stages (cleaning, chunking, metadata) never ask what kind of file it was.

document.pywhole filepython
class Loader:
    def load(self, source, raw):
        """Return {"source": ..., "title": ..., "text": ...}."""
        raise NotImplementedError

02Step 2: the Markdown loader

Find the first line that starts with # for the title. Turn each link into its words with one regular expression, drop the bold markers, and collapse whitespace so the text is one tidy paragraph per document.

03Step 3: the HTML loader, in the right order

First the title from <h1>. Then remove whole blocks that are never content (head, script, style, nav, footer). Then strip the remaining tags. Only then decode entities. In the example, &nbsp; between two sentences becomes a space and Terms &amp; conditions becomes Terms & conditions.

Step 3: the HTML loader, in the right orderdiagram
Rendering diagram…

04Step 4: pick the loader by extension

A dictionary from .md and .html to a loader object. A missing key is a reported gap (scan.pdf | no loader for .pdf), never a silent skip: a skipped file is a document your users will ask about and never find.

05Step 5: bytes and encodings

Read files as bytes and decode them on purpose. The second example decodes the same five bytes two ways and prints the code points, so the damage is visible: 4 characters become 5, and the accented letter turns into two other characters.

06Step 6: keep the source

Never throw the source away, even when it looks redundant. When an answer is wrong, the first question is always which file it came from.

Try it yourself

  1. 1

    Strip one more block

    Add <aside>Sponsored: buy our helmets</aside> inside the HTML body of the first example. Predict whether it appears in the output, then add "aside" to the list of removed blocks and run again.

  2. 2

    Write a third loader

    Add a .txt loader whose title is the first non-empty line and whose text is the rest. Register it in the dictionary and load a made-up notes.txt. Which lines of the other two loaders did you not need to touch?

  3. 3

    Corrupt a character on purpose

    In the second example, change the 195, 169 bytes to 233 (a single Latin-1 byte for é). Does UTF-8 decoding still work? Which of the two decodings is now the right one?

Code & diagrams

A Loader interface for Markdown and HTML New tab

Hand-built with regular expressions so you can see every step. For real web pages use an HTML parser and, for article pages, a main-content extractor.

Sign in to run this example in your browser.

Expected output

hours.md | Opening hours | Shops open at 7 am and close at 9 pm. See the price list for rental costs.
helmets.html | Helmets | A helmet is included free with every rental. Riders under 16 must wear one. Terms & conditions apply.
scan.pdf | no loader for .pdf
The same bytes, decoded two ways New tab

JavaScript's TextDecoder treats the label latin1 as windows-1252, which matches ISO-8859-1 for these two bytes.

Sign in to run this example in your browser.

Expected output

utf-8:   4 characters, code points 99 97 102 233
latin-1: 5 characters, code points 99 97 102 195 169
strict decoding: error, the file is not valid UTF-8
with replacement: 72 105 65533

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Forget that the head is not content

In the Python tab of the first example, change the list of removed blocks to ("script", "nav", "footer"). Run.

terminal
$ Run
── what you'll see ──
hours.md | Opening hours | Shops open at 7 am and close at 9 pm. See the price list for rental costs.
helmets.html | Helmets | Helmets | Lumen Bikes p { color: red; } A helmet is included free with every rental. Riders under 16 must wear one. Terms & conditions apply.
scan.pdf | no loader for .pdf

Myth vs fact

Myth

Stripping tags with one regular expression is enough for HTML.

Fact

It leaves scripts, styles, menus and encoded characters in the text. The order of steps matters, and real pages are messy enough that a parser is the safer choice.

Myth

Everything is UTF-8 nowadays, so encodings do not matter.

Fact

Most files are, but exports from old systems and spreadsheets are not. Decoding wrongly does not crash: it silently corrupts text, which is worse.

Myth

Unsupported files can be skipped quietly.

Fact

A skipped file is missing knowledge. Report every file that was not loaded, with the reason, and decide on purpose.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

  • ▸

    Boilerplate hurts twice. It repeats in every chunk, so keyword scoring gives those words almost no weight (Topic 1.3's IDF), while an embedding of each chunk is pulled slightly towards the menu. Both effects are small per chunk and large across a corpus.

  • ▸

    Keep the raw file (or at least its hash) next to the cleaned document. When you improve the cleaning rules you can re-run them without fetching everything again, and when an answer is wrong you can diff raw against clean.

  • ▸

    Web pages can contain text no human sees (hidden elements, comments) and, from untrusted sites, text written to steer a model. Treat everything you load from outside as data, never as instructions (Phase 10 covers defences). A loader that drops hidden elements and comments removes part of that surface.

Remember this

  1. 1

    Why this exists. Search can only find what was loaded. Suppose every page of a website starts with a menu: Home Shops Prices Help. If the loader keeps it, every chunk of every page contains those words. Keyword search then cannot tell pages apart on them, an embedding of each chunk carries a little of the menu, and the model is shown the menu again and again as if it were an answer. Cleaning at the source is cheaper than fighting it later (Topic 4.3 cleans further).

  2. 2

    A document, in this course, is three things: a source (where it came from, for example handbook/refunds.md), a title, and the text. The source makes citations possible (Phase 8) and lets you find and delete every chunk of one file (Topic 4.7). The title will be given to chunks in Topic 4.5. Everything else about the file (its date, language, who may read it) is metadata and gets its own topic (4.6).

  3. 3

    Markdown is the easy case. It is already text with light structure. The loader reads the first # heading as the title, turns a link such as [price list](prices.md) into its words (price list) because the address is noise for search, and removes the bold markers. It keeps ## headings inside the text on purpose: they are the structure that Topic 4.4 will split on.

  4. 4

    HTML needs more care. A web page is mostly boilerplate: the parts that repeat on many pages and are not the content (navigation, footers, cookie banners, scripts, styles, the <head>). The loader removes whole blocks first, then the remaining tags, then decodes entities such as &amp; and &nbsp;. The order matters: decode first and the text &lt;b&gt; becomes a real tag that the tag stripper deletes. In real projects use an HTML parser instead of regular expressions (for example BeautifulSoup in Python, jsoup in Java or cheerio in JavaScript), and for whole web pages consider a main-content extractor such as Mozilla Readability. The hand-built version here shows what those tools do.

  5. 5

    Bytes are not text. A file is bytes; a charset or encoding says how bytes become characters. Almost everything today is UTF-8, in which é takes two bytes. Decode those two bytes as Latin-1 and you get two wrong characters (é), a corruption called mojibake. Decode with a strict setting and an invalid byte raises an error you can see; decode with replacement and it silently becomes the character U+FFFD (code point 65533). Prefer the loud failure for files you control, and count replacements for files you do not.

  6. 6

    One interface, many loaders. Every loader has the same method, load(source, raw), and returns the same document shape. A registry maps file extensions to loaders. A file with no loader (the example's scan.pdf) is reported, not guessed at. When Topic 4.2 adds PDFs, nothing that calls load changes. That is the same idea as retrieve(question, k) in Topic 0.7: fix the shape at the boundary and let the inside vary.

Explain it without notes

01

Why does the order of steps in an HTML loader matter? Give an example.

02

What is mojibake, how does it happen and how do you detect it early?

03

Why return the same document shape from every loader?

04

A site has a navigation menu on every page. How exactly does it hurt retrieval?

Practice

01

Write load_text(source, raw) for plain .txt files: the title is the first non-empty line, the text is the remaining lines joined with single spaces. Print the result for a small made-up note.

02

Write decode_or_report(data) that decodes bytes as UTF-8 and returns the text, or the string "not UTF-8" if decoding fails. Test it with the bytes of cafe with an accent (UTF-8) and with a lone byte 255.

Trade-offs

  • ↔

    Regular expressions: short and dependency-free, but fragile on real HTML. A parser: more code and a library, much more reliable.

  • ↔

    Strict decoding: you hear about every bad file, but one bad byte stops the load. Replacement decoding: always loads, but hides damage unless you count it.

  • ↔

    Aggressive boilerplate removal: cleaner text, but you may delete a real warning box or a table that sat inside a footer-like element. Sample and read the removed text.

Done when you can

  • Done when you can write a loader that returns source, title and text for Markdown and for HTML.

  • Done when you can explain why entities are decoded last.

  • Done when you can show how wrong decoding corrupts text and how strict decoding exposes it.

  • Done when you can explain why unsupported files are reported rather than skipped.