Command Palette

Search for a command to run...

Hectal
Back to the lesson: Topic 4.1 — Loading Documents
RAG and Vector Search · Example 2 of 2

The same bytes, decoded two ways

JavaScript's TextDecoder treats the label latin1 as windows-1252, which matches ISO-8859-1 for these two bytes.

Loading turns a file or a web page into a clean document with a source, a title and plain text. Doing it behind one Loader interface keeps navigation menus, stray markup and broken characters out of the index.

Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.

Ready for more? This phase has 6 hands-on exercises with automatic checks.

Practice questions

Write the code in the editor, run it, then open the model answer to compare.

01

Write load_text(source, raw) for plain .txt files: the title is the first non-empty line, the text is the remaining lines joined with single spaces. Print the result for a small made-up note.

02

Write decode_or_report(data) that decodes bytes as UTF-8 and returns the text, or the string "not UTF-8" if decoding fails. Test it with the bytes of cafe with an accent (UTF-8) and with a lone byte 255.

Explain it without notes

01

Why does the order of steps in an HTML loader matter? Give an example.

02

What is mojibake, how does it happen and how do you detect it early?

03

Why return the same document shape from every loader?

04

A site has a navigation menu on every page. How exactly does it hurt retrieval?

The same bytes, decoded two ways
Sign in to run this example in your browser.

Expected output

utf-8:   4 characters, code points 99 97 102 233
latin-1: 5 characters, code points 99 97 102 195 169
strict decoding: error, the file is not valid UTF-8
with replacement: 72 105 65533