A Loader interface for Markdown and HTML
Hand-built with regular expressions so you can see every step. For real web pages use an HTML parser and, for article pages, a main-content extractor.
Loading turns a file or a web page into a clean document with a source, a title and plain text. Doing it behind one Loader interface keeps navigation menus, stray markup and broken characters out of the index.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write load_text(source, raw) for plain .txt files: the title is the first non-empty line, the text is the remaining lines joined with single spaces. Print the result for a small made-up note.
Write decode_or_report(data) that decodes bytes as UTF-8 and returns the text, or the string "not UTF-8" if decoding fails. Test it with the bytes of cafe with an accent (UTF-8) and with a lone byte 255.
Explain it without notes
Why does the order of steps in an HTML loader matter? Give an example.
What is mojibake, how does it happen and how do you detect it early?
Why return the same document shape from every loader?
A site has a navigation menu on every page. How exactly does it hurt retrieval?
Expected output
hours.md | Opening hours | Shops open at 7 am and close at 9 pm. See the price list for rental costs.
helmets.html | Helmets | A helmet is included free with every rental. Riders under 16 must wear one. Terms & conditions apply.
scan.pdf | no loader for .pdf