Exact duplicates by hash, near-duplicates by shingles
FNV-1a is written out by hand so the three languages agree. In production use a standard hash such as SHA-256 from your language's library.
Cleaning removes control characters, stray whitespace and headers repeated on every page. De-duplication removes exact copies with a hash and near-copies by comparing sets of overlapping word groups (shingles) with Jaccard similarity.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write drop_repeated(pages, share) where pages is a list of lists of lines. It returns the pages without any line that appears on more than share of the pages (count a line once per page). Test it on three pages that all start with HEADER and print the result.
Write jaccard_words(a, b) that tokenises two texts into sets of lower-case words and returns the Jaccard similarity formatted with three decimals. Print it for "the monthly pass" and "the day pass".
Explain it without notes
Walk through cleaning a PDF extraction: which steps, in what order, and why print a removal report?
How does a hash find exact duplicates, and why normalise before hashing?
What are shingles and Jaccard similarity, and why are 3-word shingles stricter than single words?
Why is Jaccard similarity not enough to decide that two versions of a policy can be merged?
Expected output
fnv1a("a") = e40c292c
original 33c77a38 new
reformatted 33c77a38 exact duplicate of original
reworded 96b3c071 new
other 8d2cd51c new
reworded vs original: words 0.818, 3-word shingles 0.652
other vs original: words 0.029, 3-word shingles 0.000