Cleaning scanned pages and removing repeated headers
Three tiny pages stand in for a real extraction. The report line is the point: always print what a cleaning rule removed.
Cleaning removes control characters, stray whitespace and headers repeated on every page. De-duplication removes exact copies with a hash and near-copies by comparing sets of overlapping word groups (shingles) with Jaccard similarity.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Write drop_repeated(pages, share) where pages is a list of lists of lines. It returns the pages without any line that appears on more than share of the pages (count a line once per page). Test it on three pages that all start with HEADER and print the result.
Write jaccard_words(a, b) that tokenises two texts into sets of lower-case words and returns the Jaccard similarity formatted with three decimals. Print it for "the monthly pass" and "the day pass".
Explain it without notes
Walk through cleaning a PDF extraction: which steps, in what order, and why print a removal report?
How does a hash find exact duplicates, and why normalise before hashing?
What are shingles and Jaccard similarity, and why are 3-word shingles stricter than single words?
Why is Jaccard similarity not enough to decide that two versions of a policy can be merged?
Expected output
removed (repeated on 3 of 3 pages): Lumen Bikes Handbook - Confidential
characters before: 278
page 1: Shops open at 7 am and close at 9 pm.
page 2: A helmet is included free with every rental.
page 3: Bikes returned late are charged 2 dollars for every 15 minutes.
characters after: 144