Topic 4.2
PDFs, Scans and Tables
In one line
A PDF stores where each piece of text sits on the page, not what order to read it in, and a scanned page has no text at all. Tables lose their meaning when flattened. Each problem has a known fix.
Think of it like this
A newspaper page photographed and cut into strips. Every strip has the right words, but if you read the strips from top to bottom across both columns you get half a sentence from the left article, half a sentence from the right, and then a bit of the page number. A PDF is a printed page: it knows where ink goes, not the order a human reads it in.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- A file format that describes where text and images sit on a printed page.
- Text layer
- The text stored inside a PDF. A scanned PDF has none.
- Extractor
- A library that pulls text, and sometimes layout, out of a PDF.
- Reading order
- The order in which a human reads the page: column by column, top to bottom.
- OCR
- Optical character recognition: software that reads text from an image.
- Scan
- A page that is only a picture of text.
- Table row sentence
- A table row rewritten as text that carries its column names, so each value keeps its label.
Step by step
01Step 1: see what an extractor gives you
Not a string but fragments: (x, y, text). The first example uses a small page with two columns, a header and a footer.
FRAGMENTS = [
(50, 20, "Lumen Bikes Handbook"), # header
(50, 100, "Shops open at 7 am"), (300, 100, "A helmet is included"),
(50, 120, "and close at 9 pm every"), (300, 120, "free with every rental."),
(50, 780, "Page 3"), # footer
]02Step 2: the naive reading order fails
Sort by y, then x. Lines of the left and right column alternate, and the header and footer are glued to the text.
03Step 3: drop the margins, then read column by column
Keep only fragments between a top and a bottom margin. Split at the gap between the columns (x = 200), sort each column top to bottom, and join left then right.
04Step 4: detect scans
After extraction, a page with no text (or a handful of characters) is probably a scan. Count them, send them to OCR, and tag their text ocr.
05Step 5: turn a table into sentences
Parse the Markdown table into rows. Read it column by column to see the problem, then write one sentence per row that repeats the header next to each value.
06Step 6: keep the table together
A table is split by the chunker like any text (Topic 4.4). Make sure a row sentence is never cut in half, and consider one chunk per table when it is short.
Try it yourself
- 1
Move the gap
Change
SPLIT_Xto 400 and runby column. Why does the output now look like theby lineone? What would go wrong on a page with three columns? - 2
Add a header-less table row
Add the row
| Student pass | 15 dollars | unlimited rides of up to 45 minutes |to the table (it is a made-up row for practice). Does the row sentence still carry its labels? - 3
Make the sentences shorter
Change the row sentence to
Monthly pass: 25 dollars, unlimited rides of up to 45 minutes.Would a question about the price still find it? Which words did you drop, and might a user ask with them?
Code & diagrams
The positions are made up to show the idea. Real extractors report many small fragments per line, and column detection is a heuristic.
Expected output
by line: Lumen Bikes Handbook Shops open at 7 am A helmet is included and close at 9 pm every free with every rental. day. Riders under 16 must wear one. Page 3
by column: Shops open at 7 am and close at 9 pm every day. A helmet is included free with every rental. Riders under 16 must wear one.The table is the new handbook document D13 (Price list). Its monthly pass row agrees with D8.
Expected output
column by column: Plan Single ride Day pass Monthly pass Price 4 dollars 12 dollars 25 dollars Includes up to 30 minutes unlimited rides of up to 30 minutes unlimited rides of up to 45 minutes
Plan: Single ride. Price: 4 dollars. Includes: up to 30 minutes.
Plan: Day pass. Price: 12 dollars. Includes: unlimited rides of up to 30 minutes.
Plan: Monthly pass. Price: 25 dollars. Includes: unlimited rides of up to 45 minutes.Not run here: these need a PDF file and the libraries. The calls were taken from each library's documentation, but check the version you install: PDFBox changed how a document is loaded between its 2.x and 3.x releases. None of these read scanned pages.
# pip install pypdf
from pypdf import PdfReader
reader = PdfReader("handbook.pdf")
pages = [page.extract_text() for page in reader.pages] # one string per page; scanned pages come back empty
print(len(pages), "pages;", sum(1 for p in pages if not p.strip()), "look like scans (no text layer)")
Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Put the column gap in the wrong place
In the Python tab of the first example, set SPLIT_X = 400. Run.
Myth vs fact
Myth
A PDF is just text in a file.
Fact
A PDF stores positioned fragments, and a scanned PDF stores only a picture. Text order and paragraphs have to be reconstructed.
Myth
If extraction returns text, it is right.
Fact
It can be in the wrong order, missing a column, or missing every table value's label, and still look like plausible text. Read samples.
Myth
Tables should be embedded as they come out of the extractor.
Fact
A flattened table separates labels from values. Rewrite rows with their headers, or the chunk cannot answer a question about a price.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
Measure extraction quality on your documents: pick 20 pages including your ugliest, extract them with two tools, and read both outputs next to the page image. A tool that wins on clean reports can lose badly on two-column manuals.
- ▸
OCR output is noisy, so mark it in the metadata (
source_kind: ocr). Near-duplicate detection (Topic 4.3) and exact-match search both behave worse on OCR text, and you will want to filter or down-weight it later. - ▸
Real extractions have more quirks than column order: words hyphenated at the end of a line (
refund-/ed), ligatures such asficoming out as one odd character, text in rotated or password-protected pages, and a footnote glued into the middle of a paragraph. Search a sample of your extracted text for stray hyphens and unusual characters before you chunk, because a split word is two tokens that match nothing. - ▸
Merged cells, multi-line headers and tables that continue on the next page break the simple row-sentence approach. For a complex table keep a structured copy (for example as a CSV or JSON) alongside the text and answer numeric questions from the structured copy, not from a chunk.
Remember this
- 1
Why this exists. Most organisations keep their knowledge in PDFs: manuals, contracts, reports, slide decks. A PDF format describes each fragment of text by position (an
x, how far from the left edge, ay, how far along the page vertically, and a font). The example countsydownward from the top, but raw PDF coordinates count upward from the bottom and libraries differ, so check which way yours goes before you write a margin rule. There is no paragraph and no reading order inside the file. A PDF extractor (a library that pulls the text out) must guess the order from positions, and on layouts with columns, side notes or tables the guess can be wrong. - 2
Reading order and columns. The example gives a two-column page in the form an extractor sees it: fragments with positions. Sorting by line (top to bottom, then left to right) interleaves the two columns:
Shops open at 7 am A helmet is included and close at 9 pm every free with every rental.Splitting by column first (everything left of a gap, then everything right of it) restores both sentences. Good extractors detect columns for you; you still check a few pages by eye. - 3
Headers, footers and page numbers. Every page repeats a title at the top and a page number at the bottom. In the example the header sits at
y = 20and the footer aty = 780, outside the body margins, so one rule drops both. Topic 4.3 removes the ones that are not at a fixed position by counting repeated lines. - 4
Scans need OCR. A scanned page is a picture. It has no text layer, so an extractor returns an empty string. OCR (optical character recognition) reads the pixels and produces text, with errors: a
1read as anl, a missing line. A practical pipeline checks every page for an empty result and sends those pages to an OCR engine such as Tesseract (a widely used open-source one) or to a hosted document-reading service, and records that the text came from OCR so you know to trust it less. - 5
Tables lose their meaning. Read a table column by column, which many extractors do, and
Priceends up far from25 dollars; the example printsPlan Single ride Day pass Monthly pass Price 4 dollars 12 dollars 25 dollars .... Nothing links25 dollarstoMonthly pass. The fix is to keep each row together and put the header next to every value:Plan: Monthly pass. Price: 25 dollars. Includes: unlimited rides of up to 45 minutes.Now a chunk about the monthly pass says its price. This is also how a model best reads a table: as sentences. - 6
Tool categories (verified against their documentation). Text-layer extraction libraries:
pypdffor Python, Apache PDFBox for Java, Mozilla PDF.js for JavaScript. Layout-aware tools that detect columns, tables and headings. OCR engines (Tesseract is open source). Hosted document-processing services. Pick by testing on your own worst pages, not by reading feature lists. This handbook gets a new document for the examples: D13 Price list, a Markdown table.
Explain it without notes
Why does text extracted from a two-column PDF often come out in the wrong order?
How do you find out that a page is a scan, and what do you do about it?
Why rewrite a table's rows as sentences before embedding?
How would you choose between two PDF extraction tools for your company's documents?
Practice
Given the fragments [(50, 10, "Page 1"), (50, 50, "Return the bike"), (50, 70, "to any shop."), (50, 790, "Confidential")], write body_text(fragments, top, bottom) that keeps only fragments with top <= y <= bottom, sorts by y and joins the text with spaces. Print it with top=40, bottom=760.
Write row_sentences(header, rows) that returns one sentence per row in the form Plan: Day pass. Price: 12 dollars. and print them for the header ["Plan", "Price"] and the rows [["Day pass", "12 dollars"], ["Single ride", "4 dollars"]].
Trade-offs
- ↔
Text-layer extraction: fast and exact on digital PDFs, useless on scans. OCR: handles scans, slower, and makes character errors.
- ↔
Row sentences: great for lookup questions, but repeat the header text and lose the table's visual grid for comparisons across rows.
- ↔
A layout-aware tool: better reading order and tables, at the cost of speed, a heavier dependency and sometimes a hosted service.
Done when you can
Done when you can explain why a PDF has no reading order and how column-aware extraction restores one.
Done when you can detect scanned pages and route them to OCR with a label.
Done when you can turn a table into row sentences that keep their labels.
Done when you can name the tool categories and say how you would choose between them.