PHASE 15Intermediate ~37 min· topic 10 of 10

Topic 15.10

Collecting Web Data Responsibly

In one line

Web scraping means fetching web pages with a program and pulling data out of them. Use an official API when one exists, read the site's robots.txt and terms, identify your program, go slowly, cache what you fetch, handle personal data with care, and parse with a real HTML parser.

Think of it like this

Copying prices from a shop's notice board into your notebook. If the shop hands you a printed price list (an API), take that. If it lets people read the board, you may copy, but you do it politely: you do not push other customers away, you do not come back every minute, you give your name if asked, and you obey the sign that says "staff only". Web scraping is the same job done by a program, which makes it faster and so easier to do impolitely.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Web scraping: Using a program to fetch web pages and extract data from them.
Crawler: A program that follows links from page to page.
robots.txt: A file at the top of a site that tells automated programs which paths they may visit.
Terms of service: The rules a site publishes for using it. They can forbid automated collection.
User-Agent: A request header that names the program making the request.
Rate limiting: Limiting how many requests are made per second or minute.
Pagination: A long list split into numbered pages, each with a link to the next.
ETag and 304: An ETag is a version tag for a page. A 304 Not Modified reply means your saved copy is still current.
Personal data: Any information that identifies a real person: a name, an email, a phone number, a photo.
Chunk: A piece of a long text, cut to a size that a search index or a model can handle.
HTML: The markup language of web pages: text wrapped in tags such as <p> and <table> that say what each part is (Topic 7.12).
BeautifulSoup: A library (installed with pip install beautifulsoup4) that reads HTML and lets you pick out parts with CSS selectors such as table.books tr.
Training data: The text a machine-learning model learns from. Scraped pages are often used as training data or as search material for AI.

Step by step

01The polite fetch loop

Every responsible scraper follows the same loop. Decide whether to scrape at all, read the rules, fetch one page, wait, parse, save, repeat until there is nothing left or the limit is reached.

The polite fetch loopdiagram

Rendering diagram…

02Read robots.txt with the standard library

RobotFileParser.parse(lines) reads the rules from lines you give it. On a real site you call set_url(...) and read(), which does the download. can_fetch(agent, url) answers yes or no for your User-Agent, and crawl_delay(agent) returns the requested pause or None.

Rules are matched by the most specific User-agent section, and * means everybody. An empty file means everything is allowed. A Disallow: / section means nothing is.

03Identify yourself and slow down

Put your program's name and a contact address in the User-Agent header. If something goes wrong on the server's side, the owner can write to you instead of blocking your whole network.

A delay is just a number you pass in. In a real script the delay becomes time.sleep(delay). Keeping it as a parameter, or passing a sleep function, makes the code testable without waiting.

main.pywhole filepython

import httpx

AGENT = "HectalLearnBot/1.0 (+https://example.com/bot; contact@example.com)"
client = httpx.Client(headers={"User-Agent": AGENT}, timeout=10.0)

04Parse into rows

With BeautifulSoup, soup.select("table.books tr") returns the rows, and find_all("td") returns the cells of a row. The result is a list of dictionaries with clean values: stripped text, a number for the price, a full address for the link.

Cleaning is part of parsing: " ".join(text.split()) collapses runs of spaces and newlines; re.sub(r"[^0-9.]", "", "Rs 1,299") keeps only digits and the dot, so the price becomes 1299.

main.pywhole filepython

import re
from urllib.parse import urljoin

def parse_price(text):
    return float(re.sub(r"[^0-9.]", "", text))

print(parse_price("Rs 1,299"))                       # 1299.0
print(urljoin("https://shop.example/", "/book/3"))   # https://shop.example/book/3

terminal

$ python main.py

── expected output ──

1299.0

https://shop.example/book/3

05Pagination and a safety cap

Follow the next link until it is missing. Always count pages and stop at a maximum, because a site bug (a "next" link that points back to page 1) would otherwise make your loop run forever and hit the server forever.

06Cache and conditional requests

A dictionary in memory is the simplest cache. A folder of saved files or a SQLite table (Topic 15.6) keeps pages between runs, so a second run does no network work at all. A conditional request asks "has this changed since the version I have?". The server replies 304 with no body if not, which costs it almost nothing.

07From pages to a dataset

To feed a search index or a model, keep the main text and drop the rest: remove script, style, nav and footer, collapse whitespace, remove duplicate paragraphs, mask emails and phone numbers, and cut the text into chunks of a sensible size with the source address kept next to each chunk. Keep the address and the date you collected it (in UTC, ISO 8601, Topic 7.11), so you can show where an answer came from and refresh it later.

Try it yourself

1
Block a path
In the first example, add Disallow: /books under User-agent: * and run it. Predict which of the three paths now prints allowed: False, and what polite_get("/books") prints.
2
Add a third page
In the table example, add a /catalog?page=3 entry with one more book, and give page 2 an a.next link to it. Predict the list of network calls before you run it. Then change the second page's next link to point back to page 1 and see how max_pages protects you (set max_pages=3).
3
Mask one more kind of personal data
In the cleaning example, add a paragraph that contains a 12-digit ID number and write a regular expression that replaces it with [id]. Decide what a safe default is when you are not sure whether something is personal data (mask it).

Code & diagrams

robots.txt, a clear User-Agent and a delay New tab

Recall from Topic 15.1: the server here is a Python function (httpx.MockTransport), so nothing touches the network. The line 'would wait' is where a real script would call time.sleep().

Expected output

/books?page=1    allowed: True
/private/admin   allowed: False
/about           allowed: True
BadBot on /about: False
crawl delay: 2
would wait 2 s before fetching /books
<ul><li>Book A</li></ul>
skipped by robots.txt: /private/admin
None
server saw: [('/books', 'HectalLearnBot/1.0 (+https://example.com/bot; contact@example.com)')]

Scrape a table across pages, clean it, and cache it New tab

The second call to scrape() fetches nothing: both pages come from the cache, as the list of network calls shows.

Expected output

{'title': 'Learning Python', 'price': 450.5, 'url': 'https://shop.example/book/1'}
{'title': 'Data at Work', 'price': 620.0, 'url': 'https://shop.example/book/2'}
{'title': 'Clean Code', 'price': 1299.0, 'url': 'https://shop.example/book/3'}
network calls: ['/catalog?page=1', '/catalog?page=2']

A throttle, conditional requests and Retry-After New tab

The clock and the sleep are parameters, so a fake clock proves the throttle works in no real time. The ETag section shows a 304 answer being served from the cache.

Expected output

request 0 at fake time 0.0
request 1 at fake time 2.0
request 2 at fake time 4.0
slept: [1.5, 1.5]
<html><body>price list v1</body></html>
<html><body>price list v1</body></html>
server answers: ['200', '304']
status: 200 waited (fake): [7, 7]

Clean a page for a dataset: text, duplicates, personal data, chunks New tab

Navigation, scripts, the footer and the repeated paragraph are gone, the email and phone number are masked, and the rest is cut into chunks.

Expected output

- Use enumerate
- Loop with enumerate instead of a counter.
- Questions? Mail [email] or call [phone].
chunks:
0 55 Use enumerate Loop with enumerate instead of a counter.
1 40 Questions? Mail [email] or call [phone].

The same script against a real site (needs the internet)python

Not runnable here: the browser has no real network. Run it on your own computer after pip install httpx beautifulsoup4. Use a site whose terms allow it, such as one that publishes a scraping sandbox, or your own site. RobotFileParser.read() downloads robots.txt for you.

import time
from urllib.robotparser import RobotFileParser

import httpx
from bs4 import BeautifulSoup

BASE = "https://example.com"
AGENT = "MyLearningBot/1.0 (+https://example.com/bot; me@example.com)"

rp = RobotFileParser()
rp.set_url(BASE + "/robots.txt")
rp.read()                                   # downloads and parses the file

with httpx.Client(headers={"User-Agent": AGENT}, timeout=10.0, follow_redirects=True) as client:
    path = "/"
    if rp.can_fetch(AGENT, BASE + path):
        time.sleep(rp.crawl_delay(AGENT) or 2)   # be slow
        r = client.get(BASE + path)
        r.raise_for_status()
        soup = BeautifulSoup(r.text, "html.parser")
        print(soup.title.get_text(strip=True))

Look for structured data first New tab

A page-specific JSON address is easier and kinder to read than HTML. find_all('a', href=True) skips anchors that have no href; plain .get returns None for them.

Expected output

[{'title': 'Learning Python', 'price': 450.5}]
everything blocked: False
empty robots allows: True
no crawl delay: None
['/a', None]
['/a']

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Trust that the table exists

Select a table by id on a page that has changed its layout.

main.pywhole filepython

from bs4 import BeautifulSoup

html = "<html><body><div class='list'><p>Pen</p></div></body></html>"
soup = BeautifulSoup(html, "html.parser")
rows = soup.find("table", id="prices").find_all("tr")
print(len(rows))

terminal

$ python main.py

── what you'll see ──

Traceback (most recent call last):

File "main.py", line 5, in <module>

rows = soup.find("table", id="prices").find_all("tr")

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

AttributeError: 'NoneType' object has no attribute 'find_all'

Break #2

No cap on the pages

Follow next links with no limit on a site whose last page links to itself.

main.pywhole filepython

path = "/catalog?page=9"
while path:                      # no page counter
    html = get_html(path)        # the last page's "next" points to itself
    path = next_link(html)

terminal

$ python main.py

── what you'll see ──

(the program never ends and the server sees the same request again and again)

Myth vs fact

Myth

If a page is public, I can collect and use anything on it.

Fact

Public does not mean free to copy or free of rules. Terms of service, copyright, database rights and privacy law can all apply, and they differ by country. Read the terms, check the law where you live, and ask when unsure.

Myth

robots.txt is a security feature that stops bad bots.

Fact

It is a request that polite programs obey. It does not block anyone, and it is not a place to hide secrets. It is still the first thing a responsible scraper reads.

Myth

Faking a browser User-Agent and rotating addresses is just good scraping practice.

Fact

Hiding who you are to get around a block is deceiving the site owner, and it can break their terms or the law. If you are blocked, the right response is to stop, slow down or ask, not to disguise.

Myth

More data is always better for an AI project.

Fact

Duplicates, navigation text, spam and personal data make a dataset worse and riskier. Fewer, clean, licensed, well-sourced documents beat a large dump.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Treat scrapers as fragile integrations. Write the parser as a pure function over saved HTML, keep a few real saved pages as test fixtures (Phase 11), and run a small "canary" fetch on a schedule that alerts you when the layout changes, instead of letting the pipeline quietly produce empty rows.
▸
At scale, fetching is done asynchronously (Topics 12.6 and 12.7) with a semaphore that caps parallel requests per host, plus a per-host rate limit; a polite crawler is slow per site and fast across many sites. Store raw responses (a file or object store) separately from parsed rows, so a bug in the parser is fixed by re-parsing, not re-downloading.
▸
Many pages render their content with JavaScript after loading. The HTML you download then has an empty shell. Look for the page's own JSON calls in the browser's network tab, for a sitemap (/sitemap.xml, readable with ElementTree, Topic 7.12), or for an RSS or Atom feed. A headless browser works but is slow, heavy and the most intrusive option.
▸
Know the limits of urllib.robotparser. It uses plain prefix matching and applies the first rule that matches, while the current robots.txt standard (RFC 9309) also supports the * and $ wildcards and picks the longest matching rule. A file that relies on wildcards can therefore be read too loosely. Also, read() fetches the file with Python's own default User-Agent, which some sites block, and it treats a 401 or 403 as "everything disallowed" and a 404 as "everything allowed". Many scrapers instead download robots.txt with their own httpx client and pass the lines to parse(). Crawl-delay is a non-standard hint that major search engines ignore, so treat it as a minimum and also follow 429 and Retry-After.
▸
If your service fetches URLs that users supply (link previews, "import from URL", an AI agent with a browse tool), you have built a request forger. Allow only http and https, resolve the host and refuse private, loopback and cloud-metadata addresses, limit redirects and response size, and set tight timeouts. This defence is called SSRF protection (server-side request forgery). A polite scraper that visits fixed, known sites does not have this problem, but a general fetch tool does.
▸
For AI datasets, record provenance with every document: source address, the time collected (UTC), the licence or terms you relied on, and the cleaning steps. Deduplicate (exact hashes first, near-duplicate methods later), strip boilerplate (libraries such as trafilatura extract the main article), and keep personal-data masking in the pipeline, not as an afterthought. Content you collect can also contain instructions aimed at models: never let retrieved text decide what your program does (Topic 16.7).

Remember this

1
First ask: is there an API or a download? Many sites offer an official API, a data export, an RSS feed or a public dataset. These are more stable than a page layout, return clean JSON, and come with rules you can read. Scraping is the second choice. If the page itself loads its data from a JSON address (you can see this in the browser's developer tools), calling that address is usually still better than parsing HTML, but check that you are allowed to use it.
2
Rules before code. robots.txt is a small text file at the root of a site (https://example.com/robots.txt) in which the owner says which paths automated programs may visit (Disallow:) and sometimes how fast (Crawl-delay:). It is a convention that polite programs follow; it is not a lock. Python's urllib.robotparser reads it for you. Separately, the site's terms of service may forbid automated collection, and the law where you live and where the site is may limit what you can copy and how you can use it. Copyright, database rights, and privacy law (such as the GDPR in Europe and India's data-protection law) can all apply. This course cannot give legal advice: check the site's terms and the law where you live, and ask for permission when in doubt.
3
Be a good visitor. Send a clear User-Agent that names your program and gives a way to contact you. Put a delay between requests and a limit on the total number of pages. Stop on 429 Too Many Requests and 503, honour a Retry-After header, and back off (Topic 15.2). Cache every page you have already fetched, and use conditional requests (ETag / If-None-Match) so that the server can answer "not changed" with an empty 304. Fetching the same page twice in a test run is both rude and slow. Collect during quiet hours, from one place, with one connection at a time to begin with.
4
Fetch, then parse. httpx fetches the page (Topic 15.1). Then you hand the text to a parser (Topic 7.12): html.parser from the standard library, or BeautifulSoup with CSS selectors such as table.books tr or a.next. Select by structure and meaning (a table, a class, an id) rather than by position ("the 7th div"), and expect the page to change: every selector is a guess about someone else's design. Write the parsing as a pure function from text to rows, so you can test it on a saved page without touching the network.
5
Real pages are messy. Tables have header rows to skip; prices come as Rs 1,299; names have stray spaces; some rows are missing cells; links are relative (/book/3) and need urllib.parse.urljoin to become full addresses. Lists are split into pages, so you follow a next link until there is none, with a hard cap on the number of pages so a loop can never run forever. Treat each field defensively: missing means None, and bad text is logged and skipped, not allowed to crash the whole run.
6
People's data. Names, emails, phone numbers and photos of real people are personal data. Collect only what you need, do not collect what you do not need (a product price needs no email addresses), remove or mask identifiers as early as possible, store the result securely, and do not publish it. For AI work, scraped text becomes training data or search material: the same care applies, plus checking that the licence allows that use, removing navigation and scripts, removing duplicates, and cutting long text into chunks. You will build such a pipeline for retrieval in Topic 16.9. And remember that a page is untrusted input: its text can try to give instructions to a model that reads it (prompt injection, Topic 16.7), so treat scraped text as data, never as commands.

Explain it without notes

Before writing a scraper, what should you check, and why?

What are robots.txt and the User-Agent header for, and how do you read robots.txt in Python?

How do you make a scraper polite? Name at least five behaviours.

How do you handle pagination safely, and what can go wrong with relative links?

What are the privacy and legal concerns when collecting web data, especially for an AI dataset?

Practice

Using RobotFileParser.parse with the lines User-agent: * and Disallow: /admin/, print whether /admin/users and /news may be fetched by MyBot.

Write parse_links(html, base) that returns the full addresses of all links with an href in a page, using BeautifulSoup and urljoin. Test it on <a href='/a'>A</a><a href='https://other.example/b'>B</a><a>none</a> with base https://site.example/.

Write a fetch_all(client, first_path, max_pages) that follows an a.next link (page 1 links to page 2, page 2 has no link) using a MockTransport and returns how many pages it fetched.

Trade-offs

↔
API versus scraping. An API is stable, documented and permitted, but may be limited, paid or missing fields. Scraping reaches anything visible, but breaks when the layout changes and carries legal and ethical risk.
↔
Fast versus polite. Fetching quickly finishes sooner but can overload a small site and get you blocked. Slow, cached, one-connection collection takes longer and is the right default.
↔
Keep raw pages versus keep only parsed rows. Raw pages let you fix a parser bug by re-parsing and prove where data came from, but need storage and may include personal data you would rather not keep. Decide per project, and delete what you do not need.

Done when you can

Done when you can say when to use an API or export instead of scraping.
Done when you can read robots.txt with RobotFileParser and explain what it is and is not.
Done when your fetcher sends a clear User-Agent, waits between requests, stops on 429 and 503, and caches pages.
Done when you can extract a table and links from HTML, follow pagination with a hard cap, and clean the values.
Done when you can name the privacy and legal questions to ask before collecting data, and know to check the terms and the law.
Done when you can turn scraped pages into clean, deduplicated, masked chunks with their source recorded.

Command Palette

Words you'll meet

Step by step

01The polite fetch loop

02Read robots.txt with the standard library

03Identify yourself and slow down

04Parse into rows

05Pagination and a safety cap

06Cache and conditional requests

07From pages to a dataset

Try it yourself

Block a path

Add a third page

Mask one more kind of personal data

Code & diagrams

Break it on purpose

Trust that the table exists

No cap on the pages

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can