robots.txt, a clear User-Agent and a delay
Recall from Topic 15.1: the server here is a Python function (httpx.MockTransport), so nothing touches the network. The line 'would wait' is where a real script would call time.sleep().
Web scraping means fetching web pages with a program and pulling data out of them. Use an official API when one exists, read the site's robots.txt and terms, identify your program, go slowly, cache what you fetch, handle personal data with care, and parse with a real HTML parser.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Using RobotFileParser.parse with the lines User-agent: * and Disallow: /admin/, print whether /admin/users and /news may be fetched by MyBot.
Write parse_links(html, base) that returns the full addresses of all links with an href in a page, using BeautifulSoup and urljoin. Test it on <a href='/a'>A</a><a href='https://other.example/b'>B</a><a>none</a> with base https://site.example/.
Write a fetch_all(client, first_path, max_pages) that follows an a.next link (page 1 links to page 2, page 2 has no link) using a MockTransport and returns how many pages it fetched.
Explain it without notes
Before writing a scraper, what should you check, and why?
What are robots.txt and the User-Agent header for, and how do you read robots.txt in Python?
How do you make a scraper polite? Name at least five behaviours.
How do you handle pagination safely, and what can go wrong with relative links?
What are the privacy and legal concerns when collecting web data, especially for an AI dataset?
Expected output
/books?page=1 allowed: True
/private/admin allowed: False
/about allowed: True
BadBot on /about: False
crawl delay: 2
would wait 2 s before fetching /books
<ul><li>Book A</li></ul>
skipped by robots.txt: /private/admin
None
server saw: [('/books', 'HectalLearnBot/1.0 (+https://example.com/bot; contact@example.com)')]