A throttle, conditional requests and Retry-After
The clock and the sleep are parameters, so a fake clock proves the throttle works in no real time. The ETag section shows a 304 answer being served from the cache.
Web scraping means fetching web pages with a program and pulling data out of them. Use an official API when one exists, read the site's robots.txt and terms, identify your program, go slowly, cache what you fetch, handle personal data with care, and parse with a real HTML parser.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Using RobotFileParser.parse with the lines User-agent: * and Disallow: /admin/, print whether /admin/users and /news may be fetched by MyBot.
Write parse_links(html, base) that returns the full addresses of all links with an href in a page, using BeautifulSoup and urljoin. Test it on <a href='/a'>A</a><a href='https://other.example/b'>B</a><a>none</a> with base https://site.example/.
Write a fetch_all(client, first_path, max_pages) that follows an a.next link (page 1 links to page 2, page 2 has no link) using a MockTransport and returns how many pages it fetched.
Explain it without notes
Before writing a scraper, what should you check, and why?
What are robots.txt and the User-Agent header for, and how do you read robots.txt in Python?
How do you make a scraper polite? Name at least five behaviours.
How do you handle pagination safely, and what can go wrong with relative links?
What are the privacy and legal concerns when collecting web data, especially for an AI dataset?
Expected output
request 0 at fake time 0.0
request 1 at fake time 2.0
request 2 at fake time 4.0
slept: [1.5, 1.5]
<html><body>price list v1</body></html>
<html><body>price list v1</body></html>
server answers: ['200', '304']
status: 200 waited (fake): [7, 7]