Look for structured data first
A page-specific JSON address is easier and kinder to read than HTML. find_all('a', href=True) skips anchors that have no href; plain .get returns None for them.
Web scraping means fetching web pages with a program and pulling data out of them. Use an official API when one exists, read the site's robots.txt and terms, identify your program, go slowly, cache what you fetch, handle personal data with care, and parse with a real HTML parser.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 6 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Using RobotFileParser.parse with the lines User-agent: * and Disallow: /admin/, print whether /admin/users and /news may be fetched by MyBot.
Write parse_links(html, base) that returns the full addresses of all links with an href in a page, using BeautifulSoup and urljoin. Test it on <a href='/a'>A</a><a href='https://other.example/b'>B</a><a>none</a> with base https://site.example/.
Write a fetch_all(client, first_path, max_pages) that follows an a.next link (page 1 links to page 2, page 2 has no link) using a MockTransport and returns how many pages it fetched.
Explain it without notes
Before writing a scraper, what should you check, and why?
What are robots.txt and the User-Agent header for, and how do you read robots.txt in Python?
How do you make a scraper polite? Name at least five behaviours.
How do you handle pagination safely, and what can go wrong with relative links?
What are the privacy and legal concerns when collecting web data, especially for an AI dataset?
Expected output
[{'title': 'Learning Python', 'price': 450.5}]
everything blocked: False
empty robots allows: True
no crawl delay: None
['/a', None]
['/a']