HTML with BeautifulSoup: find, select, extract a table
BeautifulSoup is a separate package. Here it loads by itself, because the page imports it. On your computer run pip install beautifulsoup4 first.
XML and HTML describe data as a tree of nested elements with attributes and text. Python's xml.etree.ElementTree reads and writes XML, html.parser and BeautifulSoup read messy HTML, and any XML that comes from outside needs a hardened parser such as defusedxml.
Change the code and press Run (Ctrl+Enter). Predict what will happen first, then break it on purpose and read what changes. Your edits are saved and match the lesson page.
Ready for more? This phase has 7 hands-on exercises with automatic checks.
Practice questions
Write the code in the editor, run it, then open the model answer to compare.
Parse <cart><item name='pen' qty='2'/><item name='ink' qty='5'/></cart> with ET.fromstring and print the total of all qty values (it should print 7). Hint: attributes are strings, so use int().
Write a function quantities(text) that takes the same kind of cart text and returns a dictionary that maps each item name to its quantity as an int. An item without a qty attribute counts as 1. Test it with <cart><item name='pen' qty='2'/><item name='ink'/></cart>. Hint: get accepts a default value.
Use html.parser to collect the text of every <li> in <ul><li>a</li><li>b</li></ul> into a list and print it. Hint: make a subclass of HTMLParser with a flag that is on between <li> and </li>.
Use BeautifulSoup to print every href and its link text from <p><a href='/x'>X</a> and <a href='/y'>Y</a></p>, one pair per line. Hint: find_all("a"), then a["href"] and a.get_text().
Explain it without notes
What are the parts of an XML element, and how do you read each in ElementTree?
What is the difference between find, findall, findtext and iter, and what do they return when nothing matches?
Why can a search on a namespaced document return nothing, and how do you fix it?
Why should you not parse HTML with ElementTree or a regular expression, and what should you use instead?
What are the risks of parsing untrusted XML and how do you defend against them?
Expected output
title text: Today's deals
first link: /item/1 Pen
all links : ['/item/1', '/item/2']
css select: ['10', '45']
note text : Prices include tax.
missing : None
get default: none
[{'name': 'Pen', 'price': 10, 'url': '/item/1'}, {'name': 'Notebook', 'price': 45, 'url': '/item/2'}]