PHASE 7Intermediate ~37 min· topic 12 of 12

Topic 7.12

XML and HTML: Reading Markup

In one line

XML and HTML describe data as a tree of nested elements with attributes and text. Python's xml.etree.ElementTree reads and writes XML, html.parser and BeautifulSoup read messy HTML, and any XML that comes from outside needs a hardened parser such as defusedxml.

Think of it like this

A family tree, or a set of boxes with smaller boxes inside. A box labelled "library" holds boxes labelled "book", and each book holds a "title" box and an "author" box. Some boxes carry a sticker (an id or a year). To find something you open boxes from the outside in. Markup is that idea written as text: tags name the boxes and put them inside each other.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Markup: Text with tags that describe structure, such as XML or HTML.
Element: One box in the tree: a start tag, its content and an end tag.
Tag: The name of an element, written in angle brackets: <book>.
Attribute: A name="value" pair inside a start tag, such as id="b1".
Root: The single outermost element that holds everything else.
Tree: Elements nested inside elements, like a family tree.
Namespace: A label (a web address) that keeps tag names from different vocabularies apart.
CSS selector: A short pattern such as td.price that picks elements by tag, class or id.
defusedxml: A package that wraps Python's XML parsers and switches off dangerous features.
XXE and entity expansion: Attacks on XML parsers that make them read outside files or blow up memory. The defence is a hardened parser.

Step by step

01XML as a tree

Read the markup from the outside in. The root is library. Its children are two book elements and a magazine. A book has attributes id and year, and three children.

XML as a treediagram

Rendering diagram…

02Parse, then walk the tree

ET.fromstring returns the root. root.findall("book") looks only at the direct children called book, and book.find("title").text reads the text of one child. The price text "450.50" is a string, so float() turns it into a number. (The for loop goes over the list that findall returns, Topic 2.4.)

main.pywhole filepython

import xml.etree.ElementTree as ET

root = ET.fromstring("<library><book id='b1'><title>Learning Python</title><price>450.50</price></book></library>")
for book in root.findall("book"):
    print(book.get("id"), book.find("title").text, float(book.find("price").text))

terminal

$ python main.py

── expected output ──

b1 Learning Python 450.5

03Searching: find, findall, iter, and what they return

find gives one element or None. findall always gives a list (possibly empty). iter("title") walks the whole tree. To avoid AttributeError on a missing element, use findtext("book/isbn", default="n/a") or test for None. All of these are shown together in the first runnable example below.

An attribute search is book[@id='b2']. A deeper search from any element is .//title.

04Building and writing XML

Start with Element, add children with SubElement, set .text, indent, and write. The writer escapes & for you, so ink & paper is safe. Read the file back with ET.parse to check it round-trips.

The written file starts with an XML declaration (<?xml version='1.0' encoding='utf-8'?>), a first line that tells readers the encoding. As with any text file, pass encoding="utf-8" (Topic 7.5). The full code is the first runnable example below.

05Namespaces in one minute

Feeds, office documents and many industry formats put their tags in a namespace, written as an xmlns="..." attribute. ElementTree stores the address in the tag name: {http://www.w3.org/2005/Atom}feed. A plain search for entry finds nothing and raises no error, which is confusing the first time. Give find a dictionary that maps your short prefix to the address, or use the {*} wildcard.

06HTML: events and trees

HTMLParser calls your methods in document order: start tag, data, end tag. It never complains about unclosed tags, so unclosed still yields its text. You must keep your own state (a list of links, a flag that says "inside a script"). It is small and has no dependencies. BeautifulSoup trades that for convenience and a real tree. (A CSS selector is the same pattern language web designers use to style pages.)

Use a CSS selector in BeautifulSoup to say what you want: soup.select("td.price") means every td element with class price. Always give BeautifulSoup an explicit parser name such as "html.parser", so that results do not depend on which optional parser is installed.

main.pywhole filepython

from bs4 import BeautifulSoup

soup = BeautifulSoup("<p class='note'>Prices include <b>tax</b>.</p>", "html.parser")
print(soup.select_one("p.note").get_text())

terminal

$ python main.py

── expected output ──

Prices include tax.

07Safe parsing of untrusted XML

The defence is a different import, nothing more. defusedxml raises an exception (EntitiesForbidden, ExternalReferenceForbidden) when a document uses entity definitions or external references. Your code keeps using the same find / findall calls on the result. (The snippet uses a name untrusted_text that stands for XML you received; a fuller version is the "Untrusted XML" example below.)

Add two more habits. Limit the size you read (data = upload.read(1_000_000)). Never log or echo the document back. Machine-learning projects meet this when they ingest thousands of XML files (patents, papers, feeds) from outside, so the hardened parser belongs at the front door of the pipeline.

main.pywhole filepython

# pip install defusedxml
import defusedxml.ElementTree as ET   # same API as xml.etree.ElementTree

root = ET.fromstring(untrusted_text)  # raises an exception on dangerous constructs

Try it yourself

1
Add a book
In the first example, add a third book with a price that has no currency attribute. Run it and see the None that .get("currency") returns. Then change the loop to print "?" when the currency is missing.
2
Match the namespace
In the namespace example, remove the ns argument from the first findall("a:entry", ns) call and read the error text. Then fix it with the {*} form.
3
Pick the links
In the BeautifulSoup example, write a selector that returns only the links inside the table (#prices a). Then change it to select the whole row of the item called Notebook. Predict how many elements you get each time.

Code & diagrams

Read, search, build and write XML New tab

Add a third book to the text and see every list grow. Then ask for a tag that does not exist, to see None.

Expected output

root: library {'name': 'City Library'}
children: ['book', 'book', 'magazine']
b1 2019 Learning Python 450.5 INR
b2 2022 Data at Work 620.0 INR
all titles: ['Learning Python', 'Data at Work', 'Weekly Code']
path search: Asha Rao
by attribute: Data at Work
missing: None
missing with default: n/a
<catalog version="1">
  <item name="pen">10</item>
  <item name="ink &amp; paper">3</item>
</catalog>
<?xml version='1.0' encoding='utf-8'?>
<catalog version="1">
  <item name="pen">10</item>
  <item name="ink &amp; paper">3</item>
</catalog>
[('pen', 10), ('ink & paper', 3)]
ParseError: mismatched tag: line 1, column 8

Namespaces: why find() returns nothing New tab

The empty list is not an error. Compare the tag Python shows with the tag you typed.

Expected output

tag as parsed: {http://www.w3.org/2005/Atom}feed
plain search finds: []
First post by Asha
Second post by Ben
Hectal news
any namespace: ['First post', 'Second post']

HTML with the standard library: html.parser New tab

The parser reads tags in order and never complains. Notice that the second paragraph is never closed and its text still arrives.

Expected output

links: ['/item/1', '/item/2']
text : ['Shop', "Today's deals", 'Fresh & cheap.', 'Hurry', '!', 'Pen', '10', 'Notebook', '45']
start p [('class', 'x')]
data  'Hi & '
start br []
data  'bye'
end   p
start p []
data  'unclosed'

HTML with BeautifulSoup: find, select, extract a table New tab

BeautifulSoup is a separate package. Here it loads by itself, because the page imports it. On your computer run pip install beautifulsoup4 first.

Expected output

title text: Today's deals
first link: /item/1 Pen
all links : ['/item/1', '/item/2']
css select: ['10', '45']
note text : Prices include tax.
missing   : None
get default: none
[{'name': 'Pen', 'price': 10, 'url': '/item/1'}, {'name': 'Notebook', 'price': 45, 'url': '/item/2'}]

Untrusted XML: the hardened parserpython

Not runnable here: defusedxml is a separate package that is not part of this page's runner. Install it with pip install defusedxml. The only change is the import. A document that defines entities or external references is refused with an exception instead of being expanded.

# pip install defusedxml
import defusedxml.ElementTree as ET
from defusedxml import DefusedXmlException


def parse_upload(data: bytes):
    if len(data) > 1_000_000:                  # limit the size first
        raise ValueError("file too large")
    try:
        return ET.fromstring(data)
    except DefusedXmlException:                # a forbidden construct, such as an entity definition
        raise ValueError("this XML uses features that are not allowed")


root = parse_upload(b"<order><item sku='A1'>2</item></order>")
print(root.find("item").get("sku"), root.find("item").text)

Output

A1 2

Which tool for which markupdiagram

Rendering diagram…

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Take .text of a missing element

Search for a child that is not there and read its text straight away.

main.pywhole filepython

import xml.etree.ElementTree as ET

root = ET.fromstring("<book><title>Learning Python</title></book>")
print(root.find("author").text)

terminal

$ python main.py

── what you'll see ──

Traceback (most recent call last):

File "main.py", line 4, in <module>

print(root.find("author").text)

^^^^^^^^^^^^^^^^^^^^^^

AttributeError: 'NoneType' object has no attribute 'text'

Break #2

Broken XML

Close the tags in the wrong order.

main.pywhole filepython

import xml.etree.ElementTree as ET

ET.fromstring("<a><b></a>")

terminal

$ python main.py

── what you'll see ──

xml.etree.ElementTree.ParseError: mismatched tag: line 1, column 8

Myth vs fact

Myth

HTML is just XML, so I can parse web pages with ElementTree.

Fact

Real pages have unclosed tags, bare ampersands and attributes without quotes, and ElementTree raises ParseError on them. Use html.parser or BeautifulSoup.

Myth

A regular expression is enough to pull data out of HTML.

Fact

Markup nests to any depth and changes shape between pages; a regex breaks on the first unusual space or attribute order. Use a parser and select by structure.

Myth

Built-in XML parsing is safe because it is the standard library.

Fact

The Python documentation lists the standard XML modules as vulnerable to maliciously built data (memory exhaustion and, depending on the parser, reading outside resources). Parse untrusted XML with defusedxml.

Myth

.text gives me a number if the element contains digits.

Fact

.text is always a string. Convert it yourself and handle ValueError for bad data.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
ElementTree loads the whole tree into memory. For a very large file use ET.iterparse(path, events=("end",)), handle each element as it closes, then call elem.clear() to release it. This keeps memory flat for gigabyte files, such as a Wikipedia dump or a patent archive that feeds a model's training data.
▸
BeautifulSoup is only an interface over a parser. "html.parser" is built in and lenient; "lxml" is much faster and also repairs broken markup differently; "html5lib" follows the HTML5 rules the way a browser does but is slow. Different parsers can build different trees from broken input, so name the parser explicitly and pin its version for repeatable results.
▸
Many sites build their content with JavaScript after the page loads, so the HTML you download does not contain it. Look for the JSON the page itself calls (the browser's developer tools show it), and prefer an official API (Topic 15.10). A headless browser is the last resort.
▸
Defensive settings in practice. With defusedxml, fromstring(data, forbid_dtd=True) rejects any document that has a DOCTYPE at all (it raises DTDForbidden), which is the strictest and usually right choice for API payloads, because legitimate data rarely needs a DTD. By default it already forbids entity definitions and external references. If you use lxml, build the parser explicitly with resolve_entities=False and no_network=True and leave huge_tree off. Also bound the decompressed size when the upload is gzip or zip, since a small archive can unpack to a huge file.
▸
With iterparse, elem.clear() frees the element's children, but the emptied element stays attached to its parent, so on a huge file also clear the root (or the parent) every so often, or memory still creeps up. .tail holds text after an element's closing tag, which matters for mixed content such as Hi there friend.
▸
XML Schema, XPath and XSLT are heavier tools for large enterprise systems; the lxml package supports them fully. ElementTree understands only a subset of XPath, which is enough for most work.

Remember this

1
An element is one box: <book id="b1"><title>Learning Python</title></book>. It has a tag (book), optional attributes (id="b1", written name equals quoted value), text (Learning Python) and children (other elements inside it). The one outermost element is the root. The whole thing forms a tree. XML (eXtensible Markup Language) is a strict format for data: every tag must close and tags must nest properly. HTML, the language of web pages, looks similar but is forgiving. Browsers accept unclosed tags like   and  without an end.
2
xml.etree.ElementTree (usually imported as ET, Topic 6.1) is the standard-library XML tool. ET.fromstring(text) parses text and returns the root element. ET.parse("file.xml").getroot() reads a file. On an element, .tag is the name and .attrib is a dictionary of attributes (Topic 4.4). .get("id") reads one attribute (or None), and .text is the text before the first child. Loop over an element (for child in root) to see its direct children.
3
To search the tree, use these. find(path) gives the first match or None. findall(path) gives a list. findtext(path, default) gives the text, with a default. iter(tag) gives every element of that name at any depth. The path language is small. book/title goes down one level at a time, .//title searches everywhere below, and book[@id='b2'] filters by attribute. **.text is always a string**, so convert numbers yourself with int() or float(). And find returns None when nothing matches, so root.find("x").text raises AttributeError when x is missing.
4
You can also build XML. ET.Element("catalog", {"version": "1"}) makes an element, and ET.SubElement(parent, "item", {...}) adds a child. ET.indent(root) (Python 3.9) pretty-prints. ET.tostring(root, encoding="unicode") gives text. ET.ElementTree(root).write("file.xml", encoding="utf-8", xml_declaration=True) writes a file. Characters like & and < are escaped for you (ink & paper). That is why you should build XML with this API and never by gluing strings together.
5
A namespace is a label that stops two vocabularies from clashing. Inside the tree a namespaced tag becomes {uri}name, and plain find("entry") finds nothing. Pass a prefix map (find("a:entry", ns)) or use {*}entry.
6
For HTML, do not use ElementTree: real pages are not valid XML. The standard library's html.parser.HTMLParser is an event reader. You subclass it (make your own version of it, as in Topic 8.1) and fill in handle_starttag(tag, attrs), handle_endtag(tag) and handle_data(text). You call feed(page) and your methods are called as it reads. It does not build a tree and does not repair anything. BeautifulSoup (from bs4 import BeautifulSoup, installed with pip install beautifulsoup4) builds a tree from messy HTML and gives friendly search: soup.find_all("a"), soup.select("td.price") (CSS selectors), tag["href"], tag.get_text(strip=True). In this course's browser, BeautifulSoup is available. It is the usual choice for scraping (Topic 15.10) and for cleaning web text for AI datasets.
7
Security. XML from outside (an upload, an API, a file from a stranger) is untrusted input. The XML standard has features that can expand a tiny document into gigabytes of memory (entity expansion, sometimes called a billion laughs attack) and, in some parsers and settings, make the parser read local files or contact other addresses on your network (external entities, XXE). ElementTree on its own does not fetch external entities, and the expat library in current Python releases limits the worst expansion, but that protection depends on which expat version your Python uses, and other parsers and settings behave differently. The Python documentation still lists the standard-library XML modules as not secure against maliciously constructed data, so do not rely on luck.
8
The defence is simple. Parse untrusted XML with **defusedxml** (import defusedxml.ElementTree as ET is a drop-in replacement). Set a size limit before parsing. Prefer JSON when you control the format (Topic 7.7). Never feed untrusted markup into eval or build XML by joining strings.

Explain it without notes

What are the parts of an XML element, and how do you read each in ElementTree?

What is the difference between find, findall, findtext and iter, and what do they return when nothing matches?

Why can a search on a namespaced document return nothing, and how do you fix it?

Why should you not parse HTML with ElementTree or a regular expression, and what should you use instead?

What are the risks of parsing untrusted XML and how do you defend against them?

Practice

Parse <cart><item name='pen' qty='2'/><item name='ink' qty='5'/></cart> with ET.fromstring and print the total of all qty values (it should print 7). Hint: attributes are strings, so use int().

Write a function quantities(text) that takes the same kind of cart text and returns a dictionary that maps each item name to its quantity as an int. An item without a qty attribute counts as 1. Test it with <cart><item name='pen' qty='2'/><item name='ink'/></cart>. Hint: get accepts a default value.

Use html.parser to collect the text of every <li> in <ul><li>a</li><li>b</li></ul> into a list and print it. Hint: make a subclass of HTMLParser with a flag that is on between <li> and </li>.

Use BeautifulSoup to print every href and its link text from <a href='/x'>X</a> and <a href='/y'>Y</a>, one pair per line. Hint: find_all("a"), then a["href"] and a.get_text().

Trade-offs

↔
ElementTree versus lxml. ElementTree is built in and enough for most jobs. lxml is faster, supports full XPath, XSLT and schema validation, but is a compiled dependency.
↔
html.parser versus BeautifulSoup. html.parser has no dependencies and gives events for simple extraction. BeautifulSoup saves a lot of code on real pages with its tree and selectors, at the price of a dependency and a little speed.
↔
Whole tree versus streaming. Loading a tree is easy to search but needs memory the size of the document. iterparse handles huge files but you write the logic as events.

Done when you can

Done when you can describe an element's tag, attributes, text and children, and the root of a tree.
Done when you can parse XML with ElementTree and use find, findall, findtext, iter and attributes safely.
Done when you can build an XML document and write it to a file.
Done when you can explain why a namespaced search returns nothing and fix it.
Done when you can extract links, text and table rows from HTML with html.parser and BeautifulSoup.
Done when you can explain the risks of untrusted XML and name defusedxml.

Command Palette

Words you'll meet

Step by step

01XML as a tree

02Parse, then walk the tree

03Searching: find, findall, iter, and what they return

04Building and writing XML

05Namespaces in one minute

06HTML: events and trees

07Safe parsing of untrusted XML

Try it yourself

Add a book

Match the namespace

Pick the links

Code & diagrams

Break it on purpose

Take .text of a missing element

Broken XML

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can