Regular Expressions with re

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Regular expression (regex)

A pattern written in a small special language that describes a shape of text.

Raw string

A string with an r before the quote, in which backslashes are kept as they are: r"\d".

Metacharacter

A character with a special meaning in a pattern, such as ., *, +, ?, [, (, \, ^, $.

Quantifier

A symbol that says how many times to repeat the thing before it.

Group

Part of a pattern in round brackets whose matched text can be read back afterwards.

Match object

What a successful search returns. It holds the matched text, its position and the groups.

Greedy and lazy

Greedy takes as much text as possible. Lazy (with ?) takes as little as possible.

Flag

An option, such as re.IGNORECASE, that changes how a pattern matches.

Backtracking

The engine's way of trying another way to match when the first way fails.

1. 1

   Extract the order ids

   In the first example, change the pattern in findall so it also matches lower-case ids such as a-9. There are two ways: write the character class [A-Za-z], or add the flag re.IGNORECASE. Try both.

2. 2

   Greedy against lazy

   Run the HTML example with <.+> and <.+?>. Then try <[^>]+>, which says "anything except a closing bracket". This third form is often the clearest way to avoid greediness.

3. 3

   Rearrange with groups

   Use re.sub(r"(\w+), (\w+)", r"\2 \1", "Lovelace, Ada") to turn "Last, First" into "First Last". Try a name with a middle name and see what the pattern misses.

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

import re

print(re.findall("\d+", "a1b22"))

import re

m = re.search(r"\d+", "no digits here")
print(m.group())

1. 1

   Patterns are written in raw strings: put an r before the opening quote, as in r"\d+". In a normal string Python treats a backslash as the start of its own escape ("\n" is a newline). The regex language also uses backslashes (\d means "a digit"). A raw string tells Python to leave backslashes alone, so the pattern arrives unchanged. Without it, "\d" is an invalid escape and recent Python versions print a SyntaxWarning (a warning message, not a crash). **Make r"..." a habit for every pattern.**

2. 2

   The building blocks. Most characters match themselves. . matches any one character except a newline. \d is a digit, \w a letter, digit or underscore, \s whitespace; the capital versions (\D, \W, \S) mean the opposite. [abc] is any one of a, b, c; [a-z] a range; [^abc] anything but those. ^ and $ mean the start and the end, and \b a word boundary. To match a symbol literally, put a backslash before it (\. is a real dot); re.escape(text) does this for a whole string.

3. 3

   Quantifiers say how many: * zero or more, + one or more, ? zero or one, {4} exactly four, {2,5} two to five. Groups use round brackets: (\d{4}) captures the text it matches so you can read it afterwards with group(1). A named group (?P<year>\d{4}) is read as group("year"), and (?:...) groups without capturing. | means "or": cat|dog.

4. 4

   Choose the function that matches your question. re.search(p, s) finds the first match anywhere in s. re.match(p, s) only matches at the start. re.fullmatch(p, s) requires the whole string to match, which is what you want for validation. re.findall(p, s) returns all matches as a list, re.finditer returns them one at a time with positions, re.sub(p, repl, s) replaces them, and re.split(p, s) cuts at them. Searching functions return a match object (or None), which you must check before using. Calling .group() on None is the classic crash.

5. 5

   The re module comes with Python, and you load it with import re (Phase 6 explains import). The "engine" is the part of re that does the matching.

6. 6

   Greedy quantifiers take as much as they can; lazy ones (a ? after the quantifier: *?, +?) take as little as they can. On "<b>bold</b> and <i>italic</i>", the pattern <.+> is greedy and swallows everything from the first < to the last >, while <.+?> stops at the first > and finds each tag.

7. 7

   Flags change the rules: re.IGNORECASE ignores case, re.MULTILINE makes ^ and $ work per line, re.DOTALL lets . match newlines and re.VERBOSE lets you spread a pattern over lines with comments.

8. 8

   re.compile(pattern, flags) builds the pattern once and returns a pattern object with the same methods (search, findall, ...). Use it when a pattern is used many times or is long enough to deserve a name. And a safety topic: some patterns can take a huge amount of time on some inputs (the time can double with every extra character). This is catastrophic backtracking, covered in its own step below.

• ↔

  Power against readability. A regex can say in one line what takes ten lines of string methods, but a regex nobody can read is a liability. Use re.VERBOSE, named groups and comments, and prefer plain methods when they are enough.

• ↔

  Flexibility against safety. Backtracking engines support rich patterns but have a bad worst case. For text from strangers, keep patterns simple and inputs short.

• Done when you can explain why patterns are raw strings.

• Done when you can choose between search, match, fullmatch, findall, sub and split.

• Done when you can read and write groups, named groups, character classes and quantifiers.

• Done when you can explain greedy against lazy and fix an over-greedy pattern.

• Done when you can check a match for None before using it.

• Done when you can recognise a pattern with nested quantifiers and say how to defend against catastrophic backtracking.