PHASE 9Intermediate ~26 min· topic 3 of 9

Topic 9.3

Generator Expressions and Lazy Pipelines

In one line

A generator expression looks like a list comprehension with round brackets, but it produces values one at a time instead of building a list. Chained together, generator expressions make memory-friendly pipelines, and they pair well with any, all and sum.

Think of it like this

A conveyor belt in a bakery. A list comprehension is like baking every loaf first and stacking them all in a room. A generator expression is a belt: each loaf is baked only when the packer at the end reaches for it. If the packer stops after ten loaves, the other ninety were never baked.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Generator expression: Code like (x * 2 for x in items) that makes a generator producing values on demand.
List comprehension: Code like [x * 2 for x in items] that builds the whole list at once.
Pipeline: Several steps chained so each item flows through all of them, one item at a time.
Stage: One step of a pipeline, such as stripping spaces or filtering errors.
Short-circuit: Stopping as soon as the answer is known. any stops at the first True.
Eager: Doing all the work immediately (a list comprehension). The opposite of lazy.
Consume: Pull values out of a generator, for example with for, sum or list.

Step by step

01Square brackets or round brackets

The only difference in the code is the brackets. The result is a different kind of object, and a different memory cost.

main.pywhole filepython

squares_list = [n * n for n in range(5)]   # a list, built now
squares_gen = (n * n for n in range(5))    # a generator, built later

print(squares_list)     # [0, 1, 4, 9, 16]
print(squares_gen)      # <generator object ...>  (not useful to print)
print(list(squares_gen))  # [0, 1, 4, 9, 16]

02Memory: one item or all of them

The generator's size does not depend on how many items it will produce, because it only stores the loop state. The list stores every result.

The first runnable example below checks sizes with sys.getsizeof. It prints True or False rather than the exact byte counts, because those numbers change between Python versions.

03Single-use, again

After sum(squares_gen) has consumed the generator, a second sum(squares_gen) returns 0, not an error. This is the bug from Topic 9.1 in a new costume.

If a value is going to be used twice, make it a list first, or write a small function that returns a fresh generator each time.

04any, all and sum with generator expressions

any and all take any iterable and stop as soon as the answer is known. sum(1 for ...) counts. max(..., default=0) avoids an error on empty input. "".join(...) accepts a generator too (but it builds a list internally, so a list comprehension is slightly faster there).

main.pywhole filepython

data = [3, 8, 1, 9]
print(any(x > 5 for x in data))        # True, stops at 8
print(all(x > 0 for x in data))        # True
print(sum(1 for x in data if x > 2))   # 3: how many are above 2
print(max((x for x in data if x > 100), default=0))   # 0

05Chaining stages into a pipeline

Each stage is a generator expression whose input is the stage before. Writing them as separate named steps keeps each one easy to read and easy to test.

In the log example, the three stages strip whitespace, keep only error lines, and cut off the first word. None of them runs until list(messages) pulls values through.

Chaining stages into a pipelinediagram

Rendering diagram…

06Lazy means errors appear late

If a stage has a bug (say, dividing by zero on one item), the error is raised only when that item reaches the consumer, not when the pipeline is built. A traceback from a pipeline points at the consumer line; look at the stages above it.

When a pipeline gets long or tricky, convert a stage into a named generator function (Topic 9.2). It gives you a name, a docstring and room for comments.

07A pipeline over a file

A file object is itself a lazy iterator over lines. So (line for line in open(path) if "ERROR" in line) reads the file one line at a time, and sum(1 for _ in that) counts the errors in a file of any size. Use with open(...) as f: around it so the file is closed (Topic 7.4).

Try it yourself

1
Feel the memory difference
In the first example, change range(100_000) to range(10_000_000) for the generator only (leave the list at 100,000). The generator line still returns at once. Do not try 10 million for the list in the browser; it would be slow.
2
Stop early with any
Write any(print(x) or x > 2 for x in [1, 2, 3, 4, 5]) and run it. It prints 1, 2, 3 and then stops: items 4 and 5 are never touched. (print returns None, so print(x) or x > 2 is just x > 2 with a side effect.)

Code & diagrams

List comprehension versus generator expression New tab

Change 100_000 to 10 and run it. The generator stays tiny either way. The second sum prints 0 because the generator was used up.

Expected output

list is big:       True
generator is tiny: True
type: generator
sum from list:      333328333350000
sum from generator: 333328333350000
sum again:          0
first n with n*n > 5000: 71

A lazy log pipeline New tab

Add a fourth stage that upper-cases the messages. Nothing runs until list(messages) is called.

Expected output

nothing has run yet: generator
messages: ['disk full', 'timeout']
any error?       True
all non-empty?   True
count of INFO:   3
total characters: 70

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Ask a generator expression for its length

Call len() on a generator expression.

main.pywhole filepython

squares = (n * n for n in range(5))
print(len(squares))

terminal

$ python main.py

── what you'll see ──

Traceback (most recent call last):

File "main.py", line 2, in <module>

print(len(squares))

~~~^^^^^^^^^

TypeError: object of type 'generator' has no len()

Myth vs fact

Myth

A generator expression is always faster than a list comprehension.

Fact

It uses less memory, but it is often slightly slower when you need all the values, because each value goes through the generator machinery. Choose it for memory and for early stopping.

Myth

You can loop over a generator expression as many times as you want.

Fact

It is an iterator and is used up after one pass. A second loop sees nothing.

Myth

(x for x in items) is a tuple comprehension.

Fact

There is no tuple comprehension. Round brackets make a generator. Use tuple(x for x in items) for a tuple.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
The leftmost for iterable of a generator expression is evaluated immediately, when the generator is created; everything else is evaluated lazily. So (x for x in broken_call()) fails at once if broken_call() fails, but an error in the filter or the expression appears at consumption time.
▸
Late binding (the same idea as in closures, Topic 5.7) bites generator expressions: a lazy expression that mentions an ordinary variable (not its own loop variable) reads that variable when it finally runs, not when it was written. If you change the variable in between, the expression sees the new value. Consume the generator before changing the variable, or pass the value in through a function parameter.
▸
For very hot loops, a list comprehension is usually faster than list(genexp), and a plain for loop with append is slower than both. Measure with timeit before optimising. For numeric work on large arrays, NumPy (Phase 14) beats all of them.

Remember this

1
A generator expression has the same shape as a list comprehension (Topic 4.7), with round brackets instead of square ones: (n * n for n in range(10)). It does not compute anything when written. It creates a generator object that computes one value at a time as a consumer asks. It supports the same pieces: a loop, an if filter, and several for clauses.
2
The big difference is memory. A list comprehension over a million items stores a million results at once. A generator expression stores almost nothing (its size does not depend on the number of items). If you only need to loop over the results once, as sum, max, join or a for loop do, a generator expression gives the same answer with far less memory.
3
When a generator expression is the only argument of a call, the extra brackets can be left out: sum(n * n for n in numbers) is the same as sum((n * n for n in numbers)). With two arguments you need the full brackets: max((x for x in data), default=0).
4
any(...) and all(...) work best with generator expressions because they stop early. any(x > 5 for x in data) returns True at the first match and never looks at the rest. all(...) stops at the first False. A list comprehension would compute every comparison before any even started. sum(1 for x in data if test(x)) is the standard way to count matching items.
5
A pipeline is several generator expressions chained: each takes the previous one as its input. Nothing runs until the last stage is consumed, and then each item flows through all stages one at a time. Reading a huge log file this way (strip, filter, parse) holds one line in memory, whatever the file size.
6
A generator expression is still a generator, so it is single-use, has no len(), cannot be indexed, and shows nothing useful if you print it. If you need the results twice, or need len or indexing, use a list comprehension. If the values are cheap and you need them once, use a generator expression. Choose by how the result is used, not by habit.

Explain it without notes

What is the difference between a list comprehension and a generator expression, and when do you choose each?

Why are generator expressions a good fit for any and all?

What is a lazy pipeline, and what are its benefits and its risks?

A function returns (line for line in f). Why might a caller be surprised the second time they loop over it?

Which part of a generator expression is evaluated immediately, and which part lazily? Why does it matter?

Practice

Use a generator expression with sum to add up the squares of the numbers 1 to 10.

Given words = ["apple", "kiwi", "banana", "fig"], use any and all to print whether any word has more than 5 letters, and whether all words have at least 3 letters.

Build a two-stage pipeline over [" a ", "b", " ", "c "]: first strip every string, then keep only non-empty ones. Print the result as a list.

Trade-offs

↔
Memory versus convenience. A generator expression saves memory but cannot be reused or measured. A list costs memory but is flexible. Decide from how many times, and how, the data is used.
↔
Readable stages versus clever one-liners. A long chain of nested generator expressions is hard to read. Name each stage, or switch to a generator function once the logic needs more than one line.

Done when you can

Done when you can write the same comprehension as a list and as a generator expression.
Done when you can explain why the second sum of a generator gives 0.
Done when you can use any, all and sum(1 for ...) with generator expressions.
Done when you can chain two or three stages into a pipeline and explain when each one runs.
Done when you can decide between a list and a generator for a given task.

Command Palette

Words you'll meet

Step by step

01Square brackets or round brackets

02Memory: one item or all of them

03Single-use, again

04any, all and sum with generator expressions

05Chaining stages into a pipeline

06Lazy means errors appear late

07A pipeline over a file

Try it yourself

Feel the memory difference

Stop early with any

Code & diagrams

Break it on purpose

Ask a generator expression for its length

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can