PHASE 9Intermediate ~29 min· topic 2 of 9

Topic 9.2

Generators and yield

In one line

A generator function uses yield to hand back values one at a time and freezes in place between them. It writes an iterator in a few lines, keeps its own state, and can produce infinite streams without using memory.

Think of it like this

A vending machine that makes one cup of tea each time you press the button, compared with a machine that makes a hundred cups in advance and lets them go cold. A generator makes the next value only when it is asked, then waits. It never prepares more than you asked for.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Generator function: A function that contains yield. Calling it returns a generator object instead of running the body.
Generator object: The iterator you get by calling a generator function. It runs the body step by step as you call next().
yield: Hand back a value and pause the function there, keeping all its local variables.
Resume: Continue a paused generator from the line after its last yield.
yield from: Yield every value from another iterable, one by one, in place.
send(): Resume a generator and also give it a value, which becomes the result of the paused yield expression.
Infinite stream: A generator that never stops by itself, so the consumer decides when to stop.
Streaming: Handling data piece by piece as it arrives, instead of waiting for all of it.

Step by step

01A function that pauses

A normal function runs to the end and returns once. A generator function can return many times, because each yield returns a value and pauses.

The print lines in the first example run in a surprising order. Watch for it: calling the function prints nothing; the body starts at the first next().

main.pywhole filepython

def count_up_to(limit):
    n = 1
    while n <= limit:
        yield n        # hand back n, then freeze here
        n += 1         # runs when the next value is asked for

for value in count_up_to(3):
    print(value)       # 1, 2, 3

02The life of a generator

A generator moves through four states: created (nothing has run), suspended (paused at a yield), running, and closed (finished). next() moves it from created or suspended to running, until the next yield or the end.

When the body ends, Python raises StopIteration for the caller. The value of a return statement in a generator becomes the .value of that StopIteration, but for loops ignore it.

The life of a generatordiagram

Rendering diagram…

03Why the generator is lazy

Because the body only runs between next() calls, no work is done until a value is needed. If the consumer stops after three values, the generator never computes the fourth. This is lazy evaluation.

The first runnable example below prints a line before and after each yield, so you can see the interleaving of the generator body and the caller.

04Infinite streams

A while True loop with a yield inside is a perfectly good generator. It will never end by itself, so the consumer must stop it. itertools.islice(gen, 8) takes only the first eight values.

Never call list() on an infinite generator: it tries to collect values forever and your program runs out of memory.

main.pywhole filepython

def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

from itertools import islice
print(list(islice(fibonacci(), 8)))   # [0, 1, 1, 2, 3, 5, 8, 13]

05yield from: chaining generators

yield from stream is a shorter form of for item in stream: yield item. Use it to put something before or after another stream. In the example below it adds a [DONE] marker after every chunk of a fake model reply.

It matters most for recursion (walking a tree and yielding every node) and for code that needs send and close to reach the inner generator.

main.pywhole filepython

def with_end_marker(stream):
    yield from stream      # same as: for chunk in stream: yield chunk
    yield "[DONE]"

06send and close, briefly

yield is an expression, so it can have a value: value = yield total. The caller sets that value with gen.send(x). The first call must be next(gen) to run to the first yield; sending a non-None value to a just-created generator is an error.

gen.close() ends the generator by raising GeneratorExit inside it at the paused yield. A try/finally around the yield runs its finally block then, which is how a generator releases a file or a connection when the consumer quits early.

07A generator is single-use, like every iterator

A generator object is an iterator, so Topic 9.1 applies in full: once finished, it stays finished. To start again, call the generator function again to get a new generator object.

A generator also holds its paused local variables in memory until it is finished or deleted. That is small for one generator, but a program that keeps a million suspended generators alive uses real memory.

Try it yourself

1
Change the order of the output
In the first example, change count_up_to(2) to count_up_to(1) and remove one of the print("got", next(gen)) lines. Predict which bracketed lines will now appear, then run it. The "finished" line only appears when you ask for a value past the last yield.
2
Take more from the stream
In the second example, change islice(fibonacci(), 8) to islice(fibonacci(), 20). Python integers never overflow, so you can ask for hundreds of values; the generator still uses almost no memory.
3
Stream early and stop
Add a break after the second chunk in the for chunk in ... loop. The [DONE] marker is never produced, because the generator is never asked for it.

Code & diagrams

Watching a generator pause and resume New tab

The lines in brackets come from inside the generator. Notice that creating the generator prints nothing, and each next() runs only up to the next yield.

Expected output

created the generator, nothing has run yet
  (start)
  (about to yield 1)
got 1
  (resumed after 1)
  (about to yield 2)
got 2
  (resumed after 2)
  (finished)
StopIteration: the generator is done

Infinite streams, yield from and send New tab

fake_model_stream stands in for a real model API that streams its answer. The consuming loop is the same shape as real client code.

Expected output

first 8 Fibonacci numbers: [0, 1, 1, 2, 3, 5, 8, 13]
received: 'Generators '
received: 'stream '
received: 'tokens '
received: '[DONE]'
5
15
closed

Generator statesdiagram

Rendering diagram…

A realistic generator: clean lines from a file New tab

Counters are a toy. Real programs use generators to read big files one line at a time, cleaning each line as it goes. io.StringIO acts like an open file here, so no real file is needed. Notice that nothing is read until next() asks for it.

Expected output

nothing read yet: generator
alpha
beta
the rest: ['gamma']

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Ask a finished generator for more

Call next once too often on a generator.

main.pywhole filepython

def one_item():
    yield "only"


gen = one_item()
print(next(gen))
print(next(gen))

terminal

$ python main.py

── what you'll see ──

only

Traceback (most recent call last):

File "main.py", line 7, in <module>

print(next(gen))

~~~~^^^^^

StopIteration

Break #2

Send a value to a generator that has not started

Call send(5) right after creating the generator.

main.pywhole filepython

def running_total():
    total = 0
    while True:
        value = yield total
        total += value


acc = running_total()
acc.send(5)

terminal

$ python main.py

── what you'll see ──

Traceback (most recent call last):

File "main.py", line 9, in <module>

acc.send(5)

~~~~~~~~^^^

TypeError: can't send non-None value to a just-started generator

Myth vs fact

Myth

Calling a generator function runs its code.

Fact

It only creates the generator object. The code runs a piece at a time when next() is called.

Myth

Generators are only for big data.

Fact

They are also the shortest way to write any custom iterator, and the natural way to write pipelines and streams, even for ten items.

Myth

yield and return do the same thing.

Fact

return ends the function for good. yield hands back a value and pauses, so the function can continue later.

Myth

A generator can be looped over again to get the same values.

Fact

It is an iterator and is used up after one pass. Call the function again for a fresh generator.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
A generator function compiles to a normal code object with the generator flag set. Calling it builds a frame (the saved locals and instruction pointer) wrapped in a generator object. yield suspends that frame without freeing it, and next() resumes it. Per item a generator is usually a little slower than a list comprehension (there is a resume for every value), but it needs memory for one item instead of all of them, and it can stop early. Choose it for memory and laziness, not for raw speed.
▸
inspect.getgeneratorstate(gen) reports GEN_CREATED, GEN_SUSPENDED, GEN_RUNNING or GEN_CLOSED. A generator cannot be resumed from inside itself: calling next() on a generator that is running raises ValueError: generator already executing. gen.throw(SomeError) raises the error at the paused yield, which lets the generator catch it and carry on (it then yields the next value), and gen.close() raises GeneratorExit there. A generator that catches GeneratorExit and yields again makes close() fail with RuntimeError: generator ignored GeneratorExit. Python also calls close() for you when a suspended generator is garbage collected, which is what runs its finally block.
▸
Since Python 3.7 (PEP 479) a StopIteration that leaks out of a generator body is turned into RuntimeError: generator raised StopIteration. Before that, a stray next(it) inside a generator could silently end it early and hide a bug. To end a generator early, use a plain return, never raise StopIteration. The same trap hides inside a helper: a next(iterator) with no default, called inside a generator, will turn an empty iterator into this RuntimeError, so give it a default or catch StopIteration.
▸
The return value in a generator becomes StopIteration.value, and result = yield from inner() receives it. This was the foundation of the first coroutines; async/await (Phase 12) builds on the same suspend-and-resume idea with its own syntax.
▸
Always put cleanup in try/finally around the yield, or use a context manager (Topic 9.8). Without it, a consumer that stops early leaves the file or connection open until the generator is garbage collected. In CPython that happens as soon as the last reference goes (reference counting), but other implementations, or a generator stored in a long-lived structure, can delay it. For a guarantee, close the generator explicitly or wrap it in contextlib.closing(gen).
▸
yield from inner is more than a loop: it forwards send(), throw() and close() to inner, and the whole expression evaluates to the value of inner's return. A recursive generator with yield from (walking a tree) therefore stays short. Async code has its own version, async def with yield (an async generator, read with async for), which Topic 16.5 uses for streaming.

Remember this

1
Recall from Topic 9.1: an iterator hands out one value per next() call and raises StopIteration when it is finished. A generator function looks like a normal function, but it contains the word yield. Calling it does not run the body. It returns a generator object (an iterator) and nothing else happens yet. The body starts running only when something calls next() on that object, which a for loop does for you.
2
When the body reaches yield value, the function hands value back and freezes exactly where it is: local variables, the position in the loop, everything. The next next() call resumes on the line after the yield. When the body ends (or reaches return), the generator raises StopIteration for you. This is the "state between yields" that makes generators so convenient.
3
Because values are made on demand, a generator can describe an infinite stream. A generator that yields the Fibonacci numbers forever uses almost no memory, as long as the consumer stops after taking what it needs (itertools.islice, which takes the first few values and is explained in Topic 9.4; a break; or next() a few times). A list of infinite length is impossible; a generator of infinite length is easy.
4
yield from other hands over to another iterable: every value from other is yielded in turn, as if you had written a for loop with a yield inside. It is shorter, and it also passes send() and close() through to the inner generator. It is the standard way to chain or flatten generators.
5
Generators can also receive values. value = yield result pauses, and the caller can resume with gen.send(x), which makes the paused yield expression evaluate to x. The first call must be next(gen) (or send(None)) to run up to the first yield. gen.close() stops the generator early by raising GeneratorExit at the paused yield. You will rarely need send; plain yield covers most uses.
6
Generators are how streamed model output is read. An API client gives you a generator that yields chunks of text as the model writes them. Your code loops over it and prints each chunk the moment it arrives, so the user sees the answer grow instead of waiting for the end (Topic 16.5 shows the real client). The example below uses a fake model so it runs here.

Explain it without notes

What happens when you call a generator function, and when does its body actually run?

How does a generator keep its state between values, and how is that different from an iterator class?

What is yield from for, and what does it do that a loop with yield does not?

Why can a generator describe an infinite sequence, and how do you use one safely?

What do return, raise StopIteration, close() and throw() each do inside or to a generator?

Practice

Write a generator squares(n) that yields the squares of 1 up to n. Print list(squares(5)).

Write a generator countdown(n) that yields n, n-1, ... down to 1, then the text "liftoff". Print each value on its own line with a for loop.

Write a generator chunks_of_text(text, size) that yields the text in pieces of at most size characters, then print the pieces for "streaming" with size 4.

Trade-offs

↔
Lazy and small, but single-use and not indexable. A generator holds one value at a time, so you cannot ask for len(), slice it, or loop twice. Convert to a list when you need those.
↔
Errors surface late. Because the body runs only when consumed, a bug inside the generator appears at the first next(), far from the line that created it. Test generators by consuming them fully in a test.

Done when you can

Done when you can say what happens at the call of a generator function and at each next().
Done when you can write a generator with a loop and a yield.
Done when you can stop an infinite generator safely with islice or break.
Done when you can use yield from to chain two generators.
Done when you can explain what send and close do, and when each is allowed.

Command Palette

Words you'll meet

Step by step

01A function that pauses

02The life of a generator

03Why the generator is lazy

04Infinite streams

05yield from: chaining generators

06send and close, briefly

07A generator is single-use, like every iterator

Try it yourself

Change the order of the output

Take more from the stream

Stream early and stop

Code & diagrams

Break it on purpose

Ask a finished generator for more

Send a value to a generator that has not started

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can