PHASE 9Intermediate ~29 min· topic 1 of 9

Topic 9.1

How for Really Works: Iterables and Iterators

In one line

A for loop asks an object for an iterator with iter(), then calls next() on it again and again until it raises StopIteration. Knowing the difference between an iterable (can be looped over) and an iterator (remembers its place, used up once) explains many surprises.

Think of it like this

A book and a bookmark. The book can be read many times by many people. A bookmark remembers where one reader is up to; when that reader reaches the last page, that bookmark has nothing more to give. In Python the book is an iterable (a list, a string, a file) and the bookmark is an iterator. Every for loop makes a fresh bookmark for the book.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Iterable: Any object you can loop over. It has an __iter__ method that hands out an iterator. Lists, strings and dictionaries are iterables.
Iterator: An object that gives out one value at a time with next() and remembers where it is. It is used up once it reaches the end.
iter(): The built-in that asks an iterable for a fresh iterator.
next(): The built-in that asks an iterator for its next value.
StopIteration: The special exception an iterator raises to say "nothing left". A for loop catches it for you.
Protocol: An agreement about which methods an object has. The iteration protocol is __iter__ and __next__.
Exhausted: Used up. An iterator that has already given all its values is exhausted.
Lazy: Producing a value only when it is asked for, not all at once in advance.

Step by step

01A for loop, written by hand

Here is what for n in numbers: really does. We ask the list for an iterator with iter, then pull values with next until Python raises StopIteration.

The while True loop below is a fair copy of the real thing. Python's for does the same job in C, faster, and also catches StopIteration for you.

main.pywhole filepython

numbers = [10, 20, 30]

iterator = iter(numbers)        # step 1: get a bookmark
while True:
    try:
        n = next(iterator)      # step 2: ask for the next value
    except StopIteration:       # step 3: nothing left, stop
        break
    print(n)

A for loop, written by handdiagram

Rendering diagram…

02Iterable versus iterator

A list is an iterable, so iter(list) gives a new iterator object each time. The iterator is a separate object that stores a position. Two iterators over the same list do not disturb each other.

For an iterator, iter(it) returns the same object (it itself). That is how an iterator can be used wherever an iterable is expected, such as in a for loop.

main.pywhole filepython

numbers = [10, 20, 30]
it = iter(numbers)

print(type(numbers).__name__)    # list
print(type(it).__name__)         # list_iterator
print(iter(it) is it)            # True: an iterator is its own iterator
print(iter(numbers) is numbers)  # False: a list hands out new iterators

03StopIteration is a signal, not a crash

StopIteration is how an iterator says "I am finished". Inside a for loop you never see it, because the loop catches it. If you call next yourself, you must handle it, or pass a default as the second argument: next(it, None) returns None instead of raising.

Use the default form when you want "the first match or nothing", for example next((x for x in items if x > 5), None). (The part in the inner brackets is a generator expression, Topic 9.3: it lazily produces the items above 5.)

main.pywhole filepython

it = iter([1, 2])
print(next(it, "empty"))   # 1
print(next(it, "empty"))   # 2
print(next(it, "empty"))   # empty: no error, thanks to the default

04Writing your own iterator class

An iterator class needs two methods. __iter__ returns self. __next__ returns the next value or raises StopIteration. The object keeps its own state (here self.current) between calls.

Notice what the class has to do by hand: store the position, update it, and know when to stop. Generators (Topic 9.2) do all of that for you.

main.pywhole filepython

class Countdown:
    def __init__(self, start):
        self.current = start

    def __iter__(self):
        return self

    def __next__(self):
        if self.current <= 0:
            raise StopIteration
        value = self.current
        self.current -= 1
        return value

print(list(Countdown(3)))   # [3, 2, 1]

05One-shot: why the second loop is empty

A Countdown is an iterator, so after one pass it sits at 0 forever. A second list(timer) calls __next__, which immediately raises StopIteration, so the result is []. There is no error, which is why this bug is easy to miss.

If you want a thing that can be looped over many times, make it an iterable: write __iter__ so that it returns a fresh iterator each call. A list does exactly this.

main.pywhole filepython

timer = Countdown(3)
print(list(timer))   # [3, 2, 1]
print(list(timer))   # []  the bookmark is already at the end

06Reusable iterables: a fresh iterator each time

When __iter__ returns a new iterator instead of self, every loop starts from the beginning. The class stores the data; the iterators store the positions. This is the book-and-bookmark split from the analogy.

Returning iter(self.songs) is the shortest way: it reuses the list's own iterator. A generator method (yield, Topic 9.2) is the other common way.

main.pywhole filepython

class Playlist:
    def __init__(self, songs):
        self.songs = songs

    def __iter__(self):
        return iter(self.songs)    # a new bookmark every time

playlist = Playlist(["intro", "verse"])
print(list(playlist))   # ['intro', 'verse']
print(list(playlist))   # ['intro', 'verse']  works again

07What else uses the protocol

Unpacking (a, b = pair), *args calls, in on a container without __contains__, zip, enumerate, sum, sorted and list() all call iter() on their argument. Dictionaries iterate over their keys; files iterate over their lines; strings iterate over their characters.

Dictionary iteration has one more rule: changing a dictionary's size while looping over it raises RuntimeError: dictionary changed size during iteration. Loop over list(d) (a copy) if you need to delete keys as you go.

Try it yourself

1
Predict, then run
In the first example, add print(next(it)) after the three prints and run it. You will see a traceback ending in StopIteration. Now change it to print(next(it, "empty")) and run it again.
2
Make Countdown reusable
In the second example, rewrite Countdown so that it is an iterable: remove __next__, and make __iter__ return iter(range(self.start, 0, -1)). Run it. The second pass now prints [3, 2, 1] too.
3
Loop over a dictionary and a string
Write for x in {"a": 1, "b": 2}: and for ch in "hi": with a print(x) inside each. Predict the output first: keys for the dictionary, characters for the string.

Code & diagrams

iter and next by hand New tab

Add a fourth number to the list and one more next(it) call. Predict what happens before you run.

Expected output

list type: list
iterator type: list_iterator
10
20
30
StopIteration: the iterator is used up
iter(it) is it: True
iter(numbers) is numbers: False
two iterators are separate: True

Iterator class versus iterable class New tab

Countdown is used up after one pass. Playlist gives a fresh iterator every time. Change Playlist.__iter__ to return self and see what breaks.

Expected output

first pass:  [3, 2, 1]
second pass: []
first pass:  ['intro', 'verse', 'chorus']
second pass: ['intro', 'verse', 'chorus']
tick 2
tick 1

The loop in one picturediagram

Rendering diagram…

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Call next on a list

Call next directly on a list instead of on an iterator.

main.pywhole filepython

numbers = [10, 20, 30]
print(next(numbers))

terminal

$ python main.py

── what you'll see ──

Traceback (most recent call last):

File "main.py", line 2, in <module>

print(next(numbers))

~~~~^^^^^^^^^

TypeError: 'list' object is not an iterator

Break #2

An iter that returns self without next

Write a class whose __iter__ returns self, but forget to write __next__.

main.pywhole filepython

class Broken:
    def __iter__(self):
        return self


for x in Broken():
    print(x)

terminal

$ python main.py

── what you'll see ──

Traceback (most recent call last):

File "main.py", line 6, in <module>

for x in Broken():

~~~~~~^^

TypeError: iter() returned non-iterator of type 'Broken'

Myth vs fact

Myth

A list is an iterator.

Fact

A list is an iterable. It makes iterators when you loop. That is why you can loop over a list many times, and why next(my_list) is an error.

Myth

Looping over an iterator twice gives the same values.

Fact

An iterator is used up after one pass. The second loop is empty and raises no error. Loop over the original iterable again, or build a new iterator.

Myth

StopIteration means something went wrong.

Fact

It is the normal way an iterator says it is finished. Only a StopIteration you did not expect (for example, from a stray next() call) is a bug.

Myth

for and while over a list are two unrelated tools.

Fact

A for loop is a while loop with iter, next and a try/except StopIteration done for you.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
CPython's list iterator (list_iterator) stores the list and an index. If you append to the list while looping, the loop sees the new items and may never end; if you delete items, the index skips some. Do not change a list you are looping over: loop over a copy (for x in items[:]) or build a new list.
▸
iter(callable, sentinel) is a two-argument form: it calls the function repeatedly until it returns the sentinel. iter(lambda: file.read(4096), "") reads a text file in chunks of 4096 characters (use b"" as the sentinel for a file opened in binary mode). It is an old but useful way to turn "call until done" code into a loop.
▸
Iterators are the common currency of the standard library. A function that takes "any iterable" and returns a generator composes with every other one, which is the design behind itertools (Topic 9.4) and lazy pipelines (Topic 9.3). Type hints express this as Iterable[T] for inputs and Iterator[T] for outputs (Phase 10).
▸
iter() has an older fallback: an object with no __iter__ but with a __getitem__ that accepts 0, 1, 2, ... and raises IndexError at the end is still iterable (Python calls __getitem__ with growing integers). This is why very old sequence classes can be looped over. Do not rely on it for new code: define __iter__. Note also that isinstance(x, collections.abc.Iterable) only checks for __iter__, so it misses this legacy case; iter(x) inside try/except TypeError is the reliable test.
▸
An iterator can offer __length_hint__, an estimate that list() uses to pre-size its memory (read it with operator.length_hint). It is only a hint, never relied on for correctness. The dictionary rule is stricter than the list rule: adding or removing keys during a loop raises RuntimeError at once (changing a value for an existing key is allowed), and sets behave the same way, while a list silently misbehaves.

Remember this

1
A for x in thing: loop does exactly two things. First it calls iter(thing), which returns an iterator. Then it calls next(iterator) again and again, giving each result to x. When the iterator has nothing left it raises a special exception (an error object, Topic 7.1) called StopIteration, and the loop ends quietly. That is the whole mechanism; there is no other kind of loop over a collection.
2
An iterable is any object that can give you an iterator: it has an __iter__ method (a special method, Topic 8.7). Lists, tuples, strings, dictionaries, sets, ranges and open files are all iterables. An iterator is an object with a __next__ method that returns the next value, and an __iter__ method that returns itself. Every iterator is an iterable, but a list is an iterable that is not an iterator: next([1, 2]) is an error.
3
An iterator is one-shot. It moves forward only and cannot rewind. Once StopIteration has been raised, the built-in iterators and generators stay used up (the protocol says an iterator must keep raising it), and looping over one a second time gives nothing (no error). A list, by contrast, hands out a brand new iterator each time you loop, so you can loop over a list as often as you like.
4
Because an iterator produces values only when asked, it does not have to hold them all in memory. range(1_000_000) does not store a million numbers; a file iterator reads one line at a time. This lazy behaviour is why you can loop over a file bigger than your memory, and it is the idea Topic 9.2 — Generators and yield builds on.
5
You can write your own iterator by making a class with __iter__ (return self) and __next__ (return the next value, or raise StopIteration when done). To make a reusable iterable instead, write __iter__ so it returns a fresh iterator each time, often by returning iter(self.items). Most of the time a generator (the next topic) is a much shorter way to write the same thing.
6
Many built-ins accept any iterable, not just lists: list(...), sum(...), sorted(...), max(...), set(...), dict(...), zip(...), enumerate(...) and "-".join(...). They all use the same protocol. This is why a function that only loops over its argument works with lists, tuples, files, generators and your own classes without any change.

Explain it without notes

Explain what a for loop does, step by step, in terms of iter and next.

What is the difference between an iterable and an iterator? Give an example of each and say which one can be looped over twice.

How do you write an iterator class, and what makes it different from an iterable class?

Why does looping over the same iterator a second time produce nothing, and how do you avoid the bug?

What goes wrong if you add or remove items from a list while looping over it, and what happens with a dictionary?

Practice

Use iter and next by hand to print the three letters of "abc", one per line, and then show what next(it, "done") returns afterwards.

Write an iterator class Evens that takes a limit and produces the even numbers from 0 up to but not including limit. Print list(Evens(7)).

Write an iterable class Words that takes a sentence and loops over its words. Show that you can loop over the same object twice.

Trade-offs

↔
Iterator classes are explicit but long. You manage the state and StopIteration yourself. Generators (Topic 9.2) give the same behaviour in a few lines, so write a class only when you need extra methods or attributes on the iterator.
↔
Lazy iterators save memory but cannot be indexed or measured. You cannot ask an iterator for len() or it[3]. If you need those, build a list, and pay the memory cost.

Done when you can

Done when you can explain what for does using iter, next and StopIteration.
Done when you can tell an iterable from an iterator and give two examples of each.
Done when you can write an iterator class with __iter__ and __next__.
Done when you can explain why a second loop over an iterator is empty.
Done when you can use next(it, default) to avoid StopIteration.

Command Palette

Words you'll meet

Step by step

01A for loop, written by hand

02Iterable versus iterator

03StopIteration is a signal, not a crash

04Writing your own iterator class

05One-shot: why the second loop is empty

06Reusable iterables: a fresh iterator each time

07What else uses the protocol

Try it yourself

Predict, then run

Make Countdown reusable

Loop over a dictionary and a string

Code & diagrams

Break it on purpose

Call next on a list

An __iter__ that returns self without __next__

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can

An iter that returns self without next