Topic 5.7
Closures
In one line
A closure is a function that remembers the variables of the function it was created in, even after that outer function has returned. Closures make counters, function factories and private state possible, and they have one famous trap: late binding in loops.
Think of it like this
A note in your pocket. You leave the shop with a shopping list; later, at home, the note still tells you what to buy. The shop's counter is gone, but the note remembers. A closure is an inner function that carries a note of the outer function's variables with it.
Words you'll meet
New words in this topic, in plain English. Come back here whenever one feels fuzzy.
- Closure
- An inner function together with the outer variables it remembers.
- Free variable
- A variable used by an inner function but defined in an enclosing function.
- Function factory
- A function that creates and returns customised functions.
- nonlocal
- A declaration that lets an inner function rebind a variable of its enclosing function.
- Late binding
- Looking up a variable's value when the function runs, not when it is defined.
- Cell
- The small box Python uses to share a captured variable between functions.
- State
- Information a piece of code remembers between calls.
Step by step
01An inner function that outlives its parent
make_multiplier finishes as soon as it returns times. Normally its local n would be gone. But times uses n, so Python keeps it alive in a closure.
def make_multiplier(n):
def times(x):
return x * n # n comes from the enclosing function
return times
double = make_multiplier(2)
triple = make_multiplier(3)
print(double(10), triple(10)) # 20 3002State that changes: nonlocal
A counter needs to update its variable on each call, so the inner function declares nonlocal count. Two counters made by two calls keep separate counts.
def make_counter():
count = 0
def step():
nonlocal count
count += 1
return count
return step
first = make_counter()
second = make_counter()
print(first(), first(), first()) # 1 2 3
print(second()) # 103Looking inside a closure
The function's code object lists its free variables in __code__.co_freevars, and __closure__ holds a tuple of cells, one per free variable. cell_contents is the current value. You'll rarely need this, but it proves that the value is stored in the function.
print(first.__code__.co_freevars) # ('count',)
print(first.__closure__[0].cell_contents) # 304Late binding: the loop trap
All three lambdas below read the same variable i when they are called. By then the loop has finished and i is 2.
functions = []
for i in range(3):
functions.append(lambda: i)
print([f() for f in functions]) # [2, 2, 2], not [0, 1, 2]05Two fixes
A default argument is evaluated when the lambda is created, freezing the current i. A factory call gives each function its own variable i. Pick the factory when the code is more than a one-liner.
fixed_default = [lambda i=i: i for i in range(3)]
print([f() for f in fixed_default]) # [0, 1, 2]
def make_returner(i):
return lambda: i
fixed_factory = [make_returner(i) for i in range(3)]
print([f() for f in fixed_factory]) # [0, 1, 2]06Closure or class?
A closure with one or two pieces of state and one operation is compact. If you need several operations on the same state (increment, reset, show), a class says so more clearly, and learners can inspect its attributes directly. Both are correct tools; choose by how many pieces and operations there are.
Try it yourself
- 1
Remove nonlocal
In the first example, delete the
nonlocal countline and run it. Read theUnboundLocalErrorand compare with Topic 5.5. - 2
Make a start value
Change
make_counter()tomake_counter(start=0)and setcount = start. Make a counter that starts at 100 and one that starts at 0, and call them alternately. - 3
Fix the loop another way
In the second example, make
fixed_defaultwith three functions that each returni * 10and print the list. Predict[0, 10, 20]first.
Code & diagrams
Each make_counter() call has its own count. The last two lines look inside the closure.
Expected output
1 2 3
1
free variables: ('count',)
remembered value: 3
20 30The first line shows the trap. Both fixes print 0, 1, 2.
Expected output
late binding : [2, 2, 2]
default arg : [0, 1, 2]
factory : [0, 1, 2]Each limiter remembers its own call counts. The counts live in a dict, and changing a dict's contents is not rebinding, so no nonlocal is needed. The two limiters never share state.
Expected output
api asha True
api asha True
api ravi True
api asha False
login asha True False
remembered: {'asha': 3, 'ravi': 1}
free variables: ('calls', 'max_calls')Break it on purpose
Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.
Break #1
Build functions in a loop without freezing the variable
Make three lambdas in a loop and call them after the loop.
functions = []
for i in range(3):
functions.append(lambda: i)
print([f() for f in functions])Myth vs fact
Myth
A closure copies the variable's value when it is created.
Fact
It keeps the variable itself (a cell), and reads its current value on each call.
Myth
Closures are an advanced trick you won't meet.
Fact
Every decorator, many callbacks and functools helpers rely on them. You use them even when you didn't write one.
Myth
Two functions from the same factory share their state.
Fact
Each call of the outer function creates new variables, so the closures it returns are independent.
Pro corner
Extra depth for experienced readers. New to this? Skip it for now and come back later.
- ▸
CPython stores a captured variable in a cell object that the outer and inner functions both refer to. Only variables that an inner function actually uses become cells; the compiler decides this by reading the source. That is why a closure doesn't keep everything alive, only what it uses.
- ▸
A closure keeps its cells (and anything they point to) alive as long as the function lives. Capturing a big object (a model, a DataFrame) can keep it in memory longer than you expect. Delete the function or the reference when done.
- ▸
Late binding is about variables, so it bites with any captured name that changes later, not only loop variables, and in comprehensions, nested loops and
defstatements inside loops alike. A third fix isfunctools.partial(func, i), which stores the current value at creation. A function with no free variables has__closure__equal toNone. Mutating a captured list or dict needs nononlocal; rebinding the name does (the rate-limiter example). - ▸
functools.partialandlru_cacheare alternatives to hand-written closures for pre-filled arguments and for caching. Prefer them when they fit. - ▸
Closures and threads/async: a closure with mutable state is shared by every task that holds the function, so updating it from several tasks needs care (Phase 12).
Remember this
- 1
When a function is defined inside another function and uses a variable of the outer function, Python keeps that variable alive for the inner function. The inner function plus those kept variables is a closure. The kept variables are called free variables. Each call of the outer function makes a fresh set, so two closures from two calls do not share state.
- 2
The typical shape is a function factory: the outer function takes settings and returns a customised inner function.
make_multiplier(3)returns a function that multiplies by 3.make_counter()returns a function that counts up each time it is called. The outer function is finished long before the inner one runs, yetcountis still there. - 3
To change (rebind) an enclosing variable from the inner function, declare it
nonlocal(Topic 5.5). Without it,count += 1would create a new local and fail withUnboundLocalError. Reading an enclosing variable needs no declaration. - 4
Python closures capture variables, not values. The inner function looks the variable up when it runs, not when it was defined. This is called late binding. It is harmless for factories, but it surprises people in loops:
[lambda: i for i in range(3)]makes three functions that all share the one variablei, which ends as 2, so every function returns 2. - 5
Two common fixes: bind the current value as a default argument (
lambda i=i: i, because defaults are evaluated at definition time, Topic 5.2), or make the function inside another function call, so each call has its own variable (make_returner(i)). The second is the clearer form for longer code. - 6
Closures are the idea behind decorators (Topic 9.6), callbacks (functions you hand to other code so it can call them back later) that remember context, and small stateful helpers such as rate limiters or caches. When state grows to several variables and operations, a class (Phase 8) usually reads better. Closures remember state without needing a class.
Explain it without notes
What is a closure and what problem does it solve?
Why does [lambda: i for i in range(3)] give three functions that all return 2?
When do you need nonlocal?
How do two closures made from separate calls of the same factory relate?
Practice
Write make_greeter(greeting) that returns a function taking name and returning e.g. Hi, Asha!. Create two greeters with different greetings.
Write make_accumulator() that returns a function. Each call add(n) adds n to a running total and returns it. Call it with 10, 5 and 1.
Create a list of three functions with a loop so that calling them returns 10, 20 and 30, avoiding the late-binding trap.
Trade-offs
- ↔
Closure versus class. A closure is short for one small piece of state. A class gives named attributes, several methods and easy inspection, so it wins as state grows.
- ↔
Default-argument freeze versus factory. The default trick is a single line but looks odd and can be overridden by accident when the function is called with an argument. A factory is clearer and safer.
Done when you can
Done when you can write a function factory that returns a customised inner function.
Done when you can write a counter with
nonlocaland explain why two counters are independent.Done when you can explain late binding and fix it two ways.
Done when you can show a closure's
__closure__andco_freevars.Done when you can decide between a closure and a class for a small task.