PHASE 6Intermediate ~29 min· topic 3 of 9

Topic 6.3

Packages and init.py

In one line

A package is a folder of modules that you import as one unit. Its __init__.py runs on import and decides what the package shows to the outside, absolute imports name the full path from the top, relative imports use dots, and circular imports are fixed by changing who imports whom.

Think of it like this

A filing cabinet. The cabinet is the project. Each drawer is a package (say, shop). Each paper folder inside a drawer is a module (a single Python file, such as prices.py or tax.py). On the front of the drawer there is a label sheet that says what is inside and which folders most people ask for. That label sheet is the __init__.py.

Words you'll meet

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Package: A folder of modules that Python can import as one unit, usually with an __init__.py.
Submodule: A module inside a package, such as shop.tax.
__init__.py: The file that runs when a package is imported. It defines what the package exposes.
Absolute import: An import that gives the full path from the top package: from shop.tax import add_tax.
Relative import: An import using dots to mean "here" (.) or "one level up" (..): from .tax import add_tax.
__all__: A list of names in a module that from module import * will import. It marks the public interface.
Circular import: Two modules that import each other, so one of them is used before it has finished loading.
Namespace package: A package folder with no __init__.py, which can be assembled from several folders.
Public API: The names a package promises to keep stable for its users. Everything else is internal.

Step by step

01Turn a folder into a package

Here is a small project. The folder shop has an __init__.py, so import shop works. Two modules sit beside it.

Notice that __init__.py is also just a module: it is where the package's own top-level names live.

shop/tax.pywhole filepython

RATE = 0.1


def add_tax(amount):
    return round(amount * (1 + RATE), 2)

Turn a folder into a packagediagram

Rendering diagram…

02Sibling imports: absolute and relative

prices.py needs add_tax from its sibling tax.py. Both lines below do the same job. The first names the full path; the second says "the tax module next to me".

Use absolute imports by default because anyone can read them without counting dots. Use relative imports when a package is deep or its name may change.

shop/prices.pywhole filepython

from shop.tax import add_tax      # absolute
# from .tax import add_tax        # relative: the same thing


def total(items):
    return add_tax(sum(items))

03init.py as the public face

Without help, a user must know your internal layout: from shop.prices import total. With one line in __init__.py they write from shop import total. You can then move total to another file later without breaking users, which is why large libraries do this.

__all__ documents the same list for star imports.

shop/__init__.pywhole filepython

"""The shop package."""
from .prices import total

__all__ = ["total", "VERSION"]
VERSION = "1.0"

04What importing a package loads

import shop runs shop/__init__.py, which in turn imports shop.prices, which imports shop.tax. So after that one line, all three modules are in sys.modules. import shop.tax afterwards finds shop.tax already loaded and just binds the name shop.

A package has a __path__ attribute (the list of folders that make it up). Plain modules do not. That attribute is how you tell the two apart.

main.pywhole filepython

import sys
import shop

print(shop.total([10, 20, 30]))        # 66.0
print("shop.tax" in sys.modules)       # True
print(hasattr(shop, "__path__"))       # True: it is a package

05How a circular import breaks

a.py starts loading and executes from pkg.b import b_value. That begins loading b.py, which executes from pkg.a import a_value. But a is only half-loaded: Python is still on its first line, and a_value has not been created yet. So Python reports that it cannot import the name from a *partially initialized module*.

The message names both the name and the module, and says "partially initialized", which is the signature of a cycle.

How a circular import breaksdiagram

Rendering diagram…

06Fixing a cycle

There are three cures, in order of preference. (1) Move the shared thing into a third module that both import. A cycle usually means two modules are tangled, and the shared piece wants its own home. (2) Import the module, not the name, and use pkg.b.b_value later inside functions, when both modules are finished. import pkg.b on a half-loaded module is allowed; only from ... import name needs the name right now. (3) Import inside the function that needs it, as a last resort.

A cycle is a design smell: if A needs B and B needs A, ask what they really share.

pkg/a.pywhole filepython

import pkg.b                 # the module, not a name from it

a_value = "A"


def show():
    return a_value + pkg.b.b_value     # looked up at call time, both loaded

07Namespace packages and the missing init.py

If two different folders on sys.path both contain a folder called plugins with no __init__.py, Python merges them into one namespace package whose __path__ lists both. plugins.alpha can come from one folder and plugins.beta from the other.

It is a deliberate feature for distributing one package in several pieces. It is also what happens by accident when you forget an __init__.py, and then nothing in the folder behaves like a package should: no initialisation code runs, and some tools skip the folder.

Try it yourself

1
Switch to a relative import
In the first example, change from .tax import add_tax to from shop.tax import add_tax in prices.py. The output stays the same. Now try to write from .tax import add_tax inside the main program itself (outside any package) and see the error from Topic 6.2.
2
Remove __all__
Delete the __all__ = [...] line in the generated __init__.py and run again. The star import now brings in every public name: VERSION, total and also the submodule names prices and tax that were imported as a side effect. __all__ gives you control.
3
Make the cycle again
In the second example, keep the *fixed* files but add a line x = pkg.b.b_value at the top level of pkg/a.py. The cycle bites again, because a top-level use needs the other module to be finished. Cycles are about *when* a name is needed.

Code & diagrams

Build a package, import it, use __all__ New tab

The example creates the whole `shop` package on disk (`Path(...).mkdir()` makes a folder, `write_text` fills a file; Topic 7.6 teaches `pathlib`), then imports it. Notice that `__init__.py` prints when it runs, and that the star import brings in exactly the names in `__all__`. `hasattr(shop, "__path__")` asks "does this object have a name `__path__`?" (only packages do). The `exec` line runs one line of text as Python code and `namespace` collects the names it creates. You will rarely need `exec`: here it only lets us inspect what a star import brings in.

Expected output

shop/__init__.py ran
package name: shop | is a package: True
total: 66.0
shop.tax.RATE = 0.1
star import gave: ['VERSION', 'total']

A circular import, and the fix New tab

First version: each module takes a name from the other, which fails. Second version: each imports the other *module* and uses it later inside a function. The `split` just keeps the printed message short and the same on every computer. `try`/`except ImportError` means: try the import, and if it fails with that error, print it instead of crashing (Topic 7.1).

Expected output

ImportError: cannot import name 'a_value' from partially initialized module 'pkg.a'
fixed: AB BA

A namespace package built from two folders New tab

There is no `__init__.py` anywhere, yet `import plugins` works, and it combines the two `plugins` folders. Its `__file__` is `None` because no single file defines it. `sys.path[:0] = [...]` is slice assignment (Topic 4.2): it puts the two folders at the front of the search list.

Expected output

has __init__.py: False
namespace parts: ['plugins/one', 'plugins/two']
alpha beta

A package layout and one importbash

`from shop import total` works because `__init__.py` imports `total` from `prices.py`.

project/
  main.py
  shop/
    __init__.py
    prices.py
    tax.py

Break it on purpose

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

Break #1

Two modules import names from each other

Make a.py do from pkg.b import b_value and b.py do from pkg.a import a_value, then import pkg.a.

pkg/a.pywhole filepython

from pkg.b import b_value
a_value = "A"

terminal

$ python -c "import pkg.a"

── what you'll see ──

Traceback (most recent call last):

...

ImportError: cannot import name 'a_value' from partially initialized module 'pkg.a' (most likely due to a circular import) (/home/you/project/pkg/a.py)

Myth vs fact

Myth

A folder is a package only if it has __init__.py.

Fact

Since Python 3.3 a folder without it can be imported as a namespace package. Regular packages with __init__.py are still the right default for your own code.

Myth

__init__.py must contain code.

Fact

An empty file is perfectly valid. It only has to exist to mark the folder as a regular package.

Myth

Relative imports are the modern, correct way and absolute imports are old-fashioned.

Fact

Both are fine. PEP 8 prefers absolute imports for readability; relative imports are a convenience inside a package.

Myth

Circular imports are Python bugs.

Fact

They are design problems. Python loads modules in order and cannot use a name before it exists.

Pro corner

Extra depth for experienced readers. New to this? Skip it for now and come back later.

▸
Importing a.b.c imports a, then a.b, then a.b.c, running each __init__.py on the way down. A heavy __init__.py in a parent package is paid for by every import of every child. Large libraries keep __init__.py small, or use a module-level __getattr__ (PEP 562) to load names lazily on first access.
▸
Namespace packages have a precedence rule worth knowing. While scanning sys.path in order, Python remembers folders without an __init__.py as candidate pieces, but if a regular package or module of that name turns up later on the path, it wins and the pieces are ignored. A namespace package's __path__ is recomputed if sys.path changes. Also, from pkg import name first looks for an attribute name on the package and, failing that, tries to import a submodule pkg.name, which is why from zoo import cat works even when __init__.py is empty.
▸
A package's public API is a promise. Names without a leading underscore, or listed in __all__, are the surface users depend on. Everything under _internal or _private modules can change freely. Documenting this in __init__.py is what lets libraries evolve without breaking users.
▸
Tools such as pytest change how rootdir and sys.path are set up, so a test importing shop may work in one run and fail in another if the package is not installed. Installing your project in editable mode (Topic 6.8) removes the whole class of problem.
▸
import-linter and similar tools can enforce that module A may never import B, which keeps layers of a big codebase from sliding into cycles. Many teams add them to CI (Topic 11.8).

Remember this

1
A package is a folder that Python can import as a module. Traditionally it contains a file called __init__.py (it may be empty). When you write import shop, Python runs shop/__init__.py and the resulting module object is the package. Submodules are reached with dots: shop.prices means the file shop/prices.py, and import shop.prices loads it. Packages can nest to any depth: shop.reports.monthly.
2
What goes into __init__.py? Often nothing. Often a public face for the package: it imports the most useful names from the submodules so that users can write from shop import total instead of from shop.prices import total. It can also hold a __version__ string. Keep it light: it runs every time anyone imports any part of the package, so heavy work there slows every import.
3
Recall from Topic 6.2: Python finds a package the same way it finds a module, by searching the folders in sys.path. Inside a package there are two ways to import a sibling (a module in the same package). An absolute import spells the whole path from the top: from shop.tax import add_tax. It is clear, works from anywhere, and is the style PEP 8 prefers. A relative import counts from the current file's own position: from .tax import add_tax means "the module tax in the same package as me", and .. means "one package up". Relative imports survive renaming the top package; they only work inside a package, never in a script run by path.
4
__all__ is a list of strings in a module (or __init__.py) that names what from package import * should bring in. It is also a statement of intent: "these are the public names". Names starting with an underscore are private by convention. Without __all__, a star import takes every name that does not start with an underscore.
5
A circular import happens when module A imports B while B imports A. Python starts loading A and hits import B. It starts loading B, and B says from A import something. But A is only half-loaded (Python is still on A's first line), so A does not have that name yet. The result is ImportError: cannot import name ... from partially initialized module. Fix it by changing the design: move the shared code to a third module that both import, import the *module* instead of a name and use it later inside functions, or (last resort) import inside the function that needs it.
6
Since Python 3.3 a folder without __init__.py can also be imported, as a namespace package. Namespace packages let one logical package be spread over several folders on sys.path (plugin systems and large company codebases use this). For your own projects, add __init__.py: tools, test runners and readers all expect it, and a forgotten file in a folder that *should* be a regular package causes confusing behaviour.

Explain it without notes

What is a package, and what does __init__.py do?

What is the difference between absolute and relative imports? When does each fail?

What is __all__ for?

What causes a circular import error, and how do you fix it?

Practice

Create a package zoo: a folder zoo with an empty __init__.py, and a module zoo/cat.py that contains SOUND = "meow". Then write the code that imports cat from zoo and prints cat.SOUND.

Change the __init__.py of your zoo package so that SOUND is available directly. After your change, from zoo import SOUND must work.

Write the relative-import line that zoo/dog.py would use to import SOUND from its sibling cat.py.

Trade-offs

↔
**A rich __init__.py vs an empty one.** Re-exporting names makes the API short and lets you reorganise inside, but every import pays for loading everything it pulls in, and the package becomes more tightly knit. Libraries re-export a few key names; applications often leave it empty.
↔
Relative vs absolute imports. Relative imports survive renaming the package and make it simple to move a subpackage; absolute imports are clearer to read and work in scripts and tests. Choose one style per project and enforce it with a linter.
↔
Splitting into many small modules vs few large ones. Small modules keep each file focused and reduce merge conflicts, but too many create import tangles and cycles. A cycle usually means the split cut through something that belongs together.

Done when you can

Done when you can create a package folder, import it, and explain what __init__.py does.
Done when you can write the same sibling import in absolute and relative form.
Done when you can explain what __all__ controls and what a star import takes without it.
Done when you can recognise a circular import error from its wording and apply a fix.
Done when you can say what a namespace package is and why a missing __init__.py can surprise you.

Command Palette

Words you'll meet

Step by step

01Turn a folder into a package

02Sibling imports: absolute and relative

03__init__.py as the public face

04What importing a package loads

05How a circular import breaks

06Fixing a cycle

07Namespace packages and the missing __init__.py

Try it yourself

Switch to a relative import

Remove __all__

Make the cycle again

Code & diagrams

Break it on purpose

Two modules import names from each other

Myth vs fact

Pro corner

Remember this

Explain it without notes

Practice

Trade-offs

Done when you can

03init.py as the public face

07Namespace packages and the missing init.py

Remove all