Command Palette

Search for a command to run...

Hectal
Phase 13Advanced14 of 18 in Python

Python Internals and Performance

What an object really is, how memory is freed, what bytecode and attribute lookup do, how descriptors and metaclasses work, and how to measure and speed up code.

Until now you have used Python. This phase opens the machine. You will see what a name and an object really are, how Python frees memory (reference counting and the garbage collector), what bytecode looks like, and the exact steps of an attribute lookup. Those steps explain features that look like magic: methods, property, cached_property, @classmethod, dataclasses and ORMs (libraries that turn database rows into Python objects).

Then come the two tools that frameworks use to bend the language: descriptors and metaclasses. You will rebuild property and cached_property yourself, write a descriptor that validates data, and use __init_subclass__ for a plugin registry, the same trick behind many model and plugin systems in AI libraries.

The last three topics are about speed. You will measure before you optimise (timeit, cProfile, tracemalloc), learn the habits that make Python fast (the right data structure, caching, generators, vectorising), and see why NumPy is fast: contiguous C memory, the buffer protocol and loops that run outside the interpreter. Phase 14 (NumPy and pandas) builds directly on that last idea.

0/9 · 0%
9 topics ~4 h 28 code blocks & diagrams
Start with the first topic
1
13.1

Objects, Identity and References

Every value in Python is an object with an identity, a type and a value, and a variable is just a name pointing at an object. Knowing the difference between == (same value) and is (same object) explains aliasing, argument passing and the mutable default argument trap.

36 min 1 diagram 3 code practice

2
13.2

Memory: Reference Counting and Garbage Collection

CPython frees most objects the instant their reference count hits zero, and a separate cycle collector (the gc module) cleans up objects that keep each other alive. weakref lets you refer to an object without keeping it alive, and del removes a name, not the object.

30 min 4 code practice

3
13.3

Bytecode and the dis Module

CPython compiles your source into bytecode, a list of small instructions for a stack-based virtual machine, and stores it in a code object. The dis module prints that bytecode, which shows what a line really costs and how the specialising interpreter speeds up hot code.

28 min 3 code practice

4
13.4

How Attribute Lookup Works

Reading obj.name follows a fixed search: data descriptors on the type, then the instance's __dict__, then the class and its parents in MRO order, and only then __getattr__. Methods are plain functions that become bound methods when you read them.

28 min 3 code practice

5
13.5

Descriptors

A descriptor is an object stored on a class that controls what happens when an attribute is read, set or deleted, through __get__, __set__ and __set_name__. property, methods, classmethod, staticmethod and cached_property are all descriptors.

29 min 3 code practice

6
13.6

Metaclasses and Class Creation

A class is an object made by another object, type. Knowing how class statements call type(name, bases, namespace) explains metaclasses, but most needs are met more simply by __init_subclass__ and class decorators.

26 min 1 diagram 2 code practice

7
13.7

Measuring Speed: timeit and cProfile

Never guess where a program is slow. Time small snippets with timeit, find the slow function with cProfile, find the memory hog with tracemalloc, and always compare before and after.

30 min 2 code practice

8
13.8

Making Python Fast

Python code gets faster mainly by using a better algorithm or data structure, avoiding repeated work with caches, streaming with generators, and moving loops into C through built-ins and NumPy. Only after that do local-variable tricks and native extensions matter.

29 min 2 code practice

9
13.9

Why NumPy Is Fast: C Underneath

A Python list holds pointers to separate objects, while NumPy, array and memoryview use one contiguous block of raw C data that is shared through the buffer protocol. That layout, compiled loops and releasing the GIL in native code are why NumPy is fast.

30 min 4 code practice

Practise this phase

Hands-on exercises: write the code, run it, and press Check to have it tested automatically.

1

Split a buffer without copying

Write a function split_chunks(data, size) that cuts a bytes or bytearray into pieces of size bytes (the last piece may be shorter) and returns a **list of memoryview objects**, not copies.

Intermediate
2

Write your own cached_property

Write a class lazy that works as a decorator for a method, like functools.cached_property. The first read of the attribute calls the method, and the result is stored on the instance. Later reads must not call the method again.

Advanced
3

A descriptor that validates

Write a data descriptor Ranged(low, high) that only accepts numbers from low to high (both allowed). Assigning a number outside the range must raise ValueError.

Advanced
4

A plugin registry with __init_subclass__

Make the base class Reader register every subclass automatically, with no metaclass. Use __init_subclass__.

Advanced
5

Break a reference cycle with weakref

Parent keeps a list of Child objects, and each Child keeps a reference back to its parent. That is a reference cycle: deleting the parent does not free it while the children point at it.

Advanced
6

Replace a slow scan with the right data structure

common_items(a, b) returns the items of a that also appear in b, in the order of a (an item that repeats in a stays repeated). The version below works but is slow: it scans the list b for every item.

Advanced