Overlay Filesystems, Copy-on-Write & Rootless Containers

In one line

Image layers are stacked read-only directories merged by overlayfs, with a thin writable layer per container; rootless mode and user namespaces run the whole stack without real root.

0/4 · 0%

Key ideas

01
OVERLAYFS merges several read-only LOWER directories (image layers) with one writable UPPER directory into a single view. Reading a file finds the topmost copy; modifying a file from a lower layer first COPIES it up to the upper layer (copy-on-write); deleting creates a 'whiteout' entry. That's exactly why RUN rm secret doesn't remove a secret from earlier layers (DevSecOps course, Lab 0.2).
02
Layers are shared: ten containers from the same image share the image's lower layers on disk and in page cache; each only adds its own small writable layer. Data written there disappears with the container, which is why volumes exist (Docker course, Topic 3.1).
03
Copy-on-write makes writes to large files inside the container expensive (the whole file is copied up on first write). Databases and heavy-write workloads should always write to volumes, never to the container layer.
04
ROOTLESS containers run the daemon and containers as an ordinary user, using user namespaces (root inside maps to your UID outside) and user-mode networking. A container escape then yields an unprivileged user, not host root. Podman is rootless by default; Docker supports rootless mode; Kubernetes is adding user-namespace support for pods.

Code & diagrams

overlay.shbash

docker run -d --name ov nginx:alpine
docker inspect ov --format '{{json .GraphDriver.Data}}' | jq       # LowerDir, UpperDir, MergedDir
docker exec ov sh -c 'echo hi > /usr/share/nginx/html/new.html'
sudo ls $(docker inspect ov --format '{{.GraphDriver.Data.UpperDir}}')/usr/share/nginx/html   # the copy lives in the upper layer
docker diff ov                                                     # A = added, C = changed, D = deleted
docker rm -f ov

Explain it without notes

Why should a Postgres container store its data on a volume rather than in the container's filesystem?

Practice

Run docker diff on a container after using it for a while. What does each letter mean, and what might surprise you?

Trade-offs

↔
Layered, copy-on-write storage makes images small to distribute and fast to start, but penalises heavy writes; rootless modes improve security but can limit networking features and some privileged operations.

Done when you can

I can explain lower, upper, and merged directories and copy-on-write.
I know why deleted files remain in earlier image layers.
I understand what rootless containers change.