Running Containers Properly: Restarts, Health, Limits & Logs

In one line

How a container runs matters as much as how it's built: restart policies, HEALTHCHECK, CPU and memory limits, a read-only root filesystem with dropped capabilities, correct signal handling (PID 1), and logging drivers with rotation.

0/4 · 0%

Think of it like this

Hiring a new employee. The job description (image) isn't enough; you also agree working hours (resource limits), what happens if they're off sick (restart policy), a daily check-in (healthcheck), which rooms they may enter (capabilities, read-only filesystem), and where their reports go (logging).

Key ideas

01
RESTART POLICIES: --restart unless-stopped (or always) restarts crashed containers and after a daemon or host reboot; on-failure:5 retries a limited number of times. On orchestrators (Compose in production, Kubernetes) the platform manages this instead.
02
HEALTHCHECK: HEALTHCHECK --interval=10s --timeout=3s --retries=3 CMD curl -fsS http://localhost:8080/actuator/health || exit 1 lets Docker mark the container healthy/unhealthy; Compose's depends_on: condition: service_healthy uses it (Phase 5). Kubernetes ignores Dockerfile HEALTHCHECK and uses its own probes (Kubernetes course, probes).
03
LIMITS: --memory 512m --cpus 1.5 --pids-limit 200 (cgroups, Phase 9). Without a memory limit, one leaking container can take the whole host down; with one, the container gets OOM-killed (exit 137) instead. Make sure the runtime inside respects it (JVM MaxRAMPercentage, Phase 6).
04
HARDENING at runtime: --read-only root filesystem plus --tmpfs /tmp for scratch space, --cap-drop ALL (add back only what's needed), --security-opt no-new-privileges, non-root USER (Phase 7). PID 1 and SIGNALS: the entrypoint process is PID 1 and must forward SIGTERM to the app; use exec-form ENTRYPOINT ["java", ...] or --init (tini) so docker stop shuts down gracefully instead of being killed after 10 seconds.
05
LOGGING: containers log to stdout/stderr; the default json-file driver grows forever unless you set --log-opt max-size=10m --log-opt max-file=3 (or in /etc/docker/daemon.json). Other drivers (journald, awslogs, fluentd) ship logs elsewhere; on Kubernetes, node agents collect stdout (Stateful Systems course, log pipeline).

Code & diagrams

a production-grade docker runbash

docker run -d --name api \
  --restart unless-stopped \
  --memory 768m --cpus 1.5 --pids-limit 300 \
  --read-only --tmpfs /tmp:rw,size=64m \
  --cap-drop ALL --security-opt no-new-privileges \
  --log-opt max-size=10m --log-opt max-file=3 \
  --health-cmd 'curl -fsS localhost:8080/actuator/health || exit 1' \
  --health-interval 10s --health-retries 3 \
  -p 8080:8080 --env-file /etc/shoplite/api.env \
  ghcr.io/shoplite/api:sha-3f2a1c9
docker inspect --format '{{.State.Health.Status}}' api   # healthy

/etc/docker/daemon.json (host-wide log rotation)json

{
  "log-driver": "json-file",
  "log-opts": { "max-size": "10m", "max-file": "3" },
  "live-restore": true
}

Explain it without notes

Why does docker stop sometimes take exactly 10 seconds and then the app dies uncleanly?

Practice

A host's disk fills up every few weeks and the culprit is under /var/lib/docker. What are the likely causes and fixes?

Trade-offs

↔
Read-only filesystems and dropped capabilities cut the blast radius of a compromise but require knowing exactly where the app writes; tight memory limits protect neighbours but cause OOM kills if set below real peak usage.

Done when you can

My containers have restart policies, healthchecks, limits, and rotated logs
My entrypoint handles SIGTERM correctly and runs as non-root with a read-only root filesystem