Command Palette

Search for a command to run...

Hectal
PHASE 10Advanced ~8 min· topic 3 of 4

Topic 10.3

Debugging Containers: logs, exec, inspect, events & debug containers

In one line

When a container crashes, restarts, or hangs, a fixed sequence finds the answer: status and exit code, logs, inspect, events, then get inside the namespaces, even for distroless images with no shell.

0/4 · 0%

Think of it like this

A mechanic with a car that won't start. First the dashboard lights (status, exit code), then the error log (logs), then the service record (inspect), then recent events (who touched it), and finally opening the bonnet with their own tools (a debug container).

Key ideas

  1. 01

    STATUS and EXIT CODES: docker ps -a shows Exited (137) = SIGKILL, usually OOM (docker inspect -f '{{.State.OOMKilled}}'); (143) = SIGTERM (stopped normally); (1) = the app crashed, check logs; (126/127) = the command isn't executable or not found (wrong ENTRYPOINT, missing binary, wrong architecture: 'exec format error').

  2. 02

    LOGS and INSPECT: docker logs --tail 100 -f --timestamps api (logs survive the container stopping, until it's removed), docker inspect api (env, mounts, network, restart count, health log). docker events --since 30m shows the timeline: die, oom, restart, health_status changes.

  3. 03

    GETTING INSIDE: docker exec -it api sh if the image has a shell. For DISTROLESS or scratch images with no shell, attach a debug container that shares the target's namespaces: docker debug api (Docker Desktop/Pro), or docker run -it --rm --pid container:api --network container:api nicolaka/netshoot to get full networking and process tools in the same namespaces. On a host, nsenter -t <pid> -n -p enters namespaces directly (Phase 9).

  4. 04

    NETWORK PROBLEMS: from inside the container's network namespace, test DNS (getent hosts db), connectivity (nc -vz db 5432), and what's listening (ss -ltnp). Typical culprits: the app listening on 127.0.0.1 instead of 0.0.0.0 (so the published port doesn't work), a wrong network (containers on different networks can't resolve each other's names), or using localhost to reach another container (Phase 4).

  5. 05

    RESOURCE PROBLEMS: docker stats (live CPU, memory against limit, I/O), docker top api (processes inside), cgroup files for throttling counts (Phase 9). Crash-looping container? docker run --entrypoint sh with the same image and env to explore interactively.

Code & diagrams

the debugging sequencebash
docker ps -a --filter name=api --format '{{.Names}} {{.Status}}'   # Exited (137) 2 minutes ago
docker inspect -f '{{.State.OOMKilled}} restarts={{.RestartCount}}' api
docker logs --tail 50 --timestamps api
docker events --since 30m --filter container=api
docker stats --no-stream api
# distroless image, no shell: borrow tools in the same namespaces
docker run -it --rm --pid container:api --network container:api nicolaka/netshoot \
  sh -c 'ss -ltnp; getent hosts db; nc -vz db 5432'
exit code triagediagram
Rendering diagram…

Explain it without notes

01

The app runs fine with docker run locally but curl localhost:8080 from the host gets 'connection reset'. What's the most likely cause?

Practice

01

A container restarts every 90 seconds with exit code 137, but memory looks fine in docker stats. What else could send SIGKILL?

Done when you can

  • I can triage a container by exit code, logs, inspect, and events

  • I can debug a distroless container by sharing its namespaces with a tools container