Command Palette

Search for a command to run...

Hectal
Lab 3.4·Pipeline & RuntimeHIGH

Someone opened a shell in production

Every preventive layer can fail. When someone does get inside a running container, how long until you know?

Weakness
MITRE ATT&CK T1059 · Command and scripting interpreter (containers)
Target
ShopLite containers on a Linux host
runtime threat detectionFalco rules and alertsrouting security alertsimmutable containers make anomalies obvious

Run these techniques only against the lab you own. Using them on systems you don't have permission to test is illegal.

Lab setup

  1. 01Install Falco (Linux host with a recent kernel)

    Falco watches system calls using eBPF and matches them against rules describing suspicious behaviour. On Docker Desktop (macOS/Windows), run this lab in a Linux VM or a kind cluster instead.

    terminal
    $ docker run -d --name falco --privileged \
    -v /var/run/docker.sock:/host/var/run/docker.sock -v /proc:/host/proc:ro -v /etc:/host/etc:ro \
    falcosecurity/falco:0.41.3 falco -o engine.kind=modern_ebpf
    ── output ──
    3f2a91c0e4…
    Falco itself needs elevated privileges to observe the kernel. Run it as a dedicated, tightly controlled component, never alongside application code.

The threat

  1. 01What post-compromise activity looks like

    After gaining code execution in a container, attackers typically spawn a shell, look around (read environment variables and mounted secrets, list processes), try to download tools, and reach out to other services. In an immutable, single-process container, every one of these is abnormal: the application never spawns a shell, never writes to /usr/bin, and never reads /etc/shadow.

  2. 02Simulate the signal safely

    You don't need an exploit to test detection. An operator opening an interactive shell in the container produces the same system calls Falco is watching for. (Use the pre-distroless shoplite image, which still has a shell.)

    terminal
    $ docker exec -it shoplite sh -c 'id; exit'
    ── output ──
    uid=1000(node) gid=1000(node) groups=1000(node)

What's at risk

  • Without runtime detection, an intruder inside a container is invisible until they cause visible damage.

Detect

  1. 01Falco's default rules catch it immediately

    terminal
    $ docker logs falco 2>&1 | grep -i 'shell was spawned'
    ── output ──
    Notice A shell was spawned in a container with an attached terminal (evt_type=execve user=node user_uid=1000 user_loginuid=-1 process=sh proc_exepath=/bin/busybox parent=runc command=sh -c id; exit terminal=34816 exe_flags=EXE_WRITABLE container_id=5e1c0a7b9d2f container_name=shoplite)

  2. 02Route alerts where people will see them

    Falcosidekick forwards Falco events to Slack, PagerDuty, Alertmanager, Loki, and dozens of other outputs. Sending them to Loki puts security events next to the application logs from the Observability course; Critical-priority rules also page.

    falco.yaml (excerpt)add to fileyaml
    json_output: true
http_output:
  enabled: true
  url: http://falcosidekick:2801/
# falcosidekick env: LOKI_HOSTPORT=http://loki:3100
#                    ALERTMANAGER_HOSTPORT=http://alertmanager:9093
#                    ALERTMANAGER_MINIMUMPRIORITY=critical

Defend

  1. 01Custom rules for YOUR app's normal behaviour

    The strongest rules describe what your service should never do. ShopLite only ever runs node; any other process in its container is suspicious. Tune rules until they're quiet in normal operation, because a noisy detector gets ignored just like a noisy pager.

    Hardened

    falco_rules.local.yamlwhole fileyaml
    - rule: Unexpected process in shoplite
  desc: ShopLite containers must only ever run node
  condition: >
    spawned_process and container.name startswith "shoplite"
    and not proc.name = "node"
  output: >
    Unexpected process in shoplite (proc=%proc.cmdline parent=%proc.pname
    user=%user.name container=%container.name image=%container.image.repository)
  priority: CRITICAL
  tags: [shoplite, mitre_execution]

  2. 02Make the unusual impossible, too

    Detection and prevention reinforce each other. The distroless image from Lab 2.3 has no shell to spawn, and the read-only filesystem from Lab 2.1 prevents downloading tools. What remains possible is short and specific enough that any Falco alert is worth waking up for.

    Vulnerable

    Dockerfileadd to filedocker
    FROM node:22-alpine   # has /bin/sh, wget, apk ...

    Hardened

    Dockerfileadd to filedocker
    FROM gcr.io/distroless/nodejs22-debian12:nonroot   # node only

Verify

  1. 01The custom rule fires on anything but node

    Repeat the safe simulation against the alpine-based image and confirm a CRITICAL event reaches Loki and Alertmanager. Against the distroless image there's no shell to run at all, and docker exec fails, which is the goal.

    terminal
    $ docker exec shoplite-distroless sh
    ── output ──
    OCI runtime exec failed: exec failed: unable to start container process: exec: "sh": executable file not found in $PATH: unknown

The concepts

Defence in depth, ending with detection

Every chapter of this course added a layer: no secrets to steal, trusted dependencies and images, minimal, unprivileged containers, secure pipelines and code. Runtime detection is the last layer. It assumes the others can fail and minimises how long an intruder can operate unnoticed, which is often what separates a minor incident from a breach.

Your turn

01

Which Falco default rule would fire if something inside a container read /etc/shadow, and what's its priority?

Interview questions

01

What does runtime security add on top of image scanning and hardening?

0/4 · 0%