“ShopLite restarts every six minutes. The heap graph is perfectly flat.”

1. 08:40

   ALERT

   ShopLiteRestartingFrequently: 4 restarts in 15 minutes

   terminalcopy cmd

   $ docker inspect shoplite --format 'OOMKilled={{.State.OOMKilled}} restarts={{.RestartCount}}'

   ── expected output ──

   OOMKilled=true restarts=4

2. 08:44

   DEAD END

   The heap is flat, so 'it can't be a memory leak'

   19 MB and steady. The team starts looking at 'maybe the kernel is killing it for some other reason'.

   PromQL· Prometheuscopy

   nodejs_heap_size_used_bytes{job="shoplite"}

   result

   {instance="shoplite:8080", job="shoplite"}  1.93e+07

3. 08:52

   QUERY

   Resident memory tells a different story: a sawtooth

   RSS (resident set size) is everything the process holds in RAM. It climbs linearly to the 512 MB limit, the kernel kills the process, it restarts at ~70 MB, and the cycle repeats.

   PromQL· Prometheuscopy

   process_resident_memory_bytes{job="shoplite"} / 1024 / 1024

   result

   {instance="shoplite:8080", job="shoplite"}  431.6

4. 08:55

   QUERY

   Heap flat, RSS growing: where's the memory? external

   Node Buffers are allocated OUTSIDE the V8 heap. nodejs_external_memory_bytes tracks them, and it's growing at the same slope as RSS. Something is keeping Buffers alive.

   PromQL· Prometheuscopy

   deriv(nodejs_external_memory_bytes{job="shoplite"}[5m]) / 1024 / 1024

   result

   {instance="shoplite:8080", job="shoplite"}  1.38

5. 09:20

   FINDING

   A heap snapshot shows a module-level array of Buffers growing forever

   In the lab, it's the leak array. In real code, it's usually a cache without eviction, a growing Map keyed by request data, or event listeners never removed.
6. 09:45

   RESOLVED

   Unbounded cache replaced with an LRU; RSS flat at ~90 MB

1. 01Bound what you retain

   Any in-process cache needs a size or TTL bound, such as an LRU. Switch the lab scenario off; RSS stops growing immediately, but the leaked memory is only freed when the process restarts.

   terminalcopy cmd

   $ curl -s -X POST localhost:8080/admin/chaos -H 'content-type: application/json' -d '{"scenario":"memory-leak","enabled":false}'

   docker compose restart shoplite

   ── expected output ──

   ✔ Container shoplite Started

1. 01Predictive alert: out of memory within an hour

   predict_linear over 30 minutes is robust to short spikes. for: 10m avoids firing on a single burst of allocations. The limit is hard-coded for the lab; in Kubernetes, use container_spec_memory_limit_bytes.

         - alert: ShopLiteMemoryExhaustionPredicted
           expr: predict_linear(process_resident_memory_bytes{job="shoplite"}[30m], 3600) > 512 * 1024 * 1024
           for: 10m
           labels: { severity: warning, service: shoplite }
           annotations:
             summary: "shoplite RSS is on course to exceed its 512 MB limit within an hour"

Run each one against your lab before revealing the answer. Share your own version in the comments; there's usually more than one correct query.