“Prometheus is OOM-killed every 20 minutes and every dashboard is blank. Nothing in ShopLite changed... except one new metric.”

1. 09:12

   ALERT

   Grafana: 'No data' everywhere; Prometheus container restarting

   terminalcopy cmd

   $ docker compose ps prometheus --format '{{.Status}}'

   docker inspect $(docker compose ps -q prometheus) --format '{{.State.OOMKilled}} restarts={{.RestartCount}}'

   ── expected output ──

   Up 2 minutes

   true restarts=4

2. 09:18

   QUERY

   How many series is Prometheus holding in memory?

   Every unique combination of metric name + label values is a SERIES, and every active series lives in memory (the 'head block'). Prometheus exposes its own count.

   PromQL· Prometheuscopy

   prometheus_tsdb_head_series

   result

   {instance="localhost:9090", job="prometheus"}  138412

3. 09:22

   QUERY

   Which metric owns them?

   count by (__name__) counts series per metric name. This query touches every series, so it's expensive on a big Prometheus; run it on a small time window, or use the TSDB status page (/tsdb-status), which gives the same answer from precomputed stats.

   PromQL· Prometheuscopy

   topk(5, count by (__name__) ({__name__=~".+"}))

   result

   {__name__="shoplite_cart_views_total"}                    136304
   {__name__="http_request_duration_seconds_bucket"}             396
   {__name__="nodejs_gc_duration_seconds_bucket"}                 84
   {__name__="prometheus_http_request_duration_seconds_bucket"}   72
   {__name__="process_cpu_seconds_total"}                          3

4. 09:25

   QUERY

   Which label makes it explode?

   Count distinct values per label for that metric. user_id has one value per user who ever viewed a cart, which is unbounded.

   PromQL· Prometheuscopy

   count(count by (user_id) (shoplite_cart_views_total))

   result

   {}  136304

5. 09:31

   ACTION

   Stop the bleeding at the scrape: drop the metric before it's stored

   The app can't be redeployed instantly, but Prometheus can refuse the metric right now. metric_relabel_configs run after the scrape and before ingestion.

     - job_name: shoplite
       static_configs: [{ targets: ["shoplite:8080"] }]
       metric_relabel_configs:
         - source_labels: [__name__]
           regex: shoplite_cart_views_total
           action: drop

6. 11:02

   RESOLVED

   Series count flat at ~1,900; memory stable; old series age out of the head

   Series already written stay on disk until retention removes them, but they're no longer in memory once they stop receiving samples (after ~1–2 h, when the head compacts).

1. 01Remove the label in code

   If the business needs per-user cart views, that's an analytics event (a log line or a row in a warehouse), not a metric. The metric keeps only a bounded dimension, if any.

   const cartViews = new client.Counter({ name: "shoplite_cart_views_total", help: "Cart views" });
   // ...
     cartViews.inc();                        // no user_id label
     req.log.info({ user_id }, "cart viewed"); // per-user detail goes to logs

2. 02Switch the scenario off and reload

   terminalcopy cmd

   $ curl -s -X POST localhost:8080/admin/chaos -H 'content-type: application/json' -d '{"scenario":"cardinality","enabled":false}'

   curl -s -X POST localhost:9090/-/reload

   ── expected output ──

   {"active":[]}

1. 01sample_limit on every scrape job

   ShopLite normally exposes a few hundred samples. A limit of 5,000 leaves headroom for growth while making an explosion fail the scrape, visibly and without taking Prometheus down.

     - job_name: shoplite
       sample_limit: 5000
       static_configs: [{ targets: ["shoplite:8080"] }]

2. 02Alert on series growth and on scrapes hitting the limit

   deriv gives the per-second slope of a gauge. scrape_samples_post_metric_relabeling shows each target's sample count, and prometheus_target_scrapes_exceeded_sample_limit_total counts scrapes rejected by the limit.

         - alert: PrometheusSeriesGrowingFast
           expr: deriv(prometheus_tsdb_head_series[15m]) * 3600 > 20000
           for: 10m
           labels: { severity: warning, service: prometheus }
           annotations:
             summary: "Head series growing by {{ $value | humanize }}/hour — check for a new high-cardinality label"
         - alert: ScrapeSampleLimitHit
           expr: increase(prometheus_target_scrapes_exceeded_sample_limit_total[10m]) > 0
           labels: { severity: warning, service: prometheus }
           annotations:
             summary: "A target exceeded sample_limit and its scrape was rejected"

Run each one against your lab before revealing the answer. Share your own version in the comments; there's usually more than one correct query.