“Log storage grew 40 GB overnight and every Loki query now times out. Nobody changed the logging config.”

1. 09:05

   REPORT

   'Loki is broken, even a 1-hour search times out'

   Loki is fine. It's being asked to scan fifty times more data than yesterday.
2. 09:12

   QUERY

   Which stream is producing the bytes?

   bytes_over_time measures log volume per stream. Summed by container, it gives the logging bill per service.

   LogQL· Lokicopy

   sort_desc(sum by (container) (bytes_over_time({container=~".+"}[1h])))

   result

   {container="shoplite"}    1.41 GB
   {container="payments"}    6.9 MB
   {container="alloy"}       1.2 MB
   {container="prometheus"}  0.4 MB

3. 09:15

   QUERY

   Lines per second — and what they say

   rate() over a log query gives lines per second. Grouping by the parsed msg field shows that one constant message is almost all of it, which is exactly why constant messages (Case 0.3) are so useful.

   LogQL· Lokicopy

   topk(3, sum by (msg) (rate({container="shoplite"} | json [5m])))

   result

   {msg="cache probe"}         1402.3
   {msg="request completed"}     20.1
   {msg="checkout started"}       6.0

4. 09:20

   FINDING

   A debugging line from yesterday's cache investigation shipped at INFO

   Debug output at INFO is the most common source of log floods: it's fine for the one request you're investigating, and ruinous across every request, forever.
5. 12:10

   RESOLVED

   Line dropped at the collector immediately; code fix shipped later

1. 01Stop the flood at the collector — no deploy needed

   Insert a loki.process stage between the Docker source and the writer. stage.json extracts msg; stage.drop discards lines whose msg matches. Restart Alloy, and the lines-per-second graph drops within a minute.

   loki.source.docker "containers" {
     host          = "unix:///var/run/docker.sock"
     targets       = discovery.docker.containers.targets
     relabel_rules = discovery.relabel.containers.rules
     forward_to    = [loki.process.drop_noise.receiver]   // was: loki.write.local.receiver
   }
   
   loki.process "drop_noise" {
     stage.json {
       expressions = { msg = "msg" }
     }
     stage.drop {
       source = "msg"
       value  = "cache probe"
       drop_counter_reason = "debug_noise"
     }
     forward_to = [loki.write.local.receiver]
   }

   terminalcopy cmd

   $ docker compose restart alloy

   ── expected output ──

   ✔ Container obs-lab-alloy-1 Started

2. 02Fix the code: DEBUG, not INFO

   In server.js, change req.log.info({ i }, "cache probe") to req.log.debug(...). With the default LOG_LEVEL=info, pino doesn't even serialise it. Then switch off the scenario.

   terminalcopy cmd

   $ curl -s -X POST localhost:8080/admin/chaos -H 'content-type: application/json' -d '{"scenario":"log-flood","enabled":false}'

   ── expected output ──

   {"active":[]}

1. 01Alert on log volume per service

   Grafana alerting can evaluate LogQL directly. Create an alert rule on this query with a threshold around 3× normal (ShopLite normally writes ~30 lines/s). Loki's own per-tenant limits (ingestion_rate_mb, per_stream_rate_limit) are the hard backstop.

   LogQL· Lokicopy

   sum by (container) (rate({container=~".+"}[5m])) > 100

Run each one against your lab before revealing the answer. Share your own version in the comments; there's usually more than one correct query.