“Finance wants Q3's checkout latency for the quarterly review. Prometheus keeps 15 days. And last week's restart left a 20-minute hole.”

1. Mon

   REPORT

   'Can we get p99 checkout latency per month for July, August, September?'
2. Mon

   QUERY

   How far back does Prometheus go?

   PromQL· Prometheuscopy

   (time() - prometheus_tsdb_lowest_timestamp_seconds) / 86400

   result

   {instance="localhost:9090", job="prometheus"}  15.0

3. Mon

   DEAD END

   'Just set --storage.tsdb.retention.time=400d'

   It works until the disk fills, the one server dies (and takes 13 months of history with it), or queries over months of raw data time out. Prometheus's local TSDB is designed to be fast and simple for recent data on one node. It isn't durable, clustered storage.
4. Mon

   FINDING

   The gap: one Prometheus, one restart, 20 minutes of nothing

5. Tue

   RESOLVED

   Two Prometheus replicas remote-write to Mimir with 13-month retention; Grafana queries Mimir

1. 01Add Mimir to the lab (monolithic mode, local filesystem)

   Monolithic mode runs every Mimir component in one process: perfect for a lab. In production it runs as separate, scaled components backed by S3 or GCS.

   # compose:
   #   mimir:
   #     image: grafana/mimir:2.16.0
   #     command: ["-config.file=/etc/mimir.yml"]
   #     volumes: ["./mimir/mimir.yml:/etc/mimir.yml:ro"]
   #     ports: ["9009:9009"]
   
   multitenancy_enabled: false
   server:
     http_listen_port: 9009
     log_level: warn
   
   common:
     storage:
       backend: filesystem
       filesystem: { dir: /data/mimir }
   
   blocks_storage:
     storage_prefix: blocks
     tsdb: { dir: /data/ingester }
   
   ingester:
     ring:
       replication_factor: 1
       kvstore: { store: inmemory }
       instance_addr: 127.0.0.1
   
   distributor:
     ha_tracker:
       enable_ha_tracker: false   # enable with a KV store (consul/etcd) for real HA pairs
   
   limits:
     compactor_blocks_retention_period: 400d
     accept_ha_samples: true

2. 02Remote-write from Prometheus, with replica labels

   external_labels identify this Prometheus. The cluster/__replica__ pair is what Mimir's HA tracker deduplicates on. Then add a Grafana data source of type Prometheus pointing at http://mimir:9009/prometheus and use it for long-range dashboards.

   global:
     scrape_interval: 15s
     external_labels:
       cluster: obs-lab
       __replica__: prometheus-a
   
   remote_write:
     - url: http://mimir:9009/api/v1/push
       queue_config:
         max_samples_per_send: 5000

   terminalcopy cmd

   $ curl -s -X POST localhost:9090/-/reload

   curl -s -G localhost:9009/prometheus/api/v1/query --data-urlencode 'query=count(up)' | jq -r '.data.result[0].value[1]'

   ── expected output ──

   4

1. 01Watch remote_write health

   If remote write falls behind (network, a slow backend), samples queue in memory and are eventually dropped. Alert on the lag between the newest sample ingested and the newest sample sent, and on failed samples.

   PromQL· Prometheuscopy

   (
     prometheus_remote_storage_highest_timestamp_in_seconds
     - ignoring(remote_name, url) group_right
     prometheus_remote_storage_queue_highest_sent_timestamp_seconds
   ) > 120
   or
   rate(prometheus_remote_storage_samples_failed_total[5m]) > 0

Run each one against your lab before revealing the answer. Share your own version in the comments; there's usually more than one correct query.