“During the outage, three engineers opened three different 'ShopLite' dashboards. Two were broken, and the third used the old metric names.”

1. Incident

   DEAD END

   'ShopLite Overview', 'ShopLite Overview (copy)', 'ShopLite NEW', 'priya-test-shoplite'…

   Dashboards created by clicking accumulate like console-created infrastructure did in the Terraform course: no owner, no review, and nothing notices when the metrics they query are renamed.
2. Review

   RESOLVED

   One provisioned, version-controlled dashboard per service; the rest archived

1. 01Provision dashboards from files

   A provider tells Grafana to load every JSON file in a folder. Export a dashboard you like from the UI (Share → Export → 'Export for sharing externally' off), save it in the repo, and delete the UI copy.

   apiVersion: 1
   providers:
     - name: services
       folder: Services
       type: file
       allowUiUpdates: false        # the file is the source of truth
       options:
         path: /etc/grafana/provisioning/dashboards/json

2. 02The ShopLite service dashboard (abbreviated)

   A service variable driven by the job label, and SLO, RED, and saturation rows using the queries from this course. Put the full JSON in dashboards/json/service.json.

   {
     "title": "Service overview",
     "uid": "service-overview",
     "templating": {
       "list": [{
         "name": "service", "type": "query", "datasource": { "uid": "prometheus" },
         "query": "label_values(http_request_duration_seconds_count, job)"
       }]
     },
     "panels": [
       { "type": "row", "title": "SLO" },
       { "type": "stat", "title": "Error budget remaining (28d)",
         "targets": [{ "expr": "1 - (sum(increase(http_request_duration_seconds_count{job=\"$service\", status=~\"5..\"}[28d])) / sum(increase(http_request_duration_seconds_count{job=\"$service\"}[28d]))) / 0.005" }] },
       { "type": "row", "title": "RED" },
       { "type": "timeseries", "title": "Requests/s by route",
         "targets": [{ "expr": "sum by (route) (rate(http_request_duration_seconds_count{job=\"$service\"}[$__rate_interval]))" }] },
       { "type": "timeseries", "title": "Error ratio by route",
         "targets": [{ "expr": "sum by (route) (rate(http_request_duration_seconds_count{job=\"$service\", status=~\"5..\"}[$__rate_interval])) / sum by (route) (rate(http_request_duration_seconds_count{job=\"$service\"}[$__rate_interval]))" }] },
       { "type": "timeseries", "title": "p99 latency by route",
         "targets": [{ "expr": "histogram_quantile(0.99, sum by (le, route) (rate(http_request_duration_seconds_bucket{job=\"$service\"}[$__rate_interval])))" }] },
       { "type": "row", "title": "Saturation" },
       { "type": "timeseries", "title": "Event-loop lag p99",
         "targets": [{ "expr": "nodejs_eventloop_lag_p99_seconds{job=\"$service\"}" }] }
     ]
   }

3. 03Or manage Grafana with Terraform

   If your team already uses Terraform (as ShopLite does), the grafana/grafana provider manages dashboards, folders, data sources, and alert rules alongside the infrastructure they observe, reviewed in the same pull requests.

   terraform {
     required_providers {
       grafana = { source = "grafana/grafana", version = "~> 4.0" }
     }
   }
   
   provider "grafana" {
     url  = "http://localhost:3001"
     auth = var.grafana_token
   }
   
   resource "grafana_folder" "services" {
     title = "Services"
   }
   
   resource "grafana_dashboard" "service_overview" {
     folder      = grafana_folder.services.uid
     config_json = file("${path.module}/dashboards/service.json")
   }

1. 01Close the case — tear the lab down

   That's the course. -v also removes the volumes (Prometheus, Loki, Tempo, and Mimir data). Keep the obs-lab/ folder: everything you've built is in its files, and docker compose up -d brings it all back.

   terminalcopy cmd

   $ cd obs-lab && docker compose down -v

   ── expected output ──

   ✔ Container obs-lab-k6-1 Removed

   ✔ Container shoplite Removed

   ...

   ✔ Network obs-lab_default Removed

Run each one against your lab before revealing the answer. Share your own version in the comments; there's usually more than one correct query.